Sep 16 15:48:17.639547 ip-10-0-136-228 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 16 15:48:17.639558 ip-10-0-136-228 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 16 15:48:17.639568 ip-10-0-136-228 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 16 15:48:17.640217 ip-10-0-136-228 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 16 15:48:27.664256 ip-10-0-136-228 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 16 15:48:27.664271 ip-10-0-136-228 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot e46d49af66a8480585c6896880556ccb --
Sep 16 15:49:59.439113 ip-10-0-136-228 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 15:49:59.869005 ip-10-0-136-228 kubenswrapper[2554]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 15:49:59.869005 ip-10-0-136-228 kubenswrapper[2554]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 15:49:59.869005 ip-10-0-136-228 kubenswrapper[2554]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 15:49:59.869005 ip-10-0-136-228 kubenswrapper[2554]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 15:49:59.869005 ip-10-0-136-228 kubenswrapper[2554]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 15:49:59.871010 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.870913    2554 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 15:49:59.875245 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875228    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 15:49:59.875245 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875245    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 15:49:59.875314 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875251    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 15:49:59.875314 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875254    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 15:49:59.875314 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875257    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 15:49:59.875314 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875260    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 15:49:59.875314 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875263    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 15:49:59.875314 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875267    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 15:49:59.875314 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875271    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 15:49:59.875314 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875274    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 15:49:59.875314 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875277    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 15:49:59.875314 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875280    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 15:49:59.875314 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875283    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 15:49:59.875314 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875286    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 15:49:59.875314 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875290    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 15:49:59.875314 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875292    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 15:49:59.875314 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875295    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 15:49:59.875314 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875298    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 15:49:59.875314 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875300    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 15:49:59.875314 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875303    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 15:49:59.875314 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875305    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 15:49:59.875314 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875308    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 15:49:59.875904 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875311    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 15:49:59.875904 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875314    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 15:49:59.875904 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875317    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 15:49:59.875904 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875320    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 15:49:59.875904 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875323    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 15:49:59.875904 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875325    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 15:49:59.875904 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875328    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 15:49:59.875904 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875330    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 15:49:59.875904 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875333    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 15:49:59.875904 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875337    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 15:49:59.875904 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875341    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 15:49:59.875904 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875344    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 15:49:59.875904 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875347    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 15:49:59.875904 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875349    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 15:49:59.875904 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875352    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 15:49:59.875904 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875355    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 15:49:59.875904 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875359    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 15:49:59.875904 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875362    2554 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 15:49:59.875904 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875364    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 15:49:59.875904 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875367    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 15:49:59.876406 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875370    2554 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 15:49:59.876406 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875372    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 15:49:59.876406 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875375    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 15:49:59.876406 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875377    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 15:49:59.876406 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875380    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 15:49:59.876406 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875382    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 15:49:59.876406 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875385    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 15:49:59.876406 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875387    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 15:49:59.876406 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875390    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 15:49:59.876406 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875393    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 15:49:59.876406 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875395    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 15:49:59.876406 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875397    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 15:49:59.876406 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875400    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 15:49:59.876406 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875403    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 15:49:59.876406 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875406    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 15:49:59.876406 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875409    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 15:49:59.876406 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875412    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 15:49:59.876406 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875414    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 15:49:59.876406 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875417    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 15:49:59.876406 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875420    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 15:49:59.876899 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875422    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 15:49:59.876899 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875425    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 15:49:59.876899 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875428    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 15:49:59.876899 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875430    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 15:49:59.876899 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875434    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 15:49:59.876899 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875436    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 15:49:59.876899 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875439    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 15:49:59.876899 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875443    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 15:49:59.876899 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875447    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 15:49:59.876899 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875450    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 15:49:59.876899 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875453    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 15:49:59.876899 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875456    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 15:49:59.876899 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875458    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 15:49:59.876899 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875461    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 15:49:59.876899 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875463    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 15:49:59.876899 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875466    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 15:49:59.876899 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875468    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 15:49:59.876899 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875471    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 15:49:59.876899 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875473    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 15:49:59.877408 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875476    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 15:49:59.877408 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875478    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 15:49:59.877408 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875481    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 15:49:59.877408 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875483    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 15:49:59.877408 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875486    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 15:49:59.877408 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875488    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 15:49:59.877408 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875492    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 15:49:59.877408 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875495    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 15:49:59.877408 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875498    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 15:49:59.877408 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875500    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 15:49:59.877408 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875503    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 15:49:59.877408 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875505    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 15:49:59.877408 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875507    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 15:49:59.877408 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875510    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 15:49:59.877408 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875513    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 15:49:59.877408 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875515    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 15:49:59.877408 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875518    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 15:49:59.877408 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.875520    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 15:49:59.877408 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876238    2554 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 15:49:59.877880 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876245    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 15:49:59.877880 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876248    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 15:49:59.877880 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876251    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 15:49:59.877880 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876254    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 15:49:59.877880 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876257    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 15:49:59.877880 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876260    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 15:49:59.877880 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876263    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 15:49:59.877880 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876266    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 15:49:59.877880 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876268    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 15:49:59.877880 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876271    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 15:49:59.877880 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876273    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 15:49:59.877880 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876277    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 15:49:59.877880 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876279    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 15:49:59.877880 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876282    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 15:49:59.877880 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876284    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 15:49:59.877880 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876287    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 15:49:59.877880 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876289    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 15:49:59.877880 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876293    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 15:49:59.877880 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876295    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 15:49:59.877880 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876299    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 15:49:59.878387 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876302    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 15:49:59.878387 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876305    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 15:49:59.878387 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876308    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 15:49:59.878387 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876311    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 15:49:59.878387 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876314    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 15:49:59.878387 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876317    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 15:49:59.878387 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876319    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 15:49:59.878387 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876322    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 15:49:59.878387 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876324    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 15:49:59.878387 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876327    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 15:49:59.878387 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876329    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 15:49:59.878387 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876331    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 15:49:59.878387 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876334    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 15:49:59.878387 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876338    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 15:49:59.878387 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876342    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 15:49:59.878387 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876345    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 15:49:59.878387 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876348    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 15:49:59.878387 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876351    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 15:49:59.878387 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876354    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 15:49:59.878861 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876358    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 15:49:59.878861 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876362    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 15:49:59.878861 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876365    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 15:49:59.878861 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876367    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 15:49:59.878861 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876371    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 15:49:59.878861 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876374    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 15:49:59.878861 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876376    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 15:49:59.878861 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876379    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 15:49:59.878861 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876381    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 15:49:59.878861 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876384    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 15:49:59.878861 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876388    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 15:49:59.878861 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876391    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 15:49:59.878861 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876394    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 15:49:59.878861 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876397    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 15:49:59.878861 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876400    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 15:49:59.878861 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876402    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 15:49:59.878861 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876405    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 15:49:59.878861 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876409    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 15:49:59.878861 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876413    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 15:49:59.879331 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876416    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 15:49:59.879331 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876419    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 15:49:59.879331 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876422    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 15:49:59.879331 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876425    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 15:49:59.879331 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876428    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 15:49:59.879331 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876431    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 15:49:59.879331 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876433    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 15:49:59.879331 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876436    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 15:49:59.879331 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876439    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 15:49:59.879331 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876441    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 15:49:59.879331 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876444    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 15:49:59.879331 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876446    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 15:49:59.879331 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876449    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 15:49:59.879331 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876451    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 15:49:59.879331 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876454    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 15:49:59.879331 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876456    2554 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 15:49:59.879331 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876459    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 15:49:59.879331 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876462    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 15:49:59.879331 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876464    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 15:49:59.879331 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876467    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 15:49:59.879837 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876469    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 15:49:59.879837 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876472    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 15:49:59.879837 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876474    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 15:49:59.879837 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876476    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 15:49:59.879837 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876480    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 15:49:59.879837 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876483    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 15:49:59.879837 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876485    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 15:49:59.879837 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876488    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 15:49:59.879837 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876490    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 15:49:59.879837 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876493    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 15:49:59.879837 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876496    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 15:49:59.879837 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876498    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 15:49:59.879837 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876501    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 15:49:59.879837 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876503    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 15:49:59.879837 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876506    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 15:49:59.879837 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876508    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 15:49:59.879837 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876511    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 15:49:59.879837 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876513    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 15:49:59.879837 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876515    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 15:49:59.880309 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876518    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 15:49:59.880309 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876918    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 15:49:59.880309 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876924    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 15:49:59.880309 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876926    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 15:49:59.880309 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876930    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 15:49:59.880309 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876933    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 15:49:59.880309 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876936    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 15:49:59.880309 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876939    2554 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 15:49:59.880309 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876948    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 15:49:59.880309 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876952    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 15:49:59.880309 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876956    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 15:49:59.880309 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876958    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 15:49:59.880309 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876961    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 15:49:59.880309 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876964    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 15:49:59.880309 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876966    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 15:49:59.880309 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876970    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 15:49:59.880309 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876974    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 15:49:59.880309 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876977    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 15:49:59.880309 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876982    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 15:49:59.880775 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876985    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 15:49:59.880775 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.876988    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 15:49:59.880775 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877000    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 15:49:59.880775 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877003    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 15:49:59.880775 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877006    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 15:49:59.880775 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877009    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 15:49:59.880775 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877011    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 15:49:59.880775 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877014    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 15:49:59.880775 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877017    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 15:49:59.880775 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877019    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 15:49:59.880775 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877022    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 15:49:59.880775 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877025    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 15:49:59.880775 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877027    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 15:49:59.880775 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877030    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 15:49:59.880775 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877032    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 15:49:59.880775 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877035    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 15:49:59.880775 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877038    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 15:49:59.880775 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877041    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 15:49:59.880775 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877044    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 15:49:59.880775 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877046    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 15:49:59.881284 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877049    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 15:49:59.881284 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877051    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 15:49:59.881284 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877054    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 15:49:59.881284 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877057    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 15:49:59.881284 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877059    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 15:49:59.881284 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877062    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 15:49:59.881284 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877065    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 15:49:59.881284 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877067    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 15:49:59.881284 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877070    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 15:49:59.881284 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877073    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 15:49:59.881284 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877076    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 15:49:59.881284 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877079    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 15:49:59.881284 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877081    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 15:49:59.881284 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877084    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 15:49:59.881284 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877086    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 15:49:59.881284 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877090    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 15:49:59.881284 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877092    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 15:49:59.881284 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877095    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 15:49:59.881284 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877097    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 15:49:59.881284 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877100    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 15:49:59.881773 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877102    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 15:49:59.881773 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877105    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 15:49:59.881773 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877107    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 15:49:59.881773 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877109    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 15:49:59.881773 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877112    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 15:49:59.881773 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877114    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 15:49:59.881773 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877117    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 15:49:59.881773 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877123    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 15:49:59.881773 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877125    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 15:49:59.881773 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877128    2554 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 15:49:59.881773 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877130    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 15:49:59.881773 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877133    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 15:49:59.881773 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877137    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 15:49:59.881773 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877141    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 15:49:59.881773 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877145    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 15:49:59.881773 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877148    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 15:49:59.881773 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877150    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 15:49:59.881773 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877153    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 15:49:59.881773 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877156    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 15:49:59.882293 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877158    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 15:49:59.882293 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877161    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 15:49:59.882293 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877164    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 15:49:59.882293 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877166    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 15:49:59.882293 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877169    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 15:49:59.882293 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877171    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 15:49:59.882293 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877173    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 15:49:59.882293 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877176    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 15:49:59.882293 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877179    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 15:49:59.882293 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877181    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 15:49:59.882293 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877184    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 15:49:59.882293 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877187    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 15:49:59.882293 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877190    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 15:49:59.882293 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877192    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 15:49:59.882293 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877195    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 15:49:59.882293 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877197    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 15:49:59.882293 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877199    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 15:49:59.882293 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877202    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 15:49:59.882293 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877204    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 15:49:59.882293 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877207    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 15:49:59.882777 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877211    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 15:49:59.882777 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877213    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 15:49:59.882777 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877293    2554 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 15:49:59.882777 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877333    2554 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 15:49:59.882777 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877339    2554 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 15:49:59.882777 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877346    2554 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 15:49:59.882777 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877353    2554 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 15:49:59.882777 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877356    2554 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 15:49:59.882777 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877361    2554 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 15:49:59.882777 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877365    2554 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 15:49:59.882777 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877368    2554 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 15:49:59.882777 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877371    2554 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 15:49:59.882777 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877375    2554 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 15:49:59.882777 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877378    2554 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 15:49:59.882777 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877381    2554 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 15:49:59.882777 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877384    2554 flags.go:64] FLAG: --cgroup-root=""
Sep 16 15:49:59.882777 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877387    2554 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 15:49:59.882777 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877390    2554 flags.go:64] FLAG: --client-ca-file=""
Sep 16 15:49:59.882777 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877393    2554 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 15:49:59.882777 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877396    2554 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 15:49:59.882777 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877401    2554 flags.go:64] FLAG: --cluster-domain=""
Sep 16 15:49:59.882777 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877404    2554 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 15:49:59.882777 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877407    2554 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 15:49:59.882777 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877410    2554 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 15:49:59.883356 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877414    2554 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 15:49:59.883356 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877418    2554 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 15:49:59.883356 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877421    2554 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 15:49:59.883356 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877424    2554 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 15:49:59.883356 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877428    2554 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 15:49:59.883356 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877431    2554 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 15:49:59.883356 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877434    2554 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 15:49:59.883356 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877437    2554 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 15:49:59.883356 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877441    2554 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 15:49:59.883356 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877444    2554 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 15:49:59.883356 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877449    2554 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 15:49:59.883356 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877452    2554 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 15:49:59.883356 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877455    2554 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 15:49:59.883356 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877459    2554 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 15:49:59.883356 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877462    2554 flags.go:64] FLAG: --enable-server="true"
Sep 16 15:49:59.883356 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877466    2554 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 15:49:59.883356 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877470    2554 flags.go:64] FLAG: --event-burst="100"
Sep 16 15:49:59.883356 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877474    2554 flags.go:64] FLAG: --event-qps="50"
Sep 16 15:49:59.883356 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877477    2554 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 15:49:59.883356 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877480    2554 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 15:49:59.883356 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877483    2554 flags.go:64] FLAG: --eviction-hard=""
Sep 16 15:49:59.883356 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877486    2554 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 15:49:59.883356 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877489    2554 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 15:49:59.883356 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877492    2554 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 15:49:59.883356 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877495    2554 flags.go:64] FLAG: --eviction-soft=""
Sep 16 15:49:59.883988 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877498    2554 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 15:49:59.883988 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877501    2554 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 15:49:59.883988 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877504    2554 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 15:49:59.883988 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877507    2554 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 15:49:59.883988 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877510    2554 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 15:49:59.883988 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877513    2554 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 15:49:59.883988 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877516    2554 flags.go:64] FLAG: --feature-gates=""
Sep 16 15:49:59.883988 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877520    2554 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 15:49:59.883988 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877523    2554 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 15:49:59.883988 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877526    2554 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 15:49:59.883988 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877530    2554 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 15:49:59.883988 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877533    2554 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 15:49:59.883988 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877536    2554 flags.go:64] FLAG: --help="false"
Sep 16 15:49:59.883988 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877539    2554 flags.go:64] FLAG: --hostname-override="ip-10-0-136-228.ec2.internal"
Sep 16 15:49:59.883988 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877542    2554 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 15:49:59.883988 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877545    2554 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 15:49:59.883988 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877548    2554 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 15:49:59.883988 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877551    2554 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 15:49:59.883988 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877555    2554 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 15:49:59.883988 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877557    2554 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 15:49:59.883988 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877563    2554 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 15:49:59.883988 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877566    2554 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 15:49:59.883988 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877569    2554 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 15:49:59.884580 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877573    2554 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 15:49:59.884580 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877576    2554 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 15:49:59.884580 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877579    2554 flags.go:64] FLAG: --kube-reserved=""
Sep 16 15:49:59.884580 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877582    2554 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 15:49:59.884580 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877585    2554 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 15:49:59.884580 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877588    2554 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 15:49:59.884580 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877591    2554 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 15:49:59.884580 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877594    2554 flags.go:64] FLAG: --lock-file=""
Sep 16 15:49:59.884580 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877597    2554 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 15:49:59.884580 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877600    2554 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 15:49:59.884580 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877603    2554 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 15:49:59.884580 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877609    2554 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 15:49:59.884580 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877612    2554 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 15:49:59.884580 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877615    2554 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 15:49:59.884580 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877618    2554 flags.go:64] FLAG: --logging-format="text"
Sep 16 15:49:59.884580 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877621    2554 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 15:49:59.884580 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877624    2554 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 15:49:59.884580 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877627    2554 flags.go:64] FLAG: --manifest-url=""
Sep 16 15:49:59.884580 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877630    2554 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 15:49:59.884580 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877634    2554 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 15:49:59.884580 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877637    2554 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 15:49:59.884580 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877642    2554 flags.go:64] FLAG: --max-pods="110"
Sep 16 15:49:59.884580 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877645    2554 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 15:49:59.884580 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877648    2554 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 15:49:59.884580 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877651    2554 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 15:49:59.885235 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877655    2554 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 15:49:59.885235 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877658    2554 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 15:49:59.885235 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877661    2554 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 15:49:59.885235 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877664    2554 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 15:49:59.885235 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877672    2554 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 15:49:59.885235 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877676    2554 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 15:49:59.885235 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877679    2554 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 15:49:59.885235 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877682    2554 flags.go:64] FLAG: --pod-cidr=""
Sep 16 15:49:59.885235 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877687    2554 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 15:49:59.885235 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877693    2554 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 15:49:59.885235 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877696    2554 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 15:49:59.885235 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877699    2554 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 15:49:59.885235 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877702    2554 flags.go:64] FLAG: --port="10250"
Sep 16 15:49:59.885235 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877705    2554 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 15:49:59.885235 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877708    2554 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d3e0f9028d9b2911"
Sep 16 15:49:59.885235 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877711    2554 flags.go:64] FLAG: --qos-reserved=""
Sep 16 15:49:59.885235 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877715    2554 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 15:49:59.885235 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877717    2554 flags.go:64] FLAG: --register-node="true"
Sep 16 15:49:59.885235 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877720    2554 flags.go:64] FLAG: --register-with-taints=""
Sep 16 15:49:59.885235 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877724    2554 flags.go:64] FLAG: --registry-burst="10"
Sep 16 15:49:59.885235 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877727    2554 flags.go:64] FLAG: --registry-qps="5"
Sep 16 15:49:59.885235 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877729    2554 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 15:49:59.885235 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877732    2554 flags.go:64] FLAG: --reserved-memory=""
Sep 16 15:49:59.885235 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877736    2554 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 15:49:59.885235 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877739    2554 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 15:49:59.885919 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877742    2554 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 15:49:59.885919 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877745    2554 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 15:49:59.885919 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877748    2554 flags.go:64] FLAG: --runonce="false"
Sep 16 15:49:59.885919 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877751    2554 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 15:49:59.885919 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877754    2554 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 15:49:59.885919 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877756    2554 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 15:49:59.885919 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877760    2554 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 15:49:59.885919 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877762    2554 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 15:49:59.885919 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877765    2554 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 15:49:59.885919 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877768    2554 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 15:49:59.885919 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877772    2554 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 15:49:59.885919 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877775    2554 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 15:49:59.885919 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877780    2554 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 15:49:59.885919 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877783    2554 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 15:49:59.885919 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877786    2554 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 15:49:59.885919 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877789    2554 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 15:49:59.885919 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877805    2554 flags.go:64] FLAG: --system-cgroups=""
Sep 16 15:49:59.885919 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877808    2554 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 15:49:59.885919 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877814    2554 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 15:49:59.885919 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877817    2554 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 15:49:59.885919 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877820    2554 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 15:49:59.885919 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877825    2554 flags.go:64] FLAG: --tls-min-version=""
Sep 16 15:49:59.885919 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877828    2554 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 15:49:59.885919 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877831    2554 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 15:49:59.885919 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877834    2554 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 15:49:59.886545 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877837    2554 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 15:49:59.886545 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877840    2554 flags.go:64] FLAG: --v="2"
Sep 16 15:49:59.886545 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877844    2554 flags.go:64] FLAG: --version="false"
Sep 16 15:49:59.886545 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877848    2554 flags.go:64] FLAG: --vmodule=""
Sep 16 15:49:59.886545 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877852    2554 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 15:49:59.886545 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.877855    2554 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 15:49:59.886545 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877970    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 15:49:59.886545 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877975    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 15:49:59.886545 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877978    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 15:49:59.886545 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877981    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 15:49:59.886545 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877983    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 15:49:59.886545 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877986    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 15:49:59.886545 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877989    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 15:49:59.886545 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877994    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 15:49:59.886545 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877996    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 15:49:59.886545 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.877999    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 15:49:59.886545 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878002    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 15:49:59.886545 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878004    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 15:49:59.886545 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878007    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 15:49:59.886545 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878010    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 15:49:59.886545 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878013    2554 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 15:49:59.886545 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878016    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 15:49:59.887100 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878018    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 15:49:59.887100 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878021    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 15:49:59.887100 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878026    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 15:49:59.887100 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878030    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 15:49:59.887100 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878032    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 15:49:59.887100 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878036    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 15:49:59.887100 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878038    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 15:49:59.887100 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878041    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 15:49:59.887100 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878043    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 15:49:59.887100 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878046    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 15:49:59.887100 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878049    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 15:49:59.887100 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878051    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 15:49:59.887100 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878054    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 15:49:59.887100 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878056    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 15:49:59.887100 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878061    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 15:49:59.887100 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878064    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 15:49:59.887100 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878068    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 15:49:59.887100 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878070    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 15:49:59.887100 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878073    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 15:49:59.887568 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878076    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 15:49:59.887568 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878079    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 15:49:59.887568 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878081    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 15:49:59.887568 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878084    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 15:49:59.887568 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878088    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 15:49:59.887568 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878091    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 15:49:59.887568 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878093    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 15:49:59.887568 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878096    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 15:49:59.887568 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878099    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 15:49:59.887568 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878102    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 15:49:59.887568 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878105    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 15:49:59.887568 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878108    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 15:49:59.887568 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878111    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 15:49:59.887568 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878114    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 15:49:59.887568 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878117    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 15:49:59.887568 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878119    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 15:49:59.887568 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878122    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 15:49:59.887568 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878125    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 15:49:59.887568 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878128    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 15:49:59.888058 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878130    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 15:49:59.888058 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878133    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 15:49:59.888058 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878135    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 15:49:59.888058 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878138    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 15:49:59.888058 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878140    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 15:49:59.888058 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878143    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 15:49:59.888058 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878146    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 15:49:59.888058 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878149    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 15:49:59.888058 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878151    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 15:49:59.888058 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878154    2554 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 15:49:59.888058 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878157    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 15:49:59.888058 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878159    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 15:49:59.888058 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878162    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 15:49:59.888058 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878164    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 15:49:59.888058 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878167    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 15:49:59.888058 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878170    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 15:49:59.888058 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878172    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 15:49:59.888058 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878177    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 15:49:59.888058 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878179    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 15:49:59.888058 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878182    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 15:49:59.888541 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878184    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 15:49:59.888541 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878187    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 15:49:59.888541 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878190    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 15:49:59.888541 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878193    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 15:49:59.888541 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878196    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 15:49:59.888541 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878200    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 15:49:59.888541 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878203    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 15:49:59.888541 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878206    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 15:49:59.888541 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878208    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 15:49:59.888541 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878211    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 15:49:59.888541 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878214    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 15:49:59.888541 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878217    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 15:49:59.888541 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878219    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 15:49:59.888541 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878222    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 15:49:59.888541 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878224    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 15:49:59.888541 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878227    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 15:49:59.888541 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878230    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 15:49:59.888541 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878233    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 15:49:59.888541 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878236    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 15:49:59.888541 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878238    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 15:49:59.889211 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878241    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 15:49:59.889211 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878243    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 15:49:59.889211 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878246    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 15:49:59.889211 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878248    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 15:49:59.889211 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.878251    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 15:49:59.889211 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.879141    2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 15:49:59.889211 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.886981    2554 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 15:49:59.889211 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.887002    2554 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 15:49:59.889211 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.887875    2554 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 15:49:59.889211 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.887883    2554 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 15:49:59.889211 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.887953    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 15:49:59.889211 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.887959    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 15:49:59.889211 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.887963    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 15:49:59.889211 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.887967    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 15:49:59.889769 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.887970    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 15:49:59.889769 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.887973    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 15:49:59.889769 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.887976    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 15:49:59.889769 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.887979    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 15:49:59.889769 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.887981    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 15:49:59.889769 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.887984    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 15:49:59.889769 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.887987    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 15:49:59.889769 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.887989    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 15:49:59.889769 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.887992    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 15:49:59.889769 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.887995    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 15:49:59.889769 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.887997    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 15:49:59.889769 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.887999    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 15:49:59.889769 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888002    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 15:49:59.889769 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888004    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 15:49:59.889769 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888007    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 15:49:59.889769 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888009    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 15:49:59.889769 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888014    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 15:49:59.889769 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888018    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 15:49:59.889769 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888021    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 15:49:59.890278 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888025    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 15:49:59.890278 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888028    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 15:49:59.890278 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888032    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 15:49:59.890278 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888035    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 15:49:59.890278 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888037    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 15:49:59.890278 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888040    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 15:49:59.890278 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888053    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 15:49:59.890278 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888056    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 15:49:59.890278 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888059    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 15:49:59.890278 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888062    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 15:49:59.890278 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888066    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 15:49:59.890278 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888068    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 15:49:59.890278 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888071    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 15:49:59.890278 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888074    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 15:49:59.890278 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888077    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 15:49:59.890278 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888079    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 15:49:59.890278 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888082    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 15:49:59.890278 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888085    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 15:49:59.890278 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888088    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 15:49:59.890748 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888092    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 15:49:59.890748 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888095    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 15:49:59.890748 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888098    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 15:49:59.890748 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888100    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 15:49:59.890748 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888103    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 15:49:59.890748 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888107    2554 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 15:49:59.890748 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888110    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 15:49:59.890748 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888112    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 15:49:59.890748 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888115    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 15:49:59.890748 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888118    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 15:49:59.890748 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888121    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 15:49:59.890748 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888123    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 15:49:59.890748 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888126    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 15:49:59.890748 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888128    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 15:49:59.890748 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888131    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 15:49:59.890748 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888133    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 15:49:59.890748 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888136    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 15:49:59.890748 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888138    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 15:49:59.890748 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888141    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 15:49:59.890748 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888144    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 15:49:59.891251 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888152    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 15:49:59.891251 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888155    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 15:49:59.891251 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888158    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 15:49:59.891251 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888160    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 15:49:59.891251 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888163    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 15:49:59.891251 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888165    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 15:49:59.891251 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888168    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 15:49:59.891251 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888171    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 15:49:59.891251 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888173    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 15:49:59.891251 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888176    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 15:49:59.891251 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888179    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 15:49:59.891251 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888181    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 15:49:59.891251 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888184    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 15:49:59.891251 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888187    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 15:49:59.891251 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888189    2554 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 15:49:59.891251 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888191    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 15:49:59.891251 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888194    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 15:49:59.891251 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888197    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 15:49:59.891251 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888200    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 15:49:59.891251 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888203    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 15:49:59.891745 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888205    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 15:49:59.891745 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888207    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 15:49:59.891745 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888210    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 15:49:59.891745 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888212    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 15:49:59.891745 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888215    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 15:49:59.891745 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888217    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 15:49:59.891745 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888220    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 15:49:59.891745 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888223    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 15:49:59.891745 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888225    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 15:49:59.891745 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888228    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 15:49:59.891745 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888230    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 15:49:59.891745 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888233    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 15:49:59.891745 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888235    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 15:49:59.891745 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888243    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 15:49:59.891745 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888246    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 15:49:59.891745 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888249    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 15:49:59.891745 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888251    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 15:49:59.892167 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.888256    2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 15:49:59.892167 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888416    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 15:49:59.892167 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888422    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 15:49:59.892167 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888425    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 15:49:59.892167 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888428    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 15:49:59.892167 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888431    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 15:49:59.892167 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888434    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 15:49:59.892167 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888436    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 15:49:59.892167 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888439    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 15:49:59.892167 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888442    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 15:49:59.892167 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888444    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 15:49:59.892167 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888448    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 15:49:59.892167 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888451    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 15:49:59.892167 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888454    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 15:49:59.892167 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888456    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 15:49:59.892576 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888459    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 15:49:59.892576 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888462    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 15:49:59.892576 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888465    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 15:49:59.892576 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888467    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 15:49:59.892576 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888470    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 15:49:59.892576 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888472    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 15:49:59.892576 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888475    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 15:49:59.892576 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888477    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 15:49:59.892576 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888480    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 15:49:59.892576 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888482    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 15:49:59.892576 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888485    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 15:49:59.892576 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888488    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 15:49:59.892576 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888493    2554 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 15:49:59.892576 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888496    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 15:49:59.892576 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888504    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 15:49:59.892576 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888507    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 15:49:59.892576 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888509    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 15:49:59.892576 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888513    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 15:49:59.892576 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888515    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 15:49:59.893146 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888518    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 15:49:59.893146 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888521    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 15:49:59.893146 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888523    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 15:49:59.893146 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888526    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 15:49:59.893146 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888528    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 15:49:59.893146 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888531    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 15:49:59.893146 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888534    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 15:49:59.893146 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888536    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 15:49:59.893146 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888538    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 15:49:59.893146 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888542    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 15:49:59.893146 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888545    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 15:49:59.893146 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888547    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 15:49:59.893146 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888550    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 15:49:59.893146 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888553    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 15:49:59.893146 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888555    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 15:49:59.893146 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888558    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 15:49:59.893146 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888561    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 15:49:59.893146 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888563    2554 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 15:49:59.893146 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888566    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 15:49:59.893146 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888568    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 15:49:59.893632 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888571    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 15:49:59.893632 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888573    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 15:49:59.893632 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888576    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 15:49:59.893632 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888580    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 15:49:59.893632 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888584    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 15:49:59.893632 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888587    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 15:49:59.893632 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888590    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 15:49:59.893632 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888593    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 15:49:59.893632 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888602    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 15:49:59.893632 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888605    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 15:49:59.893632 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888608    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 15:49:59.893632 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888610    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 15:49:59.893632 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888613    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 15:49:59.893632 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888616    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 15:49:59.893632 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888618    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 15:49:59.893632 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888621    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 15:49:59.893632 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888623    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 15:49:59.893632 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888626    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 15:49:59.893632 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888628    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 15:49:59.894141 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888631    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 15:49:59.894141 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888633    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 15:49:59.894141 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888636    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 15:49:59.894141 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888638    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 15:49:59.894141 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888641    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 15:49:59.894141 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888644    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 15:49:59.894141 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888646    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 15:49:59.894141 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888655    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 15:49:59.894141 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888658    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 15:49:59.894141 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888660    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 15:49:59.894141 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888663    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 15:49:59.894141 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888665    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 15:49:59.894141 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888668    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 15:49:59.894141 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888671    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 15:49:59.894141 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888673    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 15:49:59.894141 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888676    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 15:49:59.894141 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888678    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 15:49:59.894141 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888681    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 15:49:59.894141 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888684    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 15:49:59.894603 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888688    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 15:49:59.894603 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888691    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 15:49:59.894603 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888694    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 15:49:59.894603 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888704    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 15:49:59.894603 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888707    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 15:49:59.894603 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888710    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 15:49:59.894603 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888712    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 15:49:59.894603 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:49:59.888715    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 15:49:59.894603 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.888720    2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 15:49:59.894603 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.890122    2554 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 15:49:59.894603 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.892724    2554 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 15:49:59.894603 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.894148    2554 server.go:1019] "Starting client certificate rotation"
Sep 16 15:49:59.894603 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.894251    2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 15:49:59.894946 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.894290    2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 15:49:59.897218 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.897202    2554 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 15:49:59.900350 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.900333    2554 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 15:49:59.909539 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.909521    2554 log.go:25] "Validated CRI v1 runtime API"
Sep 16 15:49:59.915133 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.915116    2554 log.go:25] "Validated CRI v1 image API"
Sep 16 15:49:59.916758 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.916743    2554 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 15:49:59.920054 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.920028    2554 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9837f8f0-7e13-4bfd-9df7-7e793ea22237:/dev/nvme0n1p4 bbc738eb-0276-4e18-9032-3250aa79f8d4:/dev/nvme0n1p3]
Sep 16 15:49:59.920126 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.920054    2554 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 15:49:59.925707 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.925585    2554 manager.go:216] Machine: {Timestamp:2026-09-16 15:49:59.924091824 +0000 UTC m=+0.368784013 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098381 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a03e64afedbd8bc23f19887c8264d SystemUUID:ec2a03e6-4afe-dbd8-bc23-f19887c8264d BootID:e46d49af-66a8-4805-85c6-896880556ccb Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:fb:94:76:cf Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:fb:94:76:cf Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7e:7a:e6:e6:68:41 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 15:49:59.925707 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.925697    2554 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 15:49:59.925918 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.925829    2554 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 15:49:59.926089 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.926072    2554 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 15:49:59.929504 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.929476    2554 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 15:49:59.929654 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.929506    2554 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-228.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 15:49:59.930287 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.930276    2554 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 15:49:59.930326 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.930289    2554 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 15:49:59.930326 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.930298    2554 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 15:49:59.931073 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.931063    2554 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 15:49:59.932212 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.932202    2554 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 15:49:59.936640 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.936626    2554 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 15:49:59.936766 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.936756    2554 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 15:49:59.936817 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.936787    2554 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 15:49:59.936817 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.936807    2554 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 15:49:59.936894 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.936818    2554 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 15:49:59.936894 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.936828    2554 kubelet.go:398] "Adding apiserver pod source"
Sep 16 15:49:59.936894 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.936836    2554 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 15:49:59.937251 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.937238    2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 15:49:59.937286 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.937258    2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 15:49:59.939715 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.939700    2554 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 15:49:59.942748 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.942734    2554 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 15:49:59.942816 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.942764    2554 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 15:49:59.944900 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.944886    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 15:49:59.944900 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.944903    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 15:49:59.945010 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.944910    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 15:49:59.945010 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.944915    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 15:49:59.945010 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.944924    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 15:49:59.945010 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.944929    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 15:49:59.945010 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.944950    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 15:49:59.945010 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.944956    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 15:49:59.945010 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.944970    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 15:49:59.945010 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.944976    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 15:49:59.945010 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.944988    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 15:49:59.945010 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.944994    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 15:49:59.945838 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.945829    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 15:49:59.945838 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.945838    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 15:49:59.951028 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.951000    2554 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-228.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 15:49:59.951132 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:49:59.951031    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 15:49:59.951132 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:49:59.951048    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-228.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 15:49:59.953555 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.953537    2554 server.go:1268] "Started kubelet"
Sep 16 15:49:59.953649 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.953619    2554 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 15:49:59.953717 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.953630    2554 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 15:49:59.953717 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.953686    2554 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 15:49:59.954460 ip-10-0-136-228 systemd[1]: Started Kubernetes Kubelet.
Sep 16 15:49:59.955207 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.955184    2554 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 15:49:59.955918 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.955904    2554 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 15:49:59.959275 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:49:59.959252    2554 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 15:49:59.962305 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:49:59.961128    2554 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-228.ec2.internal.18d5d7e3b59295e6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-228.ec2.internal,UID:ip-10-0-136-228.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-228.ec2.internal,},FirstTimestamp:2026-09-16 15:49:59.95347095 +0000 UTC m=+0.398163119,LastTimestamp:2026-09-16 15:49:59.95347095 +0000 UTC m=+0.398163119,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-228.ec2.internal,}"
Sep 16 15:49:59.963705 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.963685    2554 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 15:49:59.963810 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.963703    2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 15:49:59.963879 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:49:59.963852    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-228.ec2.internal\" not found"
Sep 16 15:49:59.964489 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.964466    2554 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 15:49:59.964489 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.964470    2554 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 15:49:59.964634 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.964501    2554 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 15:49:59.964634 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.964557    2554 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 15:49:59.964634 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.964565    2554 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 15:49:59.964979 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.964955    2554 factory.go:55] Registering systemd factory
Sep 16 15:49:59.964979 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.964976    2554 factory.go:223] Registration of the systemd container factory successfully
Sep 16 15:49:59.965302 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.965288    2554 factory.go:153] Registering CRI-O factory
Sep 16 15:49:59.965302 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.965304    2554 factory.go:223] Registration of the crio container factory successfully
Sep 16 15:49:59.965439 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.965380    2554 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 15:49:59.965439 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.965405    2554 factory.go:103] Registering Raw factory
Sep 16 15:49:59.965439 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.965431    2554 manager.go:1196] Started watching for new ooms in manager
Sep 16 15:49:59.965878 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.965858    2554 manager.go:349] Starting recovery of all containers
Sep 16 15:49:59.968267 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:49:59.968237    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 15:49:59.968359 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:49:59.968238    2554 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-228.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 15:49:59.974810 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.974622    2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rcxzc"
Sep 16 15:49:59.976131 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.976114    2554 manager.go:354] Recovery completed
Sep 16 15:49:59.980990 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.980966    2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rcxzc"
Sep 16 15:49:59.982351 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.982337    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 15:49:59.984245 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.984230    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 15:49:59.984310 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.984258    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 15:49:59.984310 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.984271    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasSufficientPID"
Sep 16 15:49:59.984821 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.984805    2554 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 15:49:59.984821 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.984821    2554 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 15:49:59.984934 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.984829    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 15:49:59.984934 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.984838    2554 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 15:49:59.985561 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.985547    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 15:49:59.985561 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.985571    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 15:49:59.985644 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.985581    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasSufficientPID"
Sep 16 15:49:59.985982 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:49:59.985881    2554 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-228.ec2.internal.18d5d7e3b768253d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-228.ec2.internal,UID:ip-10-0-136-228.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-228.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-228.ec2.internal,},FirstTimestamp:2026-09-16 15:49:59.984244029 +0000 UTC m=+0.428936197,LastTimestamp:2026-09-16 15:49:59.984244029 +0000 UTC m=+0.428936197,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-228.ec2.internal,}"
Sep 16 15:49:59.986871 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.986857    2554 policy_none.go:49] "None policy: Start"
Sep 16 15:49:59.986926 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.986875    2554 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 15:49:59.986926 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.986888    2554 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 15:49:59.988900 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:49:59.988886    2554 policy_none.go:47] "Start"
Sep 16 15:50:00.043093 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.022679    2554 manager.go:337] "Starting Device Plugin manager"
Sep 16 15:50:00.043093 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:00.022717    2554 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 15:50:00.043093 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.022730    2554 server.go:87] "Starting device plugin registration server"
Sep 16 15:50:00.043093 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.023164    2554 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 15:50:00.043093 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.023178    2554 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 15:50:00.043093 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.023267    2554 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 15:50:00.043093 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.023352    2554 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 15:50:00.043093 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.023360    2554 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 15:50:00.043093 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:00.024002    2554 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 15:50:00.043093 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:00.024059    2554 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-228.ec2.internal\" not found"
Sep 16 15:50:00.069623 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.069587    2554 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 15:50:00.070841 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.070823    2554 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 15:50:00.070976 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.070877    2554 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 15:50:00.070976 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.070910    2554 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 15:50:00.071089 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:00.071035    2554 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 15:50:00.073662 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.073640    2554 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 15:50:00.123863 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.123767    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 15:50:00.124751 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.124728    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 15:50:00.124841 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.124759    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 15:50:00.124841 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.124771    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasSufficientPID"
Sep 16 15:50:00.124841 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.124813    2554 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-228.ec2.internal"
Sep 16 15:50:00.134641 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.134616    2554 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-228.ec2.internal"
Sep 16 15:50:00.134641 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:00.134643    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-228.ec2.internal\": node \"ip-10-0-136-228.ec2.internal\" not found"
Sep 16 15:50:00.146341 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:00.146315    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-228.ec2.internal\" not found"
Sep 16 15:50:00.171233 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.171207    2554 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-228.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal"]
Sep 16 15:50:00.171362 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.171283    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 15:50:00.172729 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.172710    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 15:50:00.172857 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.172749    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 15:50:00.172857 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.172764    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasSufficientPID"
Sep 16 15:50:00.175056 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.175041    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 15:50:00.175147 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.175132    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-228.ec2.internal"
Sep 16 15:50:00.175185 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.175165    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 15:50:00.175851 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.175834    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 15:50:00.175851 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.175849    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 15:50:00.175979 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.175866    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 15:50:00.175979 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.175884    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasSufficientPID"
Sep 16 15:50:00.175979 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.175868    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 15:50:00.175979 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.175951    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasSufficientPID"
Sep 16 15:50:00.178116 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.178098    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal"
Sep 16 15:50:00.178219 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.178129    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 15:50:00.178924 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.178908    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 15:50:00.179019 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.178930    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 15:50:00.179019 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.178939    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeHasSufficientPID"
Sep 16 15:50:00.194396 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:00.194369    2554 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-228.ec2.internal\" not found" node="ip-10-0-136-228.ec2.internal"
Sep 16 15:50:00.198930 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:00.198913    2554 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-228.ec2.internal\" not found" node="ip-10-0-136-228.ec2.internal"
Sep 16 15:50:00.246868 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:00.246838    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-228.ec2.internal\" not found"
Sep 16 15:50:00.266139 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.266118    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ce98f1de64686998922578fd3388801-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal\" (UID: \"6ce98f1de64686998922578fd3388801\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal"
Sep 16 15:50:00.266220 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.266144    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c8010e8ff400f9e15027bac559fe6673-config\") pod \"kube-apiserver-proxy-ip-10-0-136-228.ec2.internal\" (UID: \"c8010e8ff400f9e15027bac559fe6673\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-228.ec2.internal"
Sep 16 15:50:00.266220 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.266162    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6ce98f1de64686998922578fd3388801-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal\" (UID: \"6ce98f1de64686998922578fd3388801\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal"
Sep 16 15:50:00.347312 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:00.347267    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-228.ec2.internal\" not found"
Sep 16 15:50:00.366717 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.366686    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6ce98f1de64686998922578fd3388801-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal\" (UID: \"6ce98f1de64686998922578fd3388801\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal"
Sep 16 15:50:00.366789 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.366731    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ce98f1de64686998922578fd3388801-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal\" (UID: \"6ce98f1de64686998922578fd3388801\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal"
Sep 16 15:50:00.366789 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.366749    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c8010e8ff400f9e15027bac559fe6673-config\") pod \"kube-apiserver-proxy-ip-10-0-136-228.ec2.internal\" (UID: \"c8010e8ff400f9e15027bac559fe6673\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-228.ec2.internal"
Sep 16 15:50:00.366789 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.366694    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6ce98f1de64686998922578fd3388801-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal\" (UID: \"6ce98f1de64686998922578fd3388801\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal"
Sep 16 15:50:00.366907 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.366783    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c8010e8ff400f9e15027bac559fe6673-config\") pod \"kube-apiserver-proxy-ip-10-0-136-228.ec2.internal\" (UID: \"c8010e8ff400f9e15027bac559fe6673\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-228.ec2.internal"
Sep 16 15:50:00.366907 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.366857    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ce98f1de64686998922578fd3388801-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal\" (UID: \"6ce98f1de64686998922578fd3388801\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal"
Sep 16 15:50:00.448142 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:00.448055    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-228.ec2.internal\" not found"
Sep 16 15:50:00.497018 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.496982    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-228.ec2.internal"
Sep 16 15:50:00.502073 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.501959    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal"
Sep 16 15:50:00.549808 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:00.549763    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-228.ec2.internal\" not found"
Sep 16 15:50:00.656015 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:00.655976    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-228.ec2.internal\" not found"
Sep 16 15:50:00.757925 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:00.757826    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-228.ec2.internal\" not found"
Sep 16 15:50:00.858729 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:00.858668    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-228.ec2.internal\" not found"
Sep 16 15:50:00.894934 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.894902    2554 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 15:50:00.895416 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.895068    2554 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 15:50:00.959465 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:00.959431    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-228.ec2.internal\" not found"
Sep 16 15:50:00.987322 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.987275    2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 15:45:00 +0000 UTC" deadline="2028-06-11 20:33:04.471301828 +0000 UTC"
Sep 16 15:50:00.987322 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:00.987313    2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15220h43m3.483993072s"
Sep 16 15:50:01.061338 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:01.061293    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-228.ec2.internal\" not found"
Sep 16 15:50:01.101786 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.101750    2554 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 15:50:01.164631 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.164597    2554 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal"
Sep 16 15:50:01.178855 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.178828    2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 15:50:01.181109 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.181079    2554 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-228.ec2.internal"
Sep 16 15:50:01.191605 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.191582    2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 15:50:01.324749 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.324675    2554 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 15:50:01.357522 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:50:01.357485    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8010e8ff400f9e15027bac559fe6673.slice/crio-70cd5fc3c675d86daf31345646890597c3e82f5988e866afcc1f6dfd7d0703b9 WatchSource:0}: Error finding container 70cd5fc3c675d86daf31345646890597c3e82f5988e866afcc1f6dfd7d0703b9: Status 404 returned error can't find the container with id 70cd5fc3c675d86daf31345646890597c3e82f5988e866afcc1f6dfd7d0703b9
Sep 16 15:50:01.358050 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:50:01.358028    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ce98f1de64686998922578fd3388801.slice/crio-434b0c8e2d780d7673f2b6dd1d31b7fe1f466c1724ffa959b876f5dbd43a3122 WatchSource:0}: Error finding container 434b0c8e2d780d7673f2b6dd1d31b7fe1f466c1724ffa959b876f5dbd43a3122: Status 404 returned error can't find the container with id 434b0c8e2d780d7673f2b6dd1d31b7fe1f466c1724ffa959b876f5dbd43a3122
Sep 16 15:50:01.363712 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.363691    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 15:50:01.486631 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.486591    2554 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 15:50:01.906545 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.906518    2554 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 15:50:01.939683 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.939649    2554 apiserver.go:52] "Watching apiserver"
Sep 16 15:50:01.949072 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.949037    2554 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 15:50:01.949432 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.949408    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-gc8gs","kube-system/kube-apiserver-proxy-ip-10-0-136-228.ec2.internal","openshift-cluster-node-tuning-operator/tuned-79xdw","openshift-image-registry/node-ca-fzl79","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal","openshift-multus/multus-7gbkh","openshift-multus/multus-additional-cni-plugins-65c8p","openshift-network-diagnostics/network-check-target-vlhth","kube-system/global-pull-secret-syncer-wcrfk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvv2d","openshift-dns/node-resolver-ffp62","openshift-multus/network-metrics-daemon-wtdsz","openshift-network-operator/iptables-alerter-q9vl8","openshift-ovn-kubernetes/ovnkube-node-w2qgw"]
Sep 16 15:50:01.953549 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.953527    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gc8gs"
Sep 16 15:50:01.955743 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.955719    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ffp62"
Sep 16 15:50:01.956615 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.956593    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 15:50:01.956693 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.956668    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-fmjgz\""
Sep 16 15:50:01.956765 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.956668    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 15:50:01.958144 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.958121    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-65c8p"
Sep 16 15:50:01.958274 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.958255    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 15:50:01.958352 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.958256    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 15:50:01.958409 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.958262    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zdsmg\""
Sep 16 15:50:01.960446 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.960418    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fzl79"
Sep 16 15:50:01.961003 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.960980    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 15:50:01.961093 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.961047    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 15:50:01.961198 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.961180    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 15:50:01.961258 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.961243    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-748rb\""
Sep 16 15:50:01.961445 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.961323    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 15:50:01.961516 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.961460    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 15:50:01.962782 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.962761    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:01.963531 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.963508    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 15:50:01.963648 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.963632    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xjcsb\""
Sep 16 15:50:01.964075 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.964054    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 15:50:01.964180 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.964163    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 15:50:01.964482 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.964468    2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 15:50:01.965273 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.965254    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-79xdw"
Sep 16 15:50:01.965413 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.965397    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-9dmzp\""
Sep 16 15:50:01.965501 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.965485    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 15:50:01.967522 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.967502    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vlhth"
Sep 16 15:50:01.967609 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:01.967589    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vlhth" podUID="25af0988-5e7d-4ee0-bdcb-cc956c6789ef"
Sep 16 15:50:01.968177 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.968068    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 15:50:01.968177 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.968067    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 15:50:01.968177 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.968070    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-n6dwg\""
Sep 16 15:50:01.969760 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.969743    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wcrfk"
Sep 16 15:50:01.969854 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:01.969833    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wcrfk" podUID="c0d31180-c1e0-48f6-a5a2-d1eb9ae02886"
Sep 16 15:50:01.972198 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.972180    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvv2d"
Sep 16 15:50:01.974615 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.974598    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtdsz"
Sep 16 15:50:01.974724 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:01.974661    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtdsz" podUID="b2e625e0-e1fd-4733-97c0-2219660a6700"
Sep 16 15:50:01.975089 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.975052    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 15:50:01.975151 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.975088    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rs9mr\""
Sep 16 15:50:01.975151 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.975074    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 15:50:01.975151 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.975115    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 15:50:01.977076 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.977060    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-q9vl8"
Sep 16 15:50:01.977618 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.977601    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-multus-socket-dir-parent\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:01.977717 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.977632    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f8d2083a-9500-4604-be9f-09278b95a2c6-host\") pod \"tuned-79xdw\" (UID: \"f8d2083a-9500-4604-be9f-09278b95a2c6\") " pod="openshift-cluster-node-tuning-operator/tuned-79xdw"
Sep 16 15:50:01.977717 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.977656    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8d2083a-9500-4604-be9f-09278b95a2c6-tmp\") pod \"tuned-79xdw\" (UID: \"f8d2083a-9500-4604-be9f-09278b95a2c6\") " pod="openshift-cluster-node-tuning-operator/tuned-79xdw"
Sep 16 15:50:01.977837 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.977710    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spmz5\" (UniqueName: \"kubernetes.io/projected/ef023c4c-0a32-416e-8a44-f8e8e4302e5f-kube-api-access-spmz5\") pod \"node-resolver-ffp62\" (UID: \"ef023c4c-0a32-416e-8a44-f8e8e4302e5f\") " pod="openshift-dns/node-resolver-ffp62"
Sep 16 15:50:01.977837 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.977738    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8e64660-b4ae-45ae-b31f-1faee114f212-system-cni-dir\") pod \"multus-additional-cni-plugins-65c8p\" (UID: \"d8e64660-b4ae-45ae-b31f-1faee114f212\") " pod="openshift-multus/multus-additional-cni-plugins-65c8p"
Sep 16 15:50:01.977837 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.977786    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b3420ee-3d02-4cbf-a6b2-fc58c2da0298-host\") pod \"node-ca-fzl79\" (UID: \"7b3420ee-3d02-4cbf-a6b2-fc58c2da0298\") " pod="openshift-image-registry/node-ca-fzl79"
Sep 16 15:50:01.978030 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.977853    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f8d2083a-9500-4604-be9f-09278b95a2c6-etc-sysconfig\") pod \"tuned-79xdw\" (UID: \"f8d2083a-9500-4604-be9f-09278b95a2c6\") " pod="openshift-cluster-node-tuning-operator/tuned-79xdw"
Sep 16 15:50:01.978030 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.977890    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f8d2083a-9500-4604-be9f-09278b95a2c6-sys\") pod \"tuned-79xdw\" (UID: \"f8d2083a-9500-4604-be9f-09278b95a2c6\") " pod="openshift-cluster-node-tuning-operator/tuned-79xdw"
Sep 16 15:50:01.978030 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.977919    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8dgn\" (UniqueName: \"kubernetes.io/projected/d8e64660-b4ae-45ae-b31f-1faee114f212-kube-api-access-q8dgn\") pod \"multus-additional-cni-plugins-65c8p\" (UID: \"d8e64660-b4ae-45ae-b31f-1faee114f212\") " pod="openshift-multus/multus-additional-cni-plugins-65c8p"
Sep 16 15:50:01.978030 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.977967    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-host-run-netns\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:01.978030 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.977999    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f8d2083a-9500-4604-be9f-09278b95a2c6-etc-sysctl-conf\") pod \"tuned-79xdw\" (UID: \"f8d2083a-9500-4604-be9f-09278b95a2c6\") " pod="openshift-cluster-node-tuning-operator/tuned-79xdw"
Sep 16 15:50:01.978030 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.978024    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9e1f1b7b-ac55-41fd-b6d5-1c6257652517-konnectivity-ca\") pod \"konnectivity-agent-gc8gs\" (UID: \"9e1f1b7b-ac55-41fd-b6d5-1c6257652517\") " pod="kube-system/konnectivity-agent-gc8gs"
Sep 16 15:50:01.978255 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.978052    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d8e64660-b4ae-45ae-b31f-1faee114f212-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-65c8p\" (UID: \"d8e64660-b4ae-45ae-b31f-1faee114f212\") " pod="openshift-multus/multus-additional-cni-plugins-65c8p"
Sep 16 15:50:01.978255 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.978072    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f8d2083a-9500-4604-be9f-09278b95a2c6-etc-modprobe-d\") pod \"tuned-79xdw\" (UID: \"f8d2083a-9500-4604-be9f-09278b95a2c6\") " pod="openshift-cluster-node-tuning-operator/tuned-79xdw"
Sep 16 15:50:01.978255 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.978126    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-cni-binary-copy\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:01.978255 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.978146    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-multus-conf-dir\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:01.978255 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.978170    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8e64660-b4ae-45ae-b31f-1faee114f212-cni-binary-copy\") pod \"multus-additional-cni-plugins-65c8p\" (UID: \"d8e64660-b4ae-45ae-b31f-1faee114f212\") " pod="openshift-multus/multus-additional-cni-plugins-65c8p"
Sep 16 15:50:01.978255 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.978195    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dp9w\" (UniqueName: \"kubernetes.io/projected/7b3420ee-3d02-4cbf-a6b2-fc58c2da0298-kube-api-access-5dp9w\") pod \"node-ca-fzl79\" (UID: \"7b3420ee-3d02-4cbf-a6b2-fc58c2da0298\") " pod="openshift-image-registry/node-ca-fzl79"
Sep 16 15:50:01.978255 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.978213    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-host-var-lib-kubelet\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:01.978255 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.978231    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-multus-daemon-config\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:01.978590 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.978269    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-host-run-multus-certs\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:01.978590 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.978290    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7b3420ee-3d02-4cbf-a6b2-fc58c2da0298-serviceca\") pod \"node-ca-fzl79\" (UID: \"7b3420ee-3d02-4cbf-a6b2-fc58c2da0298\") " pod="openshift-image-registry/node-ca-fzl79"
Sep 16 15:50:01.978590 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.978307    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-multus-cni-dir\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:01.978590 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.978327    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f8d2083a-9500-4604-be9f-09278b95a2c6-etc-sysctl-d\") pod \"tuned-79xdw\" (UID: \"f8d2083a-9500-4604-be9f-09278b95a2c6\") " pod="openshift-cluster-node-tuning-operator/tuned-79xdw"
Sep 16 15:50:01.978590 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.978363    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f8d2083a-9500-4604-be9f-09278b95a2c6-run\") pod \"tuned-79xdw\" (UID: \"f8d2083a-9500-4604-be9f-09278b95a2c6\") " pod="openshift-cluster-node-tuning-operator/tuned-79xdw"
Sep 16 15:50:01.978590 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.978379    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m86z\" (UniqueName: \"kubernetes.io/projected/f8d2083a-9500-4604-be9f-09278b95a2c6-kube-api-access-7m86z\") pod \"tuned-79xdw\" (UID: \"f8d2083a-9500-4604-be9f-09278b95a2c6\") " pod="openshift-cluster-node-tuning-operator/tuned-79xdw"
Sep 16 15:50:01.978590 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.978394    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8e64660-b4ae-45ae-b31f-1faee114f212-cnibin\") pod \"multus-additional-cni-plugins-65c8p\" (UID: \"d8e64660-b4ae-45ae-b31f-1faee114f212\") " pod="openshift-multus/multus-additional-cni-plugins-65c8p"
Sep 16 15:50:01.978590 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.978438    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8e64660-b4ae-45ae-b31f-1faee114f212-tuning-conf-dir\") pod \"multus-additional-cni-plugins-65c8p\" (UID: \"d8e64660-b4ae-45ae-b31f-1faee114f212\") " pod="openshift-multus/multus-additional-cni-plugins-65c8p"
Sep 16 15:50:01.978590 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.978477    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-host-var-lib-cni-bin\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:01.978590 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.978504    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-etc-kubernetes\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:01.978590 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.978538    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8d2083a-9500-4604-be9f-09278b95a2c6-etc-kubernetes\") pod \"tuned-79xdw\" (UID: \"f8d2083a-9500-4604-be9f-09278b95a2c6\") " pod="openshift-cluster-node-tuning-operator/tuned-79xdw"
Sep 16 15:50:01.979062 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.978573    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f8d2083a-9500-4604-be9f-09278b95a2c6-etc-systemd\") pod \"tuned-79xdw\" (UID: \"f8d2083a-9500-4604-be9f-09278b95a2c6\") " pod="openshift-cluster-node-tuning-operator/tuned-79xdw"
Sep 16 15:50:01.979100 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.979084    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f8d2083a-9500-4604-be9f-09278b95a2c6-lib-modules\") pod \"tuned-79xdw\" (UID: \"f8d2083a-9500-4604-be9f-09278b95a2c6\") " pod="openshift-cluster-node-tuning-operator/tuned-79xdw"
Sep 16 15:50:01.979134 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.979108    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8d2083a-9500-4604-be9f-09278b95a2c6-var-lib-kubelet\") pod \"tuned-79xdw\" (UID: \"f8d2083a-9500-4604-be9f-09278b95a2c6\") " pod="openshift-cluster-node-tuning-operator/tuned-79xdw"
Sep 16 15:50:01.979190 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.979122    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f8d2083a-9500-4604-be9f-09278b95a2c6-etc-tuned\") pod \"tuned-79xdw\" (UID: \"f8d2083a-9500-4604-be9f-09278b95a2c6\") " pod="openshift-cluster-node-tuning-operator/tuned-79xdw"
Sep 16 15:50:01.979190 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.979179    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ef023c4c-0a32-416e-8a44-f8e8e4302e5f-hosts-file\") pod \"node-resolver-ffp62\" (UID: \"ef023c4c-0a32-416e-8a44-f8e8e4302e5f\") " pod="openshift-dns/node-resolver-ffp62"
Sep 16 15:50:01.979292 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.979211    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-system-cni-dir\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:01.979292 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.979244    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-host-run-k8s-cni-cncf-io\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:01.979292 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.979285    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-host-var-lib-cni-multus\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:01.979442 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.979314    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-hostroot\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:01.979442 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.979341    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6tqd\" (UniqueName: \"kubernetes.io/projected/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-kube-api-access-h6tqd\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:01.979442 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.979367    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9e1f1b7b-ac55-41fd-b6d5-1c6257652517-agent-certs\") pod \"konnectivity-agent-gc8gs\" (UID: \"9e1f1b7b-ac55-41fd-b6d5-1c6257652517\") " pod="kube-system/konnectivity-agent-gc8gs"
Sep 16 15:50:01.979442 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.979404    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef023c4c-0a32-416e-8a44-f8e8e4302e5f-tmp-dir\") pod \"node-resolver-ffp62\" (UID: \"ef023c4c-0a32-416e-8a44-f8e8e4302e5f\") " pod="openshift-dns/node-resolver-ffp62"
Sep 16 15:50:01.979442 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.979429    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d8e64660-b4ae-45ae-b31f-1faee114f212-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-65c8p\" (UID: \"d8e64660-b4ae-45ae-b31f-1faee114f212\") " pod="openshift-multus/multus-additional-cni-plugins-65c8p"
Sep 16 15:50:01.979684 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.979511    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-cnibin\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:01.980069 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.980053    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:01.981617 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.980935    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 15:50:01.981617 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.981155    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fj2tn\""
Sep 16 15:50:01.981617 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.981188    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 15:50:01.981617 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.981206    2554 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 15:50:01.981617 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.981259    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 15:50:01.983012 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.982988    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 15:50:01.983299 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.983280    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6tmhl\""
Sep 16 15:50:01.985135 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.985109    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 15:50:01.985252 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.985146    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 15:50:01.985252 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.985163    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 15:50:01.985252 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.985174    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 15:50:01.985252 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.985167    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 15:50:01.997060 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:01.997035    2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vb6rc"
Sep 16 15:50:02.005142 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.005121    2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vb6rc"
Sep 16 15:50:02.065497 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.065457    2554 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 15:50:02.075675 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.075626    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-228.ec2.internal" event={"ID":"c8010e8ff400f9e15027bac559fe6673","Type":"ContainerStarted","Data":"70cd5fc3c675d86daf31345646890597c3e82f5988e866afcc1f6dfd7d0703b9"}
Sep 16 15:50:02.077146 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.077118    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal" event={"ID":"6ce98f1de64686998922578fd3388801","Type":"ContainerStarted","Data":"434b0c8e2d780d7673f2b6dd1d31b7fe1f466c1724ffa959b876f5dbd43a3122"}
Sep 16 15:50:02.080145 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.080104    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f8d2083a-9500-4604-be9f-09278b95a2c6-etc-sysctl-conf\") pod \"tuned-79xdw\" (UID: \"f8d2083a-9500-4604-be9f-09278b95a2c6\") " pod="openshift-cluster-node-tuning-operator/tuned-79xdw"
Sep 16 15:50:02.080261 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.080159    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9e1f1b7b-ac55-41fd-b6d5-1c6257652517-konnectivity-ca\") pod \"konnectivity-agent-gc8gs\" (UID: \"9e1f1b7b-ac55-41fd-b6d5-1c6257652517\") " pod="kube-system/konnectivity-agent-gc8gs"
Sep 16 15:50:02.080261 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.080189    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8e64660-b4ae-45ae-b31f-1faee114f212-cni-binary-copy\") pod \"multus-additional-cni-plugins-65c8p\" (UID: \"d8e64660-b4ae-45ae-b31f-1faee114f212\") " pod="openshift-multus/multus-additional-cni-plugins-65c8p"
Sep 16 15:50:02.080261 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.080223    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24wbs\" (UniqueName: \"kubernetes.io/projected/c474ab2f-87a0-463c-806d-9d4d977c989d-kube-api-access-24wbs\") pod \"aws-ebs-csi-driver-node-pvv2d\" (UID: \"c474ab2f-87a0-463c-806d-9d4d977c989d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvv2d"
Sep 16 15:50:02.080261 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.080250    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1dd0f6bb-bc9b-4518-bda6-9b12cef15952-iptables-alerter-script\") pod \"iptables-alerter-q9vl8\" (UID: \"1dd0f6bb-bc9b-4518-bda6-9b12cef15952\") " pod="openshift-network-operator/iptables-alerter-q9vl8"
Sep 16 15:50:02.080450 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.080273    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-run-systemd\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.080450 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.080297    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-host-run-ovn-kubernetes\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.080450 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.080338    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f8d2083a-9500-4604-be9f-09278b95a2c6-etc-sysctl-conf\") pod \"tuned-79xdw\" (UID: \"f8d2083a-9500-4604-be9f-09278b95a2c6\") " pod="openshift-cluster-node-tuning-operator/tuned-79xdw"
Sep 16 15:50:02.080450 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.080344    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q45qm\" (UniqueName: \"kubernetes.io/projected/25af0988-5e7d-4ee0-bdcb-cc956c6789ef-kube-api-access-q45qm\") pod \"network-check-target-vlhth\" (UID: \"25af0988-5e7d-4ee0-bdcb-cc956c6789ef\") " pod="openshift-network-diagnostics/network-check-target-vlhth"
Sep 16 15:50:02.080633 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.080447    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-multus-daemon-config\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:02.080633 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.080480    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7b3420ee-3d02-4cbf-a6b2-fc58c2da0298-serviceca\") pod \"node-ca-fzl79\" (UID: \"7b3420ee-3d02-4cbf-a6b2-fc58c2da0298\") " pod="openshift-image-registry/node-ca-fzl79"
Sep 16 15:50:02.080633 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.080506    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m86z\" (UniqueName: \"kubernetes.io/projected/f8d2083a-9500-4604-be9f-09278b95a2c6-kube-api-access-7m86z\") pod \"tuned-79xdw\" (UID: \"f8d2083a-9500-4604-be9f-09278b95a2c6\") " pod="openshift-cluster-node-tuning-operator/tuned-79xdw"
Sep 16 15:50:02.080633 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.080537    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-etc-openvswitch\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.080633 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.080564    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-host-var-lib-cni-bin\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:02.080633 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.080587    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f8d2083a-9500-4604-be9f-09278b95a2c6-etc-tuned\") pod \"tuned-79xdw\" (UID: \"f8d2083a-9500-4604-be9f-09278b95a2c6\") " pod="openshift-cluster-node-tuning-operator/tuned-79xdw"
Sep 16 15:50:02.080633 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.080614    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ef023c4c-0a32-416e-8a44-f8e8e4302e5f-hosts-file\") pod \"node-resolver-ffp62\" (UID: \"ef023c4c-0a32-416e-8a44-f8e8e4302e5f\") " pod="openshift-dns/node-resolver-ffp62"
Sep 16 15:50:02.081105 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.080640    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-system-cni-dir\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:02.081105 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.080668    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-host-run-k8s-cni-cncf-io\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:02.081105 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.080695    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-host-var-lib-cni-multus\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:02.081105 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.080723    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-hostroot\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:02.081105 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.080750    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d8e64660-b4ae-45ae-b31f-1faee114f212-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-65c8p\" (UID: \"d8e64660-b4ae-45ae-b31f-1faee114f212\") " pod="openshift-multus/multus-additional-cni-plugins-65c8p"
Sep 16 15:50:02.081105 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.080780    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1dd0f6bb-bc9b-4518-bda6-9b12cef15952-host-slash\") pod \"iptables-alerter-q9vl8\" (UID: \"1dd0f6bb-bc9b-4518-bda6-9b12cef15952\") " pod="openshift-network-operator/iptables-alerter-q9vl8"
Sep 16 15:50:02.081105 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.080811    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0975e4b2-fcec-4ae9-839c-92b6158b6aca-ovnkube-config\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.081105 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.080869    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9e1f1b7b-ac55-41fd-b6d5-1c6257652517-konnectivity-ca\") pod \"konnectivity-agent-gc8gs\" (UID: \"9e1f1b7b-ac55-41fd-b6d5-1c6257652517\") " pod="kube-system/konnectivity-agent-gc8gs"
Sep 16 15:50:02.081105 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.080890    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z74ls\" (UniqueName: \"kubernetes.io/projected/0975e4b2-fcec-4ae9-839c-92b6158b6aca-kube-api-access-z74ls\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.081105 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.080922    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f8d2083a-9500-4604-be9f-09278b95a2c6-host\") pod \"tuned-79xdw\" (UID: \"f8d2083a-9500-4604-be9f-09278b95a2c6\") " pod="openshift-cluster-node-tuning-operator/tuned-79xdw"
Sep 16 15:50:02.081105 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.080951    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8d2083a-9500-4604-be9f-09278b95a2c6-tmp\") pod \"tuned-79xdw\" (UID: \"f8d2083a-9500-4604-be9f-09278b95a2c6\") " pod="openshift-cluster-node-tuning-operator/tuned-79xdw"
Sep 16 15:50:02.081105 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.080957    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-host-var-lib-cni-multus\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:02.081105 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.080983    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8e64660-b4ae-45ae-b31f-1faee114f212-system-cni-dir\") pod \"multus-additional-cni-plugins-65c8p\" (UID: \"d8e64660-b4ae-45ae-b31f-1faee114f212\") " pod="openshift-multus/multus-additional-cni-plugins-65c8p"
Sep 16 15:50:02.081105 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.080985    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8e64660-b4ae-45ae-b31f-1faee114f212-cni-binary-copy\") pod \"multus-additional-cni-plugins-65c8p\" (UID: \"d8e64660-b4ae-45ae-b31f-1faee114f212\") " pod="openshift-multus/multus-additional-cni-plugins-65c8p"
Sep 16 15:50:02.081105 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.081008    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c474ab2f-87a0-463c-806d-9d4d977c989d-sys-fs\") pod \"aws-ebs-csi-driver-node-pvv2d\" (UID: \"c474ab2f-87a0-463c-806d-9d4d977c989d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvv2d"
Sep 16 15:50:02.081105 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.081016    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-host-run-k8s-cni-cncf-io\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:02.081105 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.081065    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-host-cni-netd\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.081861 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.081066    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-hostroot\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:02.081861 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.081083    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-system-cni-dir\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:02.081861 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.081093    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-host-var-lib-cni-bin\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:02.081861 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.081156    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f8d2083a-9500-4604-be9f-09278b95a2c6-host\") pod \"tuned-79xdw\" (UID: \"f8d2083a-9500-4604-be9f-09278b95a2c6\") " pod="openshift-cluster-node-tuning-operator/tuned-79xdw"
Sep 16 15:50:02.081861 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.081162    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7b3420ee-3d02-4cbf-a6b2-fc58c2da0298-serviceca\") pod \"node-ca-fzl79\" (UID: \"7b3420ee-3d02-4cbf-a6b2-fc58c2da0298\") " pod="openshift-image-registry/node-ca-fzl79"
Sep 16 15:50:02.081861 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.081205    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8e64660-b4ae-45ae-b31f-1faee114f212-system-cni-dir\") pod \"multus-additional-cni-plugins-65c8p\" (UID: \"d8e64660-b4ae-45ae-b31f-1faee114f212\") " pod="openshift-multus/multus-additional-cni-plugins-65c8p"
Sep 16 15:50:02.081861 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.081215    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ef023c4c-0a32-416e-8a44-f8e8e4302e5f-hosts-file\") pod \"node-resolver-ffp62\" (UID: \"ef023c4c-0a32-416e-8a44-f8e8e4302e5f\") " pod="openshift-dns/node-resolver-ffp62"
Sep 16 15:50:02.081861 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.081219    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0975e4b2-fcec-4ae9-839c-92b6158b6aca-ovn-node-metrics-cert\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.081861 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.081260    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0975e4b2-fcec-4ae9-839c-92b6158b6aca-ovnkube-script-lib\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.081861 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.081288    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f8d2083a-9500-4604-be9f-09278b95a2c6-etc-sysconfig\") pod \"tuned-79xdw\" (UID: \"f8d2083a-9500-4604-be9f-09278b95a2c6\") " pod="openshift-cluster-node-tuning-operator/tuned-79xdw"
Sep 16 15:50:02.081861 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.081315    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c474ab2f-87a0-463c-806d-9d4d977c989d-device-dir\") pod \"aws-ebs-csi-driver-node-pvv2d\" (UID: \"c474ab2f-87a0-463c-806d-9d4d977c989d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvv2d"
Sep 16 15:50:02.081861 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.081342    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-node-log\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.081861 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.081335    2554 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 15:50:02.081861 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.081377    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f8d2083a-9500-4604-be9f-09278b95a2c6-etc-sysconfig\") pod \"tuned-79xdw\" (UID: \"f8d2083a-9500-4604-be9f-09278b95a2c6\") " pod="openshift-cluster-node-tuning-operator/tuned-79xdw"
Sep 16 15:50:02.081861 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.081392    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0975e4b2-fcec-4ae9-839c-92b6158b6aca-env-overrides\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.081861 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.081429    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-host-run-netns\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:02.081861 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.081471    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-host-run-netns\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:02.081861 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.081478    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d8e64660-b4ae-45ae-b31f-1faee114f212-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-65c8p\" (UID: \"d8e64660-b4ae-45ae-b31f-1faee114f212\") " pod="openshift-multus/multus-additional-cni-plugins-65c8p"
Sep 16 15:50:02.082738 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.081515    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c474ab2f-87a0-463c-806d-9d4d977c989d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pvv2d\" (UID: \"c474ab2f-87a0-463c-806d-9d4d977c989d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvv2d"
Sep 16 15:50:02.082738 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.081545    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-host-run-netns\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.082738 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.081571    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-run-openvswitch\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.082738 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.081598    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-run-ovn\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.082738 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.081608    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d8e64660-b4ae-45ae-b31f-1faee114f212-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-65c8p\" (UID: \"d8e64660-b4ae-45ae-b31f-1faee114f212\") " pod="openshift-multus/multus-additional-cni-plugins-65c8p"
Sep 16 15:50:02.082738 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.081637    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f8d2083a-9500-4604-be9f-09278b95a2c6-etc-modprobe-d\") pod \"tuned-79xdw\" (UID: \"f8d2083a-9500-4604-be9f-09278b95a2c6\") " pod="openshift-cluster-node-tuning-operator/tuned-79xdw"
Sep 16 15:50:02.082738 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.081691    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-cni-binary-copy\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:02.082738 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.081718    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-multus-conf-dir\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:02.082738 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.081745    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzmbx\" (UniqueName: \"kubernetes.io/projected/1dd0f6bb-bc9b-4518-bda6-9b12cef15952-kube-api-access-dzmbx\") pod \"iptables-alerter-q9vl8\" (UID: \"1dd0f6bb-bc9b-4518-bda6-9b12cef15952\") " pod="openshift-network-operator/iptables-alerter-q9vl8"
Sep 16 15:50:02.082738 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.081770    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-host-cni-bin\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.082738 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.081816    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f8d2083a-9500-4604-be9f-09278b95a2c6-etc-modprobe-d\") pod \"tuned-79xdw\" (UID: \"f8d2083a-9500-4604-be9f-09278b95a2c6\") " pod="openshift-cluster-node-tuning-operator/tuned-79xdw"
Sep 16 15:50:02.082738 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.081829    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.082738 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.081867    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-multus-daemon-config\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:02.082738 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.081877    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-multus-conf-dir\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:02.082738 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.081872    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c0d31180-c1e0-48f6-a5a2-d1eb9ae02886-dbus\") pod \"global-pull-secret-syncer-wcrfk\" (UID: \"c0d31180-c1e0-48f6-a5a2-d1eb9ae02886\") " pod="kube-system/global-pull-secret-syncer-wcrfk"
Sep 16 15:50:02.082738 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.081914    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dp9w\" (UniqueName: \"kubernetes.io/projected/7b3420ee-3d02-4cbf-a6b2-fc58c2da0298-kube-api-access-5dp9w\") pod \"node-ca-fzl79\" (UID: \"7b3420ee-3d02-4cbf-a6b2-fc58c2da0298\") " pod="openshift-image-registry/node-ca-fzl79"
Sep 16 15:50:02.082738 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.081938    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-host-var-lib-kubelet\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:02.083518 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.081949    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d8e64660-b4ae-45ae-b31f-1faee114f212-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-65c8p\" (UID: \"d8e64660-b4ae-45ae-b31f-1faee114f212\") " pod="openshift-multus/multus-additional-cni-plugins-65c8p"
Sep 16 15:50:02.083518 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.081963    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-host-run-multus-certs\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:02.083518 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.082002    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c474ab2f-87a0-463c-806d-9d4d977c989d-registration-dir\") pod \"aws-ebs-csi-driver-node-pvv2d\" (UID: \"c474ab2f-87a0-463c-806d-9d4d977c989d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvv2d"
Sep 16 15:50:02.083518 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.082008    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-host-var-lib-kubelet\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:02.083518 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.082005    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-host-run-multus-certs\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:02.083518 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.082031    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-host-kubelet\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.083518 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.082066    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0d31180-c1e0-48f6-a5a2-d1eb9ae02886-original-pull-secret\") pod \"global-pull-secret-syncer-wcrfk\" (UID: \"c0d31180-c1e0-48f6-a5a2-d1eb9ae02886\") " pod="kube-system/global-pull-secret-syncer-wcrfk"
Sep 16 15:50:02.083518 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.082103    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-multus-cni-dir\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:02.083518 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.082137    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f8d2083a-9500-4604-be9f-09278b95a2c6-etc-sysctl-d\") pod \"tuned-79xdw\" (UID: \"f8d2083a-9500-4604-be9f-09278b95a2c6\") " pod="openshift-cluster-node-tuning-operator/tuned-79xdw"
Sep 16 15:50:02.083518 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.082163    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f8d2083a-9500-4604-be9f-09278b95a2c6-run\") pod \"tuned-79xdw\" (UID: \"f8d2083a-9500-4604-be9f-09278b95a2c6\") " pod="openshift-cluster-node-tuning-operator/tuned-79xdw"
Sep 16 15:50:02.083518 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.082187    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8e64660-b4ae-45ae-b31f-1faee114f212-cnibin\") pod \"multus-additional-cni-plugins-65c8p\" (UID: \"d8e64660-b4ae-45ae-b31f-1faee114f212\") " pod="openshift-multus/multus-additional-cni-plugins-65c8p"
Sep 16 15:50:02.083518 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.082203    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-multus-cni-dir\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:02.083518 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.082219    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8e64660-b4ae-45ae-b31f-1faee114f212-tuning-conf-dir\") pod \"multus-additional-cni-plugins-65c8p\" (UID: \"d8e64660-b4ae-45ae-b31f-1faee114f212\") " pod="openshift-multus/multus-additional-cni-plugins-65c8p"
Sep 16 15:50:02.083518 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.082232    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-cni-binary-copy\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:02.083518 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.082246    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c474ab2f-87a0-463c-806d-9d4d977c989d-etc-selinux\") pod \"aws-ebs-csi-driver-node-pvv2d\" (UID: \"c474ab2f-87a0-463c-806d-9d4d977c989d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvv2d"
Sep 16 15:50:02.083518 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.082268    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8e64660-b4ae-45ae-b31f-1faee114f212-cnibin\") pod \"multus-additional-cni-plugins-65c8p\" (UID: \"d8e64660-b4ae-45ae-b31f-1faee114f212\") " pod="openshift-multus/multus-additional-cni-plugins-65c8p"
Sep 16 15:50:02.083518 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.082265    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f8d2083a-9500-4604-be9f-09278b95a2c6-etc-sysctl-d\") pod \"tuned-79xdw\" (UID: \"f8d2083a-9500-4604-be9f-09278b95a2c6\") " pod="openshift-cluster-node-tuning-operator/tuned-79xdw"
Sep 16 15:50:02.084047 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.082282    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-var-lib-openvswitch\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.084047 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.082316    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-etc-kubernetes\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:02.084047 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.082352    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8d2083a-9500-4604-be9f-09278b95a2c6-etc-kubernetes\") pod \"tuned-79xdw\" (UID: \"f8d2083a-9500-4604-be9f-09278b95a2c6\") " pod="openshift-cluster-node-tuning-operator/tuned-79xdw"
Sep 16 15:50:02.084047 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.082371    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-etc-kubernetes\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:02.084047 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.082373    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8e64660-b4ae-45ae-b31f-1faee114f212-tuning-conf-dir\") pod \"multus-additional-cni-plugins-65c8p\" (UID: \"d8e64660-b4ae-45ae-b31f-1faee114f212\") " pod="openshift-multus/multus-additional-cni-plugins-65c8p"
Sep 16 15:50:02.084047 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.082387    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f8d2083a-9500-4604-be9f-09278b95a2c6-etc-systemd\") pod \"tuned-79xdw\" (UID: \"f8d2083a-9500-4604-be9f-09278b95a2c6\") " pod="openshift-cluster-node-tuning-operator/tuned-79xdw"
Sep 16 15:50:02.084047 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.082401    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f8d2083a-9500-4604-be9f-09278b95a2c6-run\") pod \"tuned-79xdw\" (UID: \"f8d2083a-9500-4604-be9f-09278b95a2c6\") " pod="openshift-cluster-node-tuning-operator/tuned-79xdw"
Sep 16 15:50:02.084047 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.082400    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8d2083a-9500-4604-be9f-09278b95a2c6-etc-kubernetes\") pod \"tuned-79xdw\" (UID: \"f8d2083a-9500-4604-be9f-09278b95a2c6\") " pod="openshift-cluster-node-tuning-operator/tuned-79xdw"
Sep 16 15:50:02.084047 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.082430    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f8d2083a-9500-4604-be9f-09278b95a2c6-etc-systemd\") pod \"tuned-79xdw\" (UID: \"f8d2083a-9500-4604-be9f-09278b95a2c6\") " pod="openshift-cluster-node-tuning-operator/tuned-79xdw"
Sep 16 15:50:02.084047 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.082423    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f8d2083a-9500-4604-be9f-09278b95a2c6-lib-modules\") pod \"tuned-79xdw\" (UID: \"f8d2083a-9500-4604-be9f-09278b95a2c6\") " pod="openshift-cluster-node-tuning-operator/tuned-79xdw"
Sep 16 15:50:02.084047 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.082474    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8d2083a-9500-4604-be9f-09278b95a2c6-var-lib-kubelet\") pod \"tuned-79xdw\" (UID: \"f8d2083a-9500-4604-be9f-09278b95a2c6\") " pod="openshift-cluster-node-tuning-operator/tuned-79xdw"
Sep 16 15:50:02.084047 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.082493    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-log-socket\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.084047 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.082507    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f8d2083a-9500-4604-be9f-09278b95a2c6-lib-modules\") pod \"tuned-79xdw\" (UID: \"f8d2083a-9500-4604-be9f-09278b95a2c6\") " pod="openshift-cluster-node-tuning-operator/tuned-79xdw"
Sep 16 15:50:02.084047 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.082543    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8d2083a-9500-4604-be9f-09278b95a2c6-var-lib-kubelet\") pod \"tuned-79xdw\" (UID: \"f8d2083a-9500-4604-be9f-09278b95a2c6\") " pod="openshift-cluster-node-tuning-operator/tuned-79xdw"
Sep 16 15:50:02.084047 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.082510    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6tqd\" (UniqueName: \"kubernetes.io/projected/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-kube-api-access-h6tqd\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:02.084047 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.082586    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9e1f1b7b-ac55-41fd-b6d5-1c6257652517-agent-certs\") pod \"konnectivity-agent-gc8gs\" (UID: \"9e1f1b7b-ac55-41fd-b6d5-1c6257652517\") " pod="kube-system/konnectivity-agent-gc8gs"
Sep 16 15:50:02.084047 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.082602    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef023c4c-0a32-416e-8a44-f8e8e4302e5f-tmp-dir\") pod \"node-resolver-ffp62\" (UID: \"ef023c4c-0a32-416e-8a44-f8e8e4302e5f\") " pod="openshift-dns/node-resolver-ffp62"
Sep 16 15:50:02.084047 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.082627    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c474ab2f-87a0-463c-806d-9d4d977c989d-socket-dir\") pod \"aws-ebs-csi-driver-node-pvv2d\" (UID: \"c474ab2f-87a0-463c-806d-9d4d977c989d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvv2d"
Sep 16 15:50:02.084579 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.082646    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-systemd-units\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.084579 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.082669    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-cnibin\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:02.084579 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.082694    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-multus-socket-dir-parent\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:02.084579 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.082718    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spmz5\" (UniqueName: \"kubernetes.io/projected/ef023c4c-0a32-416e-8a44-f8e8e4302e5f-kube-api-access-spmz5\") pod \"node-resolver-ffp62\" (UID: \"ef023c4c-0a32-416e-8a44-f8e8e4302e5f\") " pod="openshift-dns/node-resolver-ffp62"
Sep 16 15:50:02.084579 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.082740    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2e625e0-e1fd-4733-97c0-2219660a6700-metrics-certs\") pod \"network-metrics-daemon-wtdsz\" (UID: \"b2e625e0-e1fd-4733-97c0-2219660a6700\") " pod="openshift-multus/network-metrics-daemon-wtdsz"
Sep 16 15:50:02.084579 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.082744    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-cnibin\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:02.084579 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.082776    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-host-slash\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.084579 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.082780    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-multus-socket-dir-parent\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:02.084579 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.082822    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c0d31180-c1e0-48f6-a5a2-d1eb9ae02886-kubelet-config\") pod \"global-pull-secret-syncer-wcrfk\" (UID: \"c0d31180-c1e0-48f6-a5a2-d1eb9ae02886\") " pod="kube-system/global-pull-secret-syncer-wcrfk"
Sep 16 15:50:02.084579 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.082964    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b3420ee-3d02-4cbf-a6b2-fc58c2da0298-host\") pod \"node-ca-fzl79\" (UID: \"7b3420ee-3d02-4cbf-a6b2-fc58c2da0298\") " pod="openshift-image-registry/node-ca-fzl79"
Sep 16 15:50:02.084579 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.083006    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f8d2083a-9500-4604-be9f-09278b95a2c6-sys\") pod \"tuned-79xdw\" (UID: \"f8d2083a-9500-4604-be9f-09278b95a2c6\") " pod="openshift-cluster-node-tuning-operator/tuned-79xdw"
Sep 16 15:50:02.084579 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.083023    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef023c4c-0a32-416e-8a44-f8e8e4302e5f-tmp-dir\") pod \"node-resolver-ffp62\" (UID: \"ef023c4c-0a32-416e-8a44-f8e8e4302e5f\") " pod="openshift-dns/node-resolver-ffp62"
Sep 16 15:50:02.084579 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.083057    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b3420ee-3d02-4cbf-a6b2-fc58c2da0298-host\") pod \"node-ca-fzl79\" (UID: \"7b3420ee-3d02-4cbf-a6b2-fc58c2da0298\") " pod="openshift-image-registry/node-ca-fzl79"
Sep 16 15:50:02.084579 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.083097    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8dgn\" (UniqueName: \"kubernetes.io/projected/d8e64660-b4ae-45ae-b31f-1faee114f212-kube-api-access-q8dgn\") pod \"multus-additional-cni-plugins-65c8p\" (UID: \"d8e64660-b4ae-45ae-b31f-1faee114f212\") " pod="openshift-multus/multus-additional-cni-plugins-65c8p"
Sep 16 15:50:02.084579 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.083116    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f8d2083a-9500-4604-be9f-09278b95a2c6-sys\") pod \"tuned-79xdw\" (UID: \"f8d2083a-9500-4604-be9f-09278b95a2c6\") " pod="openshift-cluster-node-tuning-operator/tuned-79xdw"
Sep 16 15:50:02.084579 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.083127    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fgn7\" (UniqueName: \"kubernetes.io/projected/b2e625e0-e1fd-4733-97c0-2219660a6700-kube-api-access-5fgn7\") pod \"network-metrics-daemon-wtdsz\" (UID: \"b2e625e0-e1fd-4733-97c0-2219660a6700\") " pod="openshift-multus/network-metrics-daemon-wtdsz"
Sep 16 15:50:02.085306 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.085281    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f8d2083a-9500-4604-be9f-09278b95a2c6-etc-tuned\") pod \"tuned-79xdw\" (UID: \"f8d2083a-9500-4604-be9f-09278b95a2c6\") " pod="openshift-cluster-node-tuning-operator/tuned-79xdw"
Sep 16 15:50:02.085392 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.085314    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8d2083a-9500-4604-be9f-09278b95a2c6-tmp\") pod \"tuned-79xdw\" (UID: \"f8d2083a-9500-4604-be9f-09278b95a2c6\") " pod="openshift-cluster-node-tuning-operator/tuned-79xdw"
Sep 16 15:50:02.085433 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.085380    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9e1f1b7b-ac55-41fd-b6d5-1c6257652517-agent-certs\") pod \"konnectivity-agent-gc8gs\" (UID: \"9e1f1b7b-ac55-41fd-b6d5-1c6257652517\") " pod="kube-system/konnectivity-agent-gc8gs"
Sep 16 15:50:02.098975 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.098951    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dp9w\" (UniqueName: \"kubernetes.io/projected/7b3420ee-3d02-4cbf-a6b2-fc58c2da0298-kube-api-access-5dp9w\") pod \"node-ca-fzl79\" (UID: \"7b3420ee-3d02-4cbf-a6b2-fc58c2da0298\") " pod="openshift-image-registry/node-ca-fzl79"
Sep 16 15:50:02.099364 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.099339    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m86z\" (UniqueName: \"kubernetes.io/projected/f8d2083a-9500-4604-be9f-09278b95a2c6-kube-api-access-7m86z\") pod \"tuned-79xdw\" (UID: \"f8d2083a-9500-4604-be9f-09278b95a2c6\") " pod="openshift-cluster-node-tuning-operator/tuned-79xdw"
Sep 16 15:50:02.099492 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.099477    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6tqd\" (UniqueName: \"kubernetes.io/projected/c30e3d8c-1676-44b8-9a25-fce32e24c7bc-kube-api-access-h6tqd\") pod \"multus-7gbkh\" (UID: \"c30e3d8c-1676-44b8-9a25-fce32e24c7bc\") " pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:02.099971 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.099954    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spmz5\" (UniqueName: \"kubernetes.io/projected/ef023c4c-0a32-416e-8a44-f8e8e4302e5f-kube-api-access-spmz5\") pod \"node-resolver-ffp62\" (UID: \"ef023c4c-0a32-416e-8a44-f8e8e4302e5f\") " pod="openshift-dns/node-resolver-ffp62"
Sep 16 15:50:02.100020 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.099992    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8dgn\" (UniqueName: \"kubernetes.io/projected/d8e64660-b4ae-45ae-b31f-1faee114f212-kube-api-access-q8dgn\") pod \"multus-additional-cni-plugins-65c8p\" (UID: \"d8e64660-b4ae-45ae-b31f-1faee114f212\") " pod="openshift-multus/multus-additional-cni-plugins-65c8p"
Sep 16 15:50:02.183542 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.183444    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c474ab2f-87a0-463c-806d-9d4d977c989d-socket-dir\") pod \"aws-ebs-csi-driver-node-pvv2d\" (UID: \"c474ab2f-87a0-463c-806d-9d4d977c989d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvv2d"
Sep 16 15:50:02.183542 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.183479    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-systemd-units\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.183796 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.183562    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-systemd-units\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.183796 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.183588    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2e625e0-e1fd-4733-97c0-2219660a6700-metrics-certs\") pod \"network-metrics-daemon-wtdsz\" (UID: \"b2e625e0-e1fd-4733-97c0-2219660a6700\") " pod="openshift-multus/network-metrics-daemon-wtdsz"
Sep 16 15:50:02.183796 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.183611    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-host-slash\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.183796 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.183636    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c0d31180-c1e0-48f6-a5a2-d1eb9ae02886-kubelet-config\") pod \"global-pull-secret-syncer-wcrfk\" (UID: \"c0d31180-c1e0-48f6-a5a2-d1eb9ae02886\") " pod="kube-system/global-pull-secret-syncer-wcrfk"
Sep 16 15:50:02.183796 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.183650    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c474ab2f-87a0-463c-806d-9d4d977c989d-socket-dir\") pod \"aws-ebs-csi-driver-node-pvv2d\" (UID: \"c474ab2f-87a0-463c-806d-9d4d977c989d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvv2d"
Sep 16 15:50:02.183796 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.183663    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fgn7\" (UniqueName: \"kubernetes.io/projected/b2e625e0-e1fd-4733-97c0-2219660a6700-kube-api-access-5fgn7\") pod \"network-metrics-daemon-wtdsz\" (UID: \"b2e625e0-e1fd-4733-97c0-2219660a6700\") " pod="openshift-multus/network-metrics-daemon-wtdsz"
Sep 16 15:50:02.183796 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.183692    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-host-slash\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.183796 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.183709    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24wbs\" (UniqueName: \"kubernetes.io/projected/c474ab2f-87a0-463c-806d-9d4d977c989d-kube-api-access-24wbs\") pod \"aws-ebs-csi-driver-node-pvv2d\" (UID: \"c474ab2f-87a0-463c-806d-9d4d977c989d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvv2d"
Sep 16 15:50:02.183796 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:02.183715    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 15:50:02.183796 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.183738    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1dd0f6bb-bc9b-4518-bda6-9b12cef15952-iptables-alerter-script\") pod \"iptables-alerter-q9vl8\" (UID: \"1dd0f6bb-bc9b-4518-bda6-9b12cef15952\") " pod="openshift-network-operator/iptables-alerter-q9vl8"
Sep 16 15:50:02.183796 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.183759    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-run-systemd\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.183796 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.183752    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c0d31180-c1e0-48f6-a5a2-d1eb9ae02886-kubelet-config\") pod \"global-pull-secret-syncer-wcrfk\" (UID: \"c0d31180-c1e0-48f6-a5a2-d1eb9ae02886\") " pod="kube-system/global-pull-secret-syncer-wcrfk"
Sep 16 15:50:02.183796 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:02.183788    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2e625e0-e1fd-4733-97c0-2219660a6700-metrics-certs podName:b2e625e0-e1fd-4733-97c0-2219660a6700 nodeName:}" failed. No retries permitted until 2026-09-16 15:50:02.683764614 +0000 UTC m=+3.128456794 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b2e625e0-e1fd-4733-97c0-2219660a6700-metrics-certs") pod "network-metrics-daemon-wtdsz" (UID: "b2e625e0-e1fd-4733-97c0-2219660a6700") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 15:50:02.184501 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.183806    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-run-systemd\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.184501 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.183839    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-host-run-ovn-kubernetes\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.184501 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.183868    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q45qm\" (UniqueName: \"kubernetes.io/projected/25af0988-5e7d-4ee0-bdcb-cc956c6789ef-kube-api-access-q45qm\") pod \"network-check-target-vlhth\" (UID: \"25af0988-5e7d-4ee0-bdcb-cc956c6789ef\") " pod="openshift-network-diagnostics/network-check-target-vlhth"
Sep 16 15:50:02.184501 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.183896    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-etc-openvswitch\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.184501 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.183926    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1dd0f6bb-bc9b-4518-bda6-9b12cef15952-host-slash\") pod \"iptables-alerter-q9vl8\" (UID: \"1dd0f6bb-bc9b-4518-bda6-9b12cef15952\") " pod="openshift-network-operator/iptables-alerter-q9vl8"
Sep 16 15:50:02.184501 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.183950    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0975e4b2-fcec-4ae9-839c-92b6158b6aca-ovnkube-config\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.184501 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.183978    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z74ls\" (UniqueName: \"kubernetes.io/projected/0975e4b2-fcec-4ae9-839c-92b6158b6aca-kube-api-access-z74ls\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.184501 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.183984    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-etc-openvswitch\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.184501 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.184007    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c474ab2f-87a0-463c-806d-9d4d977c989d-sys-fs\") pod \"aws-ebs-csi-driver-node-pvv2d\" (UID: \"c474ab2f-87a0-463c-806d-9d4d977c989d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvv2d"
Sep 16 15:50:02.184501 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.184031    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-host-cni-netd\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.184501 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.184056    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0975e4b2-fcec-4ae9-839c-92b6158b6aca-ovn-node-metrics-cert\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.184501 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.184066    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-host-run-ovn-kubernetes\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.184501 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.184081    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0975e4b2-fcec-4ae9-839c-92b6158b6aca-ovnkube-script-lib\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.184501 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.184107    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c474ab2f-87a0-463c-806d-9d4d977c989d-device-dir\") pod \"aws-ebs-csi-driver-node-pvv2d\" (UID: \"c474ab2f-87a0-463c-806d-9d4d977c989d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvv2d"
Sep 16 15:50:02.184501 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.184122    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-host-cni-netd\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.184501 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.184131    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-node-log\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.184501 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.184164    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c474ab2f-87a0-463c-806d-9d4d977c989d-sys-fs\") pod \"aws-ebs-csi-driver-node-pvv2d\" (UID: \"c474ab2f-87a0-463c-806d-9d4d977c989d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvv2d"
Sep 16 15:50:02.185272 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.184197    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c474ab2f-87a0-463c-806d-9d4d977c989d-device-dir\") pod \"aws-ebs-csi-driver-node-pvv2d\" (UID: \"c474ab2f-87a0-463c-806d-9d4d977c989d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvv2d"
Sep 16 15:50:02.185272 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.184190    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-node-log\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.185272 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.184209    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0975e4b2-fcec-4ae9-839c-92b6158b6aca-env-overrides\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.185272 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.184265    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1dd0f6bb-bc9b-4518-bda6-9b12cef15952-host-slash\") pod \"iptables-alerter-q9vl8\" (UID: \"1dd0f6bb-bc9b-4518-bda6-9b12cef15952\") " pod="openshift-network-operator/iptables-alerter-q9vl8"
Sep 16 15:50:02.185272 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.184265    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c474ab2f-87a0-463c-806d-9d4d977c989d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pvv2d\" (UID: \"c474ab2f-87a0-463c-806d-9d4d977c989d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvv2d"
Sep 16 15:50:02.185272 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.184298    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-host-run-netns\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.185272 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.184316    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c474ab2f-87a0-463c-806d-9d4d977c989d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pvv2d\" (UID: \"c474ab2f-87a0-463c-806d-9d4d977c989d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvv2d"
Sep 16 15:50:02.185272 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.184321    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-run-openvswitch\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.185272 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.184353    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-run-openvswitch\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.185272 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.184360    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-run-ovn\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.185272 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.184394    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzmbx\" (UniqueName: \"kubernetes.io/projected/1dd0f6bb-bc9b-4518-bda6-9b12cef15952-kube-api-access-dzmbx\") pod \"iptables-alerter-q9vl8\" (UID: \"1dd0f6bb-bc9b-4518-bda6-9b12cef15952\") " pod="openshift-network-operator/iptables-alerter-q9vl8"
Sep 16 15:50:02.185272 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.184423    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-host-cni-bin\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.185272 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.184452    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.185272 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.184468    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-run-ovn\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.185272 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.184481    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c0d31180-c1e0-48f6-a5a2-d1eb9ae02886-dbus\") pod \"global-pull-secret-syncer-wcrfk\" (UID: \"c0d31180-c1e0-48f6-a5a2-d1eb9ae02886\") " pod="kube-system/global-pull-secret-syncer-wcrfk"
Sep 16 15:50:02.185272 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.184510    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c474ab2f-87a0-463c-806d-9d4d977c989d-registration-dir\") pod \"aws-ebs-csi-driver-node-pvv2d\" (UID: \"c474ab2f-87a0-463c-806d-9d4d977c989d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvv2d"
Sep 16 15:50:02.185272 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.184523    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1dd0f6bb-bc9b-4518-bda6-9b12cef15952-iptables-alerter-script\") pod \"iptables-alerter-q9vl8\" (UID: \"1dd0f6bb-bc9b-4518-bda6-9b12cef15952\") " pod="openshift-network-operator/iptables-alerter-q9vl8"
Sep 16 15:50:02.185968 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.184534    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-host-kubelet\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.185968 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.184562    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0d31180-c1e0-48f6-a5a2-d1eb9ae02886-original-pull-secret\") pod \"global-pull-secret-syncer-wcrfk\" (UID: \"c0d31180-c1e0-48f6-a5a2-d1eb9ae02886\") " pod="kube-system/global-pull-secret-syncer-wcrfk"
Sep 16 15:50:02.185968 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.184594    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c474ab2f-87a0-463c-806d-9d4d977c989d-etc-selinux\") pod \"aws-ebs-csi-driver-node-pvv2d\" (UID: \"c474ab2f-87a0-463c-806d-9d4d977c989d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvv2d"
Sep 16 15:50:02.185968 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.184597    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.185968 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.184624    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-var-lib-openvswitch\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.185968 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.184658    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-log-socket\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.185968 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.184726    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-log-socket\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.185968 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.184780    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0975e4b2-fcec-4ae9-839c-92b6158b6aca-env-overrides\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.185968 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.184785    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c0d31180-c1e0-48f6-a5a2-d1eb9ae02886-dbus\") pod \"global-pull-secret-syncer-wcrfk\" (UID: \"c0d31180-c1e0-48f6-a5a2-d1eb9ae02886\") " pod="kube-system/global-pull-secret-syncer-wcrfk"
Sep 16 15:50:02.185968 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.184836    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c474ab2f-87a0-463c-806d-9d4d977c989d-registration-dir\") pod \"aws-ebs-csi-driver-node-pvv2d\" (UID: \"c474ab2f-87a0-463c-806d-9d4d977c989d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvv2d"
Sep 16 15:50:02.185968 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.184862    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-host-cni-bin\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.185968 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.184900    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0975e4b2-fcec-4ae9-839c-92b6158b6aca-ovnkube-script-lib\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.185968 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.184905    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0975e4b2-fcec-4ae9-839c-92b6158b6aca-ovnkube-config\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.185968 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.184949    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-host-kubelet\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.185968 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.184988    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-var-lib-openvswitch\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.185968 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:02.184995    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 15:50:02.185968 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.184995    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c474ab2f-87a0-463c-806d-9d4d977c989d-etc-selinux\") pod \"aws-ebs-csi-driver-node-pvv2d\" (UID: \"c474ab2f-87a0-463c-806d-9d4d977c989d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvv2d"
Sep 16 15:50:02.185968 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.185043    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0975e4b2-fcec-4ae9-839c-92b6158b6aca-host-run-netns\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.186834 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:02.185069    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0d31180-c1e0-48f6-a5a2-d1eb9ae02886-original-pull-secret podName:c0d31180-c1e0-48f6-a5a2-d1eb9ae02886 nodeName:}" failed. No retries permitted until 2026-09-16 15:50:02.685050362 +0000 UTC m=+3.129742536 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c0d31180-c1e0-48f6-a5a2-d1eb9ae02886-original-pull-secret") pod "global-pull-secret-syncer-wcrfk" (UID: "c0d31180-c1e0-48f6-a5a2-d1eb9ae02886") : object "kube-system"/"original-pull-secret" not registered
Sep 16 15:50:02.186834 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.186789    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0975e4b2-fcec-4ae9-839c-92b6158b6aca-ovn-node-metrics-cert\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.196587 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:02.196551    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 15:50:02.196587 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:02.196580    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 15:50:02.196786 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:02.196596    2554 projected.go:196] Error preparing data for projected volume kube-api-access-q45qm for pod openshift-network-diagnostics/network-check-target-vlhth: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 15:50:02.196786 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:02.196705    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25af0988-5e7d-4ee0-bdcb-cc956c6789ef-kube-api-access-q45qm podName:25af0988-5e7d-4ee0-bdcb-cc956c6789ef nodeName:}" failed. No retries permitted until 2026-09-16 15:50:02.696684825 +0000 UTC m=+3.141377004 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-q45qm" (UniqueName: "kubernetes.io/projected/25af0988-5e7d-4ee0-bdcb-cc956c6789ef-kube-api-access-q45qm") pod "network-check-target-vlhth" (UID: "25af0988-5e7d-4ee0-bdcb-cc956c6789ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 15:50:02.197819 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.197775    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z74ls\" (UniqueName: \"kubernetes.io/projected/0975e4b2-fcec-4ae9-839c-92b6158b6aca-kube-api-access-z74ls\") pod \"ovnkube-node-w2qgw\" (UID: \"0975e4b2-fcec-4ae9-839c-92b6158b6aca\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.198034 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.198012    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fgn7\" (UniqueName: \"kubernetes.io/projected/b2e625e0-e1fd-4733-97c0-2219660a6700-kube-api-access-5fgn7\") pod \"network-metrics-daemon-wtdsz\" (UID: \"b2e625e0-e1fd-4733-97c0-2219660a6700\") " pod="openshift-multus/network-metrics-daemon-wtdsz"
Sep 16 15:50:02.198096 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.198063    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24wbs\" (UniqueName: \"kubernetes.io/projected/c474ab2f-87a0-463c-806d-9d4d977c989d-kube-api-access-24wbs\") pod \"aws-ebs-csi-driver-node-pvv2d\" (UID: \"c474ab2f-87a0-463c-806d-9d4d977c989d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvv2d"
Sep 16 15:50:02.198136 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.198062    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzmbx\" (UniqueName: \"kubernetes.io/projected/1dd0f6bb-bc9b-4518-bda6-9b12cef15952-kube-api-access-dzmbx\") pod \"iptables-alerter-q9vl8\" (UID: \"1dd0f6bb-bc9b-4518-bda6-9b12cef15952\") " pod="openshift-network-operator/iptables-alerter-q9vl8"
Sep 16 15:50:02.266885 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.266839    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gc8gs"
Sep 16 15:50:02.271163 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.271140    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ffp62"
Sep 16 15:50:02.277273 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.277251    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-65c8p"
Sep 16 15:50:02.278889 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:50:02.278852    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e1f1b7b_ac55_41fd_b6d5_1c6257652517.slice/crio-3247a8542f6121041e7d0c6ab2de2da3deb1f552ac88e93f1437458aea0c6748 WatchSource:0}: Error finding container 3247a8542f6121041e7d0c6ab2de2da3deb1f552ac88e93f1437458aea0c6748: Status 404 returned error can't find the container with id 3247a8542f6121041e7d0c6ab2de2da3deb1f552ac88e93f1437458aea0c6748
Sep 16 15:50:02.282493 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:50:02.282468    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef023c4c_0a32_416e_8a44_f8e8e4302e5f.slice/crio-3a114150785315abec3f22392b2bc0e48c1fc95d3823d66734c2a5e275931e34 WatchSource:0}: Error finding container 3a114150785315abec3f22392b2bc0e48c1fc95d3823d66734c2a5e275931e34: Status 404 returned error can't find the container with id 3a114150785315abec3f22392b2bc0e48c1fc95d3823d66734c2a5e275931e34
Sep 16 15:50:02.285641 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.285624    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fzl79"
Sep 16 15:50:02.288821 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:50:02.288797    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8e64660_b4ae_45ae_b31f_1faee114f212.slice/crio-7b10f70236272fa9ca8c561392fc188f5d3517011a47f69033be9e4103427788 WatchSource:0}: Error finding container 7b10f70236272fa9ca8c561392fc188f5d3517011a47f69033be9e4103427788: Status 404 returned error can't find the container with id 7b10f70236272fa9ca8c561392fc188f5d3517011a47f69033be9e4103427788
Sep 16 15:50:02.292286 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.292267    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7gbkh"
Sep 16 15:50:02.295771 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:50:02.295747    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b3420ee_3d02_4cbf_a6b2_fc58c2da0298.slice/crio-6060120d41cadf767af19cc85a7add5259355d599f498b8449304adae575b40e WatchSource:0}: Error finding container 6060120d41cadf767af19cc85a7add5259355d599f498b8449304adae575b40e: Status 404 returned error can't find the container with id 6060120d41cadf767af19cc85a7add5259355d599f498b8449304adae575b40e
Sep 16 15:50:02.297397 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.297379    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-79xdw"
Sep 16 15:50:02.302756 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:50:02.302588    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc30e3d8c_1676_44b8_9a25_fce32e24c7bc.slice/crio-6c4577a0d2a100f0f789af7057c4c1027d9b2204a2c827b62aafc5e003cd607c WatchSource:0}: Error finding container 6c4577a0d2a100f0f789af7057c4c1027d9b2204a2c827b62aafc5e003cd607c: Status 404 returned error can't find the container with id 6c4577a0d2a100f0f789af7057c4c1027d9b2204a2c827b62aafc5e003cd607c
Sep 16 15:50:02.303273 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.303255    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvv2d"
Sep 16 15:50:02.307036 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:50:02.307016    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8d2083a_9500_4604_be9f_09278b95a2c6.slice/crio-50781a786a0be04c553e58e1012541ffc3a0c119023a18e6d82fcc0fa1336315 WatchSource:0}: Error finding container 50781a786a0be04c553e58e1012541ffc3a0c119023a18e6d82fcc0fa1336315: Status 404 returned error can't find the container with id 50781a786a0be04c553e58e1012541ffc3a0c119023a18e6d82fcc0fa1336315
Sep 16 15:50:02.310764 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.310743    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-q9vl8"
Sep 16 15:50:02.312896 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:50:02.312853    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc474ab2f_87a0_463c_806d_9d4d977c989d.slice/crio-31c036d26019267a5738305cf4ebcbc218d06a376deb03575eece2eea69c69f2 WatchSource:0}: Error finding container 31c036d26019267a5738305cf4ebcbc218d06a376deb03575eece2eea69c69f2: Status 404 returned error can't find the container with id 31c036d26019267a5738305cf4ebcbc218d06a376deb03575eece2eea69c69f2
Sep 16 15:50:02.314811 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.314648    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:02.321870 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:50:02.321838    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dd0f6bb_bc9b_4518_bda6_9b12cef15952.slice/crio-1597de985dca4bd630a14c23f495c16e2519fa60b64378cba05d303b2965649c WatchSource:0}: Error finding container 1597de985dca4bd630a14c23f495c16e2519fa60b64378cba05d303b2965649c: Status 404 returned error can't find the container with id 1597de985dca4bd630a14c23f495c16e2519fa60b64378cba05d303b2965649c
Sep 16 15:50:02.326382 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:50:02.326360    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0975e4b2_fcec_4ae9_839c_92b6158b6aca.slice/crio-5e83d881cc391b9c9acfdd445e211ce09bc4442cbdece5d9fcd9d31b43794b20 WatchSource:0}: Error finding container 5e83d881cc391b9c9acfdd445e211ce09bc4442cbdece5d9fcd9d31b43794b20: Status 404 returned error can't find the container with id 5e83d881cc391b9c9acfdd445e211ce09bc4442cbdece5d9fcd9d31b43794b20
Sep 16 15:50:02.689341 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.689305    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0d31180-c1e0-48f6-a5a2-d1eb9ae02886-original-pull-secret\") pod \"global-pull-secret-syncer-wcrfk\" (UID: \"c0d31180-c1e0-48f6-a5a2-d1eb9ae02886\") " pod="kube-system/global-pull-secret-syncer-wcrfk"
Sep 16 15:50:02.689548 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.689364    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2e625e0-e1fd-4733-97c0-2219660a6700-metrics-certs\") pod \"network-metrics-daemon-wtdsz\" (UID: \"b2e625e0-e1fd-4733-97c0-2219660a6700\") " pod="openshift-multus/network-metrics-daemon-wtdsz"
Sep 16 15:50:02.689548 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:02.689467    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 15:50:02.689664 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:02.689566    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0d31180-c1e0-48f6-a5a2-d1eb9ae02886-original-pull-secret podName:c0d31180-c1e0-48f6-a5a2-d1eb9ae02886 nodeName:}" failed. No retries permitted until 2026-09-16 15:50:03.689541264 +0000 UTC m=+4.134233437 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c0d31180-c1e0-48f6-a5a2-d1eb9ae02886-original-pull-secret") pod "global-pull-secret-syncer-wcrfk" (UID: "c0d31180-c1e0-48f6-a5a2-d1eb9ae02886") : object "kube-system"/"original-pull-secret" not registered
Sep 16 15:50:02.689664 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:02.689474    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 15:50:02.689757 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:02.689666    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2e625e0-e1fd-4733-97c0-2219660a6700-metrics-certs podName:b2e625e0-e1fd-4733-97c0-2219660a6700 nodeName:}" failed. No retries permitted until 2026-09-16 15:50:03.689644395 +0000 UTC m=+4.134336573 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b2e625e0-e1fd-4733-97c0-2219660a6700-metrics-certs") pod "network-metrics-daemon-wtdsz" (UID: "b2e625e0-e1fd-4733-97c0-2219660a6700") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 15:50:02.790074 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:02.790035    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q45qm\" (UniqueName: \"kubernetes.io/projected/25af0988-5e7d-4ee0-bdcb-cc956c6789ef-kube-api-access-q45qm\") pod \"network-check-target-vlhth\" (UID: \"25af0988-5e7d-4ee0-bdcb-cc956c6789ef\") " pod="openshift-network-diagnostics/network-check-target-vlhth"
Sep 16 15:50:02.790292 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:02.790202    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 15:50:02.790292 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:02.790222    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 15:50:02.790292 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:02.790235    2554 projected.go:196] Error preparing data for projected volume kube-api-access-q45qm for pod openshift-network-diagnostics/network-check-target-vlhth: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 15:50:02.790292 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:02.790292    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25af0988-5e7d-4ee0-bdcb-cc956c6789ef-kube-api-access-q45qm podName:25af0988-5e7d-4ee0-bdcb-cc956c6789ef nodeName:}" failed. No retries permitted until 2026-09-16 15:50:03.790273744 +0000 UTC m=+4.234965916 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-q45qm" (UniqueName: "kubernetes.io/projected/25af0988-5e7d-4ee0-bdcb-cc956c6789ef-kube-api-access-q45qm") pod "network-check-target-vlhth" (UID: "25af0988-5e7d-4ee0-bdcb-cc956c6789ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 15:50:03.009246 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:03.009149    2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 15:45:02 +0000 UTC" deadline="2028-07-01 20:23:04.533556148 +0000 UTC"
Sep 16 15:50:03.009246 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:03.009193    2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15700h33m1.52436731s"
Sep 16 15:50:03.072226 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:03.072186    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtdsz"
Sep 16 15:50:03.072693 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:03.072525    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtdsz" podUID="b2e625e0-e1fd-4733-97c0-2219660a6700"
Sep 16 15:50:03.085776 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:03.085738    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7gbkh" event={"ID":"c30e3d8c-1676-44b8-9a25-fce32e24c7bc","Type":"ContainerStarted","Data":"6c4577a0d2a100f0f789af7057c4c1027d9b2204a2c827b62aafc5e003cd607c"}
Sep 16 15:50:03.090960 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:03.090926    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ffp62" event={"ID":"ef023c4c-0a32-416e-8a44-f8e8e4302e5f","Type":"ContainerStarted","Data":"3a114150785315abec3f22392b2bc0e48c1fc95d3823d66734c2a5e275931e34"}
Sep 16 15:50:03.097184 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:03.097148    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gc8gs" event={"ID":"9e1f1b7b-ac55-41fd-b6d5-1c6257652517","Type":"ContainerStarted","Data":"3247a8542f6121041e7d0c6ab2de2da3deb1f552ac88e93f1437458aea0c6748"}
Sep 16 15:50:03.109478 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:03.109422    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw" event={"ID":"0975e4b2-fcec-4ae9-839c-92b6158b6aca","Type":"ContainerStarted","Data":"5e83d881cc391b9c9acfdd445e211ce09bc4442cbdece5d9fcd9d31b43794b20"}
Sep 16 15:50:03.122748 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:03.122708    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvv2d" event={"ID":"c474ab2f-87a0-463c-806d-9d4d977c989d","Type":"ContainerStarted","Data":"31c036d26019267a5738305cf4ebcbc218d06a376deb03575eece2eea69c69f2"}
Sep 16 15:50:03.132917 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:03.132863    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-79xdw" event={"ID":"f8d2083a-9500-4604-be9f-09278b95a2c6","Type":"ContainerStarted","Data":"50781a786a0be04c553e58e1012541ffc3a0c119023a18e6d82fcc0fa1336315"}
Sep 16 15:50:03.140186 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:03.140145    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fzl79" event={"ID":"7b3420ee-3d02-4cbf-a6b2-fc58c2da0298","Type":"ContainerStarted","Data":"6060120d41cadf767af19cc85a7add5259355d599f498b8449304adae575b40e"}
Sep 16 15:50:03.143283 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:03.143251    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-65c8p" event={"ID":"d8e64660-b4ae-45ae-b31f-1faee114f212","Type":"ContainerStarted","Data":"7b10f70236272fa9ca8c561392fc188f5d3517011a47f69033be9e4103427788"}
Sep 16 15:50:03.146705 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:03.146636    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-q9vl8" event={"ID":"1dd0f6bb-bc9b-4518-bda6-9b12cef15952","Type":"ContainerStarted","Data":"1597de985dca4bd630a14c23f495c16e2519fa60b64378cba05d303b2965649c"}
Sep 16 15:50:03.701497 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:03.701398    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0d31180-c1e0-48f6-a5a2-d1eb9ae02886-original-pull-secret\") pod \"global-pull-secret-syncer-wcrfk\" (UID: \"c0d31180-c1e0-48f6-a5a2-d1eb9ae02886\") " pod="kube-system/global-pull-secret-syncer-wcrfk"
Sep 16 15:50:03.701497 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:03.701466    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2e625e0-e1fd-4733-97c0-2219660a6700-metrics-certs\") pod \"network-metrics-daemon-wtdsz\" (UID: \"b2e625e0-e1fd-4733-97c0-2219660a6700\") " pod="openshift-multus/network-metrics-daemon-wtdsz"
Sep 16 15:50:03.701777 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:03.701557    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 15:50:03.701777 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:03.701642    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0d31180-c1e0-48f6-a5a2-d1eb9ae02886-original-pull-secret podName:c0d31180-c1e0-48f6-a5a2-d1eb9ae02886 nodeName:}" failed. No retries permitted until 2026-09-16 15:50:05.701619771 +0000 UTC m=+6.146311947 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c0d31180-c1e0-48f6-a5a2-d1eb9ae02886-original-pull-secret") pod "global-pull-secret-syncer-wcrfk" (UID: "c0d31180-c1e0-48f6-a5a2-d1eb9ae02886") : object "kube-system"/"original-pull-secret" not registered
Sep 16 15:50:03.701777 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:03.701645    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 15:50:03.701777 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:03.701701    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2e625e0-e1fd-4733-97c0-2219660a6700-metrics-certs podName:b2e625e0-e1fd-4733-97c0-2219660a6700 nodeName:}" failed. No retries permitted until 2026-09-16 15:50:05.701683907 +0000 UTC m=+6.146376077 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b2e625e0-e1fd-4733-97c0-2219660a6700-metrics-certs") pod "network-metrics-daemon-wtdsz" (UID: "b2e625e0-e1fd-4733-97c0-2219660a6700") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 15:50:03.802950 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:03.802873    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q45qm\" (UniqueName: \"kubernetes.io/projected/25af0988-5e7d-4ee0-bdcb-cc956c6789ef-kube-api-access-q45qm\") pod \"network-check-target-vlhth\" (UID: \"25af0988-5e7d-4ee0-bdcb-cc956c6789ef\") " pod="openshift-network-diagnostics/network-check-target-vlhth"
Sep 16 15:50:03.803176 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:03.803091    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 15:50:03.803176 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:03.803113    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 15:50:03.803176 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:03.803127    2554 projected.go:196] Error preparing data for projected volume kube-api-access-q45qm for pod openshift-network-diagnostics/network-check-target-vlhth: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 15:50:03.803430 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:03.803191    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25af0988-5e7d-4ee0-bdcb-cc956c6789ef-kube-api-access-q45qm podName:25af0988-5e7d-4ee0-bdcb-cc956c6789ef nodeName:}" failed. No retries permitted until 2026-09-16 15:50:05.803171659 +0000 UTC m=+6.247863827 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-q45qm" (UniqueName: "kubernetes.io/projected/25af0988-5e7d-4ee0-bdcb-cc956c6789ef-kube-api-access-q45qm") pod "network-check-target-vlhth" (UID: "25af0988-5e7d-4ee0-bdcb-cc956c6789ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 15:50:04.010438 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:04.010361    2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 15:45:02 +0000 UTC" deadline="2028-04-14 18:39:44.843108514 +0000 UTC"
Sep 16 15:50:04.010438 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:04.010433    2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13826h49m40.832680044s"
Sep 16 15:50:04.074350 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:04.074316    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vlhth"
Sep 16 15:50:04.074555 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:04.074458    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vlhth" podUID="25af0988-5e7d-4ee0-bdcb-cc956c6789ef"
Sep 16 15:50:04.074971 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:04.074948    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wcrfk"
Sep 16 15:50:04.075123 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:04.075068    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wcrfk" podUID="c0d31180-c1e0-48f6-a5a2-d1eb9ae02886"
Sep 16 15:50:05.071595 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:05.071559    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtdsz"
Sep 16 15:50:05.072141 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:05.071713    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtdsz" podUID="b2e625e0-e1fd-4733-97c0-2219660a6700"
Sep 16 15:50:05.718403 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:05.718366    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2e625e0-e1fd-4733-97c0-2219660a6700-metrics-certs\") pod \"network-metrics-daemon-wtdsz\" (UID: \"b2e625e0-e1fd-4733-97c0-2219660a6700\") " pod="openshift-multus/network-metrics-daemon-wtdsz"
Sep 16 15:50:05.718613 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:05.718462    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0d31180-c1e0-48f6-a5a2-d1eb9ae02886-original-pull-secret\") pod \"global-pull-secret-syncer-wcrfk\" (UID: \"c0d31180-c1e0-48f6-a5a2-d1eb9ae02886\") " pod="kube-system/global-pull-secret-syncer-wcrfk"
Sep 16 15:50:05.718613 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:05.718599    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 15:50:05.718732 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:05.718659    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0d31180-c1e0-48f6-a5a2-d1eb9ae02886-original-pull-secret podName:c0d31180-c1e0-48f6-a5a2-d1eb9ae02886 nodeName:}" failed. No retries permitted until 2026-09-16 15:50:09.718638736 +0000 UTC m=+10.163330909 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c0d31180-c1e0-48f6-a5a2-d1eb9ae02886-original-pull-secret") pod "global-pull-secret-syncer-wcrfk" (UID: "c0d31180-c1e0-48f6-a5a2-d1eb9ae02886") : object "kube-system"/"original-pull-secret" not registered
Sep 16 15:50:05.719104 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:05.719084    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 15:50:05.719185 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:05.719150    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2e625e0-e1fd-4733-97c0-2219660a6700-metrics-certs podName:b2e625e0-e1fd-4733-97c0-2219660a6700 nodeName:}" failed. No retries permitted until 2026-09-16 15:50:09.719132631 +0000 UTC m=+10.163824805 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b2e625e0-e1fd-4733-97c0-2219660a6700-metrics-certs") pod "network-metrics-daemon-wtdsz" (UID: "b2e625e0-e1fd-4733-97c0-2219660a6700") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 15:50:05.819756 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:05.819137    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q45qm\" (UniqueName: \"kubernetes.io/projected/25af0988-5e7d-4ee0-bdcb-cc956c6789ef-kube-api-access-q45qm\") pod \"network-check-target-vlhth\" (UID: \"25af0988-5e7d-4ee0-bdcb-cc956c6789ef\") " pod="openshift-network-diagnostics/network-check-target-vlhth"
Sep 16 15:50:05.819756 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:05.819296    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 15:50:05.819756 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:05.819316    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 15:50:05.819756 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:05.819331    2554 projected.go:196] Error preparing data for projected volume kube-api-access-q45qm for pod openshift-network-diagnostics/network-check-target-vlhth: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 15:50:05.819756 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:05.819401    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25af0988-5e7d-4ee0-bdcb-cc956c6789ef-kube-api-access-q45qm podName:25af0988-5e7d-4ee0-bdcb-cc956c6789ef nodeName:}" failed. No retries permitted until 2026-09-16 15:50:09.819382218 +0000 UTC m=+10.264074395 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-q45qm" (UniqueName: "kubernetes.io/projected/25af0988-5e7d-4ee0-bdcb-cc956c6789ef-kube-api-access-q45qm") pod "network-check-target-vlhth" (UID: "25af0988-5e7d-4ee0-bdcb-cc956c6789ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 15:50:06.071986 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:06.071950    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vlhth"
Sep 16 15:50:06.072548 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:06.072098    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vlhth" podUID="25af0988-5e7d-4ee0-bdcb-cc956c6789ef"
Sep 16 15:50:06.072548 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:06.072503    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wcrfk"
Sep 16 15:50:06.072674 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:06.072604    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wcrfk" podUID="c0d31180-c1e0-48f6-a5a2-d1eb9ae02886"
Sep 16 15:50:07.072364 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:07.072199    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtdsz"
Sep 16 15:50:07.072364 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:07.072355    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtdsz" podUID="b2e625e0-e1fd-4733-97c0-2219660a6700"
Sep 16 15:50:08.071816 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:08.071769    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vlhth"
Sep 16 15:50:08.072026 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:08.071824    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wcrfk"
Sep 16 15:50:08.072026 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:08.071948    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vlhth" podUID="25af0988-5e7d-4ee0-bdcb-cc956c6789ef"
Sep 16 15:50:08.072151 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:08.072041    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wcrfk" podUID="c0d31180-c1e0-48f6-a5a2-d1eb9ae02886"
Sep 16 15:50:09.071540 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:09.071502    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtdsz"
Sep 16 15:50:09.072078 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:09.071714    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtdsz" podUID="b2e625e0-e1fd-4733-97c0-2219660a6700"
Sep 16 15:50:09.757486 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:09.757442    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2e625e0-e1fd-4733-97c0-2219660a6700-metrics-certs\") pod \"network-metrics-daemon-wtdsz\" (UID: \"b2e625e0-e1fd-4733-97c0-2219660a6700\") " pod="openshift-multus/network-metrics-daemon-wtdsz"
Sep 16 15:50:09.757683 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:09.757566    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0d31180-c1e0-48f6-a5a2-d1eb9ae02886-original-pull-secret\") pod \"global-pull-secret-syncer-wcrfk\" (UID: \"c0d31180-c1e0-48f6-a5a2-d1eb9ae02886\") " pod="kube-system/global-pull-secret-syncer-wcrfk"
Sep 16 15:50:09.757683 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:09.757586    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 15:50:09.757683 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:09.757673    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2e625e0-e1fd-4733-97c0-2219660a6700-metrics-certs podName:b2e625e0-e1fd-4733-97c0-2219660a6700 nodeName:}" failed. No retries permitted until 2026-09-16 15:50:17.757650306 +0000 UTC m=+18.202342479 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b2e625e0-e1fd-4733-97c0-2219660a6700-metrics-certs") pod "network-metrics-daemon-wtdsz" (UID: "b2e625e0-e1fd-4733-97c0-2219660a6700") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 15:50:09.757889 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:09.757689    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 15:50:09.757889 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:09.757746    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0d31180-c1e0-48f6-a5a2-d1eb9ae02886-original-pull-secret podName:c0d31180-c1e0-48f6-a5a2-d1eb9ae02886 nodeName:}" failed. No retries permitted until 2026-09-16 15:50:17.757727547 +0000 UTC m=+18.202419713 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c0d31180-c1e0-48f6-a5a2-d1eb9ae02886-original-pull-secret") pod "global-pull-secret-syncer-wcrfk" (UID: "c0d31180-c1e0-48f6-a5a2-d1eb9ae02886") : object "kube-system"/"original-pull-secret" not registered
Sep 16 15:50:09.858782 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:09.858740    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q45qm\" (UniqueName: \"kubernetes.io/projected/25af0988-5e7d-4ee0-bdcb-cc956c6789ef-kube-api-access-q45qm\") pod \"network-check-target-vlhth\" (UID: \"25af0988-5e7d-4ee0-bdcb-cc956c6789ef\") " pod="openshift-network-diagnostics/network-check-target-vlhth"
Sep 16 15:50:09.858989 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:09.858976    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 15:50:09.859061 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:09.858999    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 15:50:09.859061 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:09.859012    2554 projected.go:196] Error preparing data for projected volume kube-api-access-q45qm for pod openshift-network-diagnostics/network-check-target-vlhth: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 15:50:09.859177 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:09.859077    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25af0988-5e7d-4ee0-bdcb-cc956c6789ef-kube-api-access-q45qm podName:25af0988-5e7d-4ee0-bdcb-cc956c6789ef nodeName:}" failed. No retries permitted until 2026-09-16 15:50:17.859056935 +0000 UTC m=+18.303749111 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-q45qm" (UniqueName: "kubernetes.io/projected/25af0988-5e7d-4ee0-bdcb-cc956c6789ef-kube-api-access-q45qm") pod "network-check-target-vlhth" (UID: "25af0988-5e7d-4ee0-bdcb-cc956c6789ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 15:50:10.072224 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:10.072178    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vlhth"
Sep 16 15:50:10.072872 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:10.072295    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wcrfk"
Sep 16 15:50:10.072872 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:10.072317    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vlhth" podUID="25af0988-5e7d-4ee0-bdcb-cc956c6789ef"
Sep 16 15:50:10.072872 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:10.072376    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wcrfk" podUID="c0d31180-c1e0-48f6-a5a2-d1eb9ae02886"
Sep 16 15:50:11.071247 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:11.071209    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtdsz"
Sep 16 15:50:11.071459 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:11.071375    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtdsz" podUID="b2e625e0-e1fd-4733-97c0-2219660a6700"
Sep 16 15:50:12.071488 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:12.071441    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vlhth"
Sep 16 15:50:12.071877 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:12.071502    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wcrfk"
Sep 16 15:50:12.071877 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:12.071610    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wcrfk" podUID="c0d31180-c1e0-48f6-a5a2-d1eb9ae02886"
Sep 16 15:50:12.071877 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:12.071690    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vlhth" podUID="25af0988-5e7d-4ee0-bdcb-cc956c6789ef"
Sep 16 15:50:13.071978 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:13.071945    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtdsz"
Sep 16 15:50:13.072327 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:13.072051    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtdsz" podUID="b2e625e0-e1fd-4733-97c0-2219660a6700"
Sep 16 15:50:14.071637 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:14.071598    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wcrfk"
Sep 16 15:50:14.071821 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:14.071598    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vlhth"
Sep 16 15:50:14.071821 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:14.071737    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wcrfk" podUID="c0d31180-c1e0-48f6-a5a2-d1eb9ae02886"
Sep 16 15:50:14.071946 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:14.071842    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vlhth" podUID="25af0988-5e7d-4ee0-bdcb-cc956c6789ef"
Sep 16 15:50:15.071125 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:15.071093    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtdsz"
Sep 16 15:50:15.071607 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:15.071225    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtdsz" podUID="b2e625e0-e1fd-4733-97c0-2219660a6700"
Sep 16 15:50:16.072114 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:16.072077    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wcrfk"
Sep 16 15:50:16.072485 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:16.072074    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vlhth"
Sep 16 15:50:16.072485 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:16.072212    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wcrfk" podUID="c0d31180-c1e0-48f6-a5a2-d1eb9ae02886"
Sep 16 15:50:16.072485 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:16.072253    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vlhth" podUID="25af0988-5e7d-4ee0-bdcb-cc956c6789ef"
Sep 16 15:50:17.071852 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:17.071817    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtdsz"
Sep 16 15:50:17.072067 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:17.071960    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtdsz" podUID="b2e625e0-e1fd-4733-97c0-2219660a6700"
Sep 16 15:50:17.822532 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:17.822482    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0d31180-c1e0-48f6-a5a2-d1eb9ae02886-original-pull-secret\") pod \"global-pull-secret-syncer-wcrfk\" (UID: \"c0d31180-c1e0-48f6-a5a2-d1eb9ae02886\") " pod="kube-system/global-pull-secret-syncer-wcrfk"
Sep 16 15:50:17.823013 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:17.822546    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2e625e0-e1fd-4733-97c0-2219660a6700-metrics-certs\") pod \"network-metrics-daemon-wtdsz\" (UID: \"b2e625e0-e1fd-4733-97c0-2219660a6700\") " pod="openshift-multus/network-metrics-daemon-wtdsz"
Sep 16 15:50:17.823013 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:17.822662    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 15:50:17.823013 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:17.822696    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 15:50:17.823013 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:17.822751    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0d31180-c1e0-48f6-a5a2-d1eb9ae02886-original-pull-secret podName:c0d31180-c1e0-48f6-a5a2-d1eb9ae02886 nodeName:}" failed. No retries permitted until 2026-09-16 15:50:33.822728425 +0000 UTC m=+34.267420604 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c0d31180-c1e0-48f6-a5a2-d1eb9ae02886-original-pull-secret") pod "global-pull-secret-syncer-wcrfk" (UID: "c0d31180-c1e0-48f6-a5a2-d1eb9ae02886") : object "kube-system"/"original-pull-secret" not registered
Sep 16 15:50:17.823013 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:17.822773    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2e625e0-e1fd-4733-97c0-2219660a6700-metrics-certs podName:b2e625e0-e1fd-4733-97c0-2219660a6700 nodeName:}" failed. No retries permitted until 2026-09-16 15:50:33.822762172 +0000 UTC m=+34.267454331 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b2e625e0-e1fd-4733-97c0-2219660a6700-metrics-certs") pod "network-metrics-daemon-wtdsz" (UID: "b2e625e0-e1fd-4733-97c0-2219660a6700") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 15:50:17.923178 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:17.923137    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q45qm\" (UniqueName: \"kubernetes.io/projected/25af0988-5e7d-4ee0-bdcb-cc956c6789ef-kube-api-access-q45qm\") pod \"network-check-target-vlhth\" (UID: \"25af0988-5e7d-4ee0-bdcb-cc956c6789ef\") " pod="openshift-network-diagnostics/network-check-target-vlhth"
Sep 16 15:50:17.923375 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:17.923364    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 15:50:17.923441 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:17.923386    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 15:50:17.923441 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:17.923399    2554 projected.go:196] Error preparing data for projected volume kube-api-access-q45qm for pod openshift-network-diagnostics/network-check-target-vlhth: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 15:50:17.923516 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:17.923463    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25af0988-5e7d-4ee0-bdcb-cc956c6789ef-kube-api-access-q45qm podName:25af0988-5e7d-4ee0-bdcb-cc956c6789ef nodeName:}" failed. No retries permitted until 2026-09-16 15:50:33.923446013 +0000 UTC m=+34.368138173 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-q45qm" (UniqueName: "kubernetes.io/projected/25af0988-5e7d-4ee0-bdcb-cc956c6789ef-kube-api-access-q45qm") pod "network-check-target-vlhth" (UID: "25af0988-5e7d-4ee0-bdcb-cc956c6789ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 15:50:18.071184 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:18.071151    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wcrfk"
Sep 16 15:50:18.071184 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:18.071186    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vlhth"
Sep 16 15:50:18.071423 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:18.071254    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wcrfk" podUID="c0d31180-c1e0-48f6-a5a2-d1eb9ae02886"
Sep 16 15:50:18.071423 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:18.071309    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vlhth" podUID="25af0988-5e7d-4ee0-bdcb-cc956c6789ef"
Sep 16 15:50:19.071520 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:19.071491    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtdsz"
Sep 16 15:50:19.072065 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:19.071619    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtdsz" podUID="b2e625e0-e1fd-4733-97c0-2219660a6700"
Sep 16 15:50:20.072539 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:20.072501    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vlhth"
Sep 16 15:50:20.072986 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:20.072615    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vlhth" podUID="25af0988-5e7d-4ee0-bdcb-cc956c6789ef"
Sep 16 15:50:20.072986 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:20.072723    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wcrfk"
Sep 16 15:50:20.072986 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:20.072832    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wcrfk" podUID="c0d31180-c1e0-48f6-a5a2-d1eb9ae02886"
Sep 16 15:50:21.071604 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:21.071562    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtdsz"
Sep 16 15:50:21.071767 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:21.071693    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtdsz" podUID="b2e625e0-e1fd-4733-97c0-2219660a6700"
Sep 16 15:50:22.072097 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:22.072049    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vlhth"
Sep 16 15:50:22.072645 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:22.072104    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wcrfk"
Sep 16 15:50:22.072645 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:22.072189    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vlhth" podUID="25af0988-5e7d-4ee0-bdcb-cc956c6789ef"
Sep 16 15:50:22.072645 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:22.072330    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wcrfk" podUID="c0d31180-c1e0-48f6-a5a2-d1eb9ae02886"
Sep 16 15:50:23.072035 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:23.071825    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtdsz"
Sep 16 15:50:23.072288 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:23.072171    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtdsz" podUID="b2e625e0-e1fd-4733-97c0-2219660a6700"
Sep 16 15:50:23.202001 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:23.201963    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw" event={"ID":"0975e4b2-fcec-4ae9-839c-92b6158b6aca","Type":"ContainerStarted","Data":"79ea8631c397eb4fd191fe5bc00938ee1868e281dc46ad0dc4ef35149e2cb204"}
Sep 16 15:50:23.202160 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:23.202059    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw" event={"ID":"0975e4b2-fcec-4ae9-839c-92b6158b6aca","Type":"ContainerStarted","Data":"2ecf4e87d85fb33001c59599a121808952e745e42c5278f71277043844eac547"}
Sep 16 15:50:23.202160 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:23.202111    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw" event={"ID":"0975e4b2-fcec-4ae9-839c-92b6158b6aca","Type":"ContainerStarted","Data":"41cf9baaf0ef428e43de2fc7800e35a55542099575754fc7b725dd0c7dc1c2cf"}
Sep 16 15:50:23.202246 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:23.202158    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw" event={"ID":"0975e4b2-fcec-4ae9-839c-92b6158b6aca","Type":"ContainerStarted","Data":"a659402011fa38b102e326c685041643a069d22929f186c2689d6f2a5417195f"}
Sep 16 15:50:23.202246 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:23.202202    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw" event={"ID":"0975e4b2-fcec-4ae9-839c-92b6158b6aca","Type":"ContainerStarted","Data":"89f6ab9d0f2214e8f57b0b26754b084a5b819b5a76582beea0e85bbb76f45242"}
Sep 16 15:50:23.202246 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:23.202232    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw" event={"ID":"0975e4b2-fcec-4ae9-839c-92b6158b6aca","Type":"ContainerStarted","Data":"37c91c5782d823b9a8dfc1395253bbb3f4389c3a91e809ae13c9c3e4e3396220"}
Sep 16 15:50:23.203867 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:23.203834    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvv2d" event={"ID":"c474ab2f-87a0-463c-806d-9d4d977c989d","Type":"ContainerStarted","Data":"44d4bda1bea108573909270b1e3dcb27c77dbbebd894d210681810128af45fbc"}
Sep 16 15:50:23.205479 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:23.205454    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-79xdw" event={"ID":"f8d2083a-9500-4604-be9f-09278b95a2c6","Type":"ContainerStarted","Data":"b1720af56aebc1ab9e489289c4a9fd31dd398ec2f1ff717f01132857104d7819"}
Sep 16 15:50:23.207042 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:23.207017    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fzl79" event={"ID":"7b3420ee-3d02-4cbf-a6b2-fc58c2da0298","Type":"ContainerStarted","Data":"eaac3a74f4cb936f871c507c6846d2cff71b998e2df4341f58af5f85cd688c05"}
Sep 16 15:50:23.208744 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:23.208717    2554 generic.go:356] "Generic (PLEG): container finished" podID="d8e64660-b4ae-45ae-b31f-1faee114f212" containerID="bd3aceb640f65b7b6e9eb59f2a93ecca75c34a472b57fca9c974da9996cecfad" exitCode=0
Sep 16 15:50:23.208844 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:23.208785    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-65c8p" event={"ID":"d8e64660-b4ae-45ae-b31f-1faee114f212","Type":"ContainerDied","Data":"bd3aceb640f65b7b6e9eb59f2a93ecca75c34a472b57fca9c974da9996cecfad"}
Sep 16 15:50:23.210569 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:23.210545    2554 generic.go:356] "Generic (PLEG): container finished" podID="6ce98f1de64686998922578fd3388801" containerID="cfb23e173a62f86f70712288740adc0cd9386831dee6c2ad221b42c1369f395a" exitCode=0
Sep 16 15:50:23.210673 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:23.210628    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal" event={"ID":"6ce98f1de64686998922578fd3388801","Type":"ContainerDied","Data":"cfb23e173a62f86f70712288740adc0cd9386831dee6c2ad221b42c1369f395a"}
Sep 16 15:50:23.214250 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:23.214195    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-228.ec2.internal" event={"ID":"c8010e8ff400f9e15027bac559fe6673","Type":"ContainerStarted","Data":"1740506ac2ef68df4ea1864225277214b787b79cef6202392a44675077320500"}
Sep 16 15:50:23.216343 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:23.216321    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7gbkh" event={"ID":"c30e3d8c-1676-44b8-9a25-fce32e24c7bc","Type":"ContainerStarted","Data":"8c15dc83649fd856cdf5beb986866c19d2a003afa2532b77bed626d99aa456be"}
Sep 16 15:50:23.218042 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:23.218018    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ffp62" event={"ID":"ef023c4c-0a32-416e-8a44-f8e8e4302e5f","Type":"ContainerStarted","Data":"8498b6953425b24f5f7636692d77fbd98ddc81c4a3c6ebe56cef68e38c733b74"}
Sep 16 15:50:23.219471 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:23.219452    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gc8gs" event={"ID":"9e1f1b7b-ac55-41fd-b6d5-1c6257652517","Type":"ContainerStarted","Data":"b03c0b0b2ef10cadac92999a4be79b4816acec36de5894c9ccda06e35e28c4cd"}
Sep 16 15:50:23.231672 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:23.231619    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-79xdw" podStartSLOduration=2.383192028 podStartE2EDuration="22.231602518s" totalImagesPullingTime="19.84841049s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 15:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:50:23.231482501 +0000 UTC m=+23.676174716" watchObservedRunningTime="2026-09-16 15:50:23.231602518 +0000 UTC m=+23.676294699"
Sep 16 15:50:23.245806 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:23.245736    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gc8gs" podStartSLOduration=7.408044861 podStartE2EDuration="23.245722985s" totalImagesPullingTime="15.837678124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:50:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:50:23.2457006 +0000 UTC m=+23.690392781" watchObservedRunningTime="2026-09-16 15:50:23.245722985 +0000 UTC m=+23.690415161"
Sep 16 15:50:23.268277 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:23.268224    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-fzl79" podStartSLOduration=2.428734462 podStartE2EDuration="22.268212554s" totalImagesPullingTime="19.839478092s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 15:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:50:23.268167678 +0000 UTC m=+23.712859877" watchObservedRunningTime="2026-09-16 15:50:23.268212554 +0000 UTC m=+23.712904732"
Sep 16 15:50:23.286756 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:23.286697    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-228.ec2.internal" podStartSLOduration=22.286678761 podStartE2EDuration="22.286678761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 15:50:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:50:23.286567091 +0000 UTC m=+23.731259365" watchObservedRunningTime="2026-09-16 15:50:23.286678761 +0000 UTC m=+23.731370938"
Sep 16 15:50:23.318221 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:23.318175    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-ffp62" podStartSLOduration=3.4632578609999998 podStartE2EDuration="23.318161147s" totalImagesPullingTime="19.854903286s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 15:50:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:50:23.317742841 +0000 UTC m=+23.762435040" watchObservedRunningTime="2026-09-16 15:50:23.318161147 +0000 UTC m=+23.762853325"
Sep 16 15:50:23.466492 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:23.466459    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gc8gs"
Sep 16 15:50:23.467141 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:23.467124    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gc8gs"
Sep 16 15:50:23.483153 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:23.483097    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-7gbkh" podStartSLOduration=2.570208981 podStartE2EDuration="22.483080257s" totalImagesPullingTime="19.912871276s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 15:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:50:23.355856632 +0000 UTC m=+23.800548838" watchObservedRunningTime="2026-09-16 15:50:23.483080257 +0000 UTC m=+23.927772433"
Sep 16 15:50:23.743039 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:23.743006    2554 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 15:50:24.040996 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:24.040840    2554 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T15:50:23.743033223Z","UUID":"750feb22-55ed-4227-aa57-fccbf41e157f","Handler":null,"Name":"","Endpoint":""}
Sep 16 15:50:24.042439 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:24.042414    2554 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 15:50:24.042560 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:24.042454    2554 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 15:50:24.071891 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:24.071861    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vlhth"
Sep 16 15:50:24.071891 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:24.071877    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wcrfk"
Sep 16 15:50:24.072114 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:24.071970    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vlhth" podUID="25af0988-5e7d-4ee0-bdcb-cc956c6789ef"
Sep 16 15:50:24.072114 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:24.072093    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wcrfk" podUID="c0d31180-c1e0-48f6-a5a2-d1eb9ae02886"
Sep 16 15:50:24.225205 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:24.225164    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvv2d" event={"ID":"c474ab2f-87a0-463c-806d-9d4d977c989d","Type":"ContainerStarted","Data":"763bd9c1d8cad80b570e1599fb404b8cf511211c22e7ab536f7f74b446c65865"}
Sep 16 15:50:24.227402 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:24.227377    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal" event={"ID":"6ce98f1de64686998922578fd3388801","Type":"ContainerStarted","Data":"6e2207af54cd786fd4d666d68e889bdae42454380dfecc2ab20483406af04baf"}
Sep 16 15:50:24.229212 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:24.229139    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-q9vl8" event={"ID":"1dd0f6bb-bc9b-4518-bda6-9b12cef15952","Type":"ContainerStarted","Data":"38ad846c569d80fcb62fefed8658a2b15cc7e96f8284d859446a9c50a118a13b"}
Sep 16 15:50:24.230110 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:24.229901    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gc8gs"
Sep 16 15:50:24.230188 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:24.230123    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gc8gs"
Sep 16 15:50:24.244070 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:24.244019    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-228.ec2.internal" podStartSLOduration=23.244000604 podStartE2EDuration="23.244000604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 15:50:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:50:24.242953349 +0000 UTC m=+24.687645549" watchObservedRunningTime="2026-09-16 15:50:24.244000604 +0000 UTC m=+24.688692784"
Sep 16 15:50:24.258832 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:24.258721    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-q9vl8" podStartSLOduration=3.448023386 podStartE2EDuration="23.258705099s" totalImagesPullingTime="19.810681713s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 15:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:50:24.258639241 +0000 UTC m=+24.703331448" watchObservedRunningTime="2026-09-16 15:50:24.258705099 +0000 UTC m=+24.703397278"
Sep 16 15:50:25.071564 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:25.071354    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtdsz"
Sep 16 15:50:25.071756 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:25.071656    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtdsz" podUID="b2e625e0-e1fd-4733-97c0-2219660a6700"
Sep 16 15:50:25.235947 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:25.235908    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw" event={"ID":"0975e4b2-fcec-4ae9-839c-92b6158b6aca","Type":"ContainerStarted","Data":"4bccbd527939b34c17a41de28c26596eb469bc22e2c88a853bcf208edee38eea"}
Sep 16 15:50:25.238638 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:25.238602    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvv2d" event={"ID":"c474ab2f-87a0-463c-806d-9d4d977c989d","Type":"ContainerStarted","Data":"dcc504baba8c4afcf74553b760fbb644033dfc8a252f142feb6575919dcddbff"}
Sep 16 15:50:25.258350 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:25.258283    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvv2d" podStartSLOduration=2.422877093 podStartE2EDuration="24.258266834s" totalImagesPullingTime="21.835389741s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 15:50:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:50:25.25803927 +0000 UTC m=+25.702731467" watchObservedRunningTime="2026-09-16 15:50:25.258266834 +0000 UTC m=+25.702959027"
Sep 16 15:50:26.071442 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:26.071411    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vlhth"
Sep 16 15:50:26.071662 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:26.071414    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wcrfk"
Sep 16 15:50:26.071662 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:26.071519    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vlhth" podUID="25af0988-5e7d-4ee0-bdcb-cc956c6789ef"
Sep 16 15:50:26.071662 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:26.071613    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wcrfk" podUID="c0d31180-c1e0-48f6-a5a2-d1eb9ae02886"
Sep 16 15:50:27.071150 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:27.071112    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtdsz"
Sep 16 15:50:27.071585 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:27.071252    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtdsz" podUID="b2e625e0-e1fd-4733-97c0-2219660a6700"
Sep 16 15:50:28.071364 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:28.071169    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vlhth"
Sep 16 15:50:28.071749 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:28.071172    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wcrfk"
Sep 16 15:50:28.071749 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:28.071453    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vlhth" podUID="25af0988-5e7d-4ee0-bdcb-cc956c6789ef"
Sep 16 15:50:28.071749 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:28.071521    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wcrfk" podUID="c0d31180-c1e0-48f6-a5a2-d1eb9ae02886"
Sep 16 15:50:28.251433 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:28.251395    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw" event={"ID":"0975e4b2-fcec-4ae9-839c-92b6158b6aca","Type":"ContainerStarted","Data":"8688858b1386c561ddc9387c327d516dd4dcf43506ff0d0c583b4b78c1252650"}
Sep 16 15:50:28.251751 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:28.251715    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:28.253308 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:28.253282    2554 generic.go:356] "Generic (PLEG): container finished" podID="d8e64660-b4ae-45ae-b31f-1faee114f212" containerID="4bab4777555e5fd3c127d42a9add1e6d7635ab797f92e654e8d15cf31e6f62d8" exitCode=0
Sep 16 15:50:28.253406 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:28.253332    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-65c8p" event={"ID":"d8e64660-b4ae-45ae-b31f-1faee114f212","Type":"ContainerDied","Data":"4bab4777555e5fd3c127d42a9add1e6d7635ab797f92e654e8d15cf31e6f62d8"}
Sep 16 15:50:28.267096 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:28.267075    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:28.288007 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:28.287965    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw" podStartSLOduration=7.40407294 podStartE2EDuration="27.287951737s" totalImagesPullingTime="19.883878797s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 15:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:50:28.287103893 +0000 UTC m=+28.731796091" watchObservedRunningTime="2026-09-16 15:50:28.287951737 +0000 UTC m=+28.732643924"
Sep 16 15:50:29.071674 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:29.071636    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtdsz"
Sep 16 15:50:29.072366 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:29.071769    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtdsz" podUID="b2e625e0-e1fd-4733-97c0-2219660a6700"
Sep 16 15:50:29.255868 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:29.255835    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:29.256028 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:29.255916    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:29.270169 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:29.270139    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:50:29.403850 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:29.403750    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wcrfk"]
Sep 16 15:50:29.404008 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:29.403880    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wcrfk"
Sep 16 15:50:29.404074 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:29.404036    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wcrfk" podUID="c0d31180-c1e0-48f6-a5a2-d1eb9ae02886"
Sep 16 15:50:29.404480 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:29.404448    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wtdsz"]
Sep 16 15:50:29.404601 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:29.404548    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtdsz"
Sep 16 15:50:29.404685 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:29.404648    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtdsz" podUID="b2e625e0-e1fd-4733-97c0-2219660a6700"
Sep 16 15:50:29.406837 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:29.406813    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vlhth"]
Sep 16 15:50:29.406941 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:29.406912    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vlhth"
Sep 16 15:50:29.406998 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:29.406982    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vlhth" podUID="25af0988-5e7d-4ee0-bdcb-cc956c6789ef"
Sep 16 15:50:31.071906 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:31.071846    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtdsz"
Sep 16 15:50:31.072389 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:31.071859    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vlhth"
Sep 16 15:50:31.072389 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:31.071966    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtdsz" podUID="b2e625e0-e1fd-4733-97c0-2219660a6700"
Sep 16 15:50:31.072389 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:31.071986    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wcrfk"
Sep 16 15:50:31.072389 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:31.072081    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vlhth" podUID="25af0988-5e7d-4ee0-bdcb-cc956c6789ef"
Sep 16 15:50:31.072389 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:31.072158    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wcrfk" podUID="c0d31180-c1e0-48f6-a5a2-d1eb9ae02886"
Sep 16 15:50:32.264055 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:32.263810    2554 generic.go:356] "Generic (PLEG): container finished" podID="d8e64660-b4ae-45ae-b31f-1faee114f212" containerID="dcb3ac2aff550fc95a4a0b78f50d68fbf739a033655bafff4605966f2bd2e5f7" exitCode=0
Sep 16 15:50:32.264055 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:32.263899    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-65c8p" event={"ID":"d8e64660-b4ae-45ae-b31f-1faee114f212","Type":"ContainerDied","Data":"dcb3ac2aff550fc95a4a0b78f50d68fbf739a033655bafff4605966f2bd2e5f7"}
Sep 16 15:50:33.071641 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:33.071558    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtdsz"
Sep 16 15:50:33.071641 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:33.071601    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wcrfk"
Sep 16 15:50:33.071833 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:33.071565    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vlhth"
Sep 16 15:50:33.071833 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:33.071681    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtdsz" podUID="b2e625e0-e1fd-4733-97c0-2219660a6700"
Sep 16 15:50:33.071833 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:33.071750    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vlhth" podUID="25af0988-5e7d-4ee0-bdcb-cc956c6789ef"
Sep 16 15:50:33.071833 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:33.071823    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wcrfk" podUID="c0d31180-c1e0-48f6-a5a2-d1eb9ae02886"
Sep 16 15:50:33.268767 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:33.268738    2554 generic.go:356] "Generic (PLEG): container finished" podID="d8e64660-b4ae-45ae-b31f-1faee114f212" containerID="e541e1c363e327f1dac2014ecb0dbf74abdc6521d182aeabf721df946258491d" exitCode=0
Sep 16 15:50:33.268767 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:33.268773    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-65c8p" event={"ID":"d8e64660-b4ae-45ae-b31f-1faee114f212","Type":"ContainerDied","Data":"e541e1c363e327f1dac2014ecb0dbf74abdc6521d182aeabf721df946258491d"}
Sep 16 15:50:33.835493 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:33.835458    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0d31180-c1e0-48f6-a5a2-d1eb9ae02886-original-pull-secret\") pod \"global-pull-secret-syncer-wcrfk\" (UID: \"c0d31180-c1e0-48f6-a5a2-d1eb9ae02886\") " pod="kube-system/global-pull-secret-syncer-wcrfk"
Sep 16 15:50:33.835673 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:33.835504    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2e625e0-e1fd-4733-97c0-2219660a6700-metrics-certs\") pod \"network-metrics-daemon-wtdsz\" (UID: \"b2e625e0-e1fd-4733-97c0-2219660a6700\") " pod="openshift-multus/network-metrics-daemon-wtdsz"
Sep 16 15:50:33.835673 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:33.835623    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 15:50:33.835787 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:33.835683    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2e625e0-e1fd-4733-97c0-2219660a6700-metrics-certs podName:b2e625e0-e1fd-4733-97c0-2219660a6700 nodeName:}" failed. No retries permitted until 2026-09-16 15:51:05.835664789 +0000 UTC m=+66.280356951 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b2e625e0-e1fd-4733-97c0-2219660a6700-metrics-certs") pod "network-metrics-daemon-wtdsz" (UID: "b2e625e0-e1fd-4733-97c0-2219660a6700") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 15:50:33.835787 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:33.835624    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 15:50:33.835921 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:33.835789    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0d31180-c1e0-48f6-a5a2-d1eb9ae02886-original-pull-secret podName:c0d31180-c1e0-48f6-a5a2-d1eb9ae02886 nodeName:}" failed. No retries permitted until 2026-09-16 15:51:05.835768885 +0000 UTC m=+66.280461055 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c0d31180-c1e0-48f6-a5a2-d1eb9ae02886-original-pull-secret") pod "global-pull-secret-syncer-wcrfk" (UID: "c0d31180-c1e0-48f6-a5a2-d1eb9ae02886") : object "kube-system"/"original-pull-secret" not registered
Sep 16 15:50:33.936121 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:33.936030    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q45qm\" (UniqueName: \"kubernetes.io/projected/25af0988-5e7d-4ee0-bdcb-cc956c6789ef-kube-api-access-q45qm\") pod \"network-check-target-vlhth\" (UID: \"25af0988-5e7d-4ee0-bdcb-cc956c6789ef\") " pod="openshift-network-diagnostics/network-check-target-vlhth"
Sep 16 15:50:33.936260 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:33.936192    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 15:50:33.936260 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:33.936213    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 15:50:33.936260 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:33.936223    2554 projected.go:196] Error preparing data for projected volume kube-api-access-q45qm for pod openshift-network-diagnostics/network-check-target-vlhth: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 15:50:33.936409 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:33.936283    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25af0988-5e7d-4ee0-bdcb-cc956c6789ef-kube-api-access-q45qm podName:25af0988-5e7d-4ee0-bdcb-cc956c6789ef nodeName:}" failed. No retries permitted until 2026-09-16 15:51:05.936264471 +0000 UTC m=+66.380956632 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-q45qm" (UniqueName: "kubernetes.io/projected/25af0988-5e7d-4ee0-bdcb-cc956c6789ef-kube-api-access-q45qm") pod "network-check-target-vlhth" (UID: "25af0988-5e7d-4ee0-bdcb-cc956c6789ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 15:50:34.822016 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.821990    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-228.ec2.internal" event="NodeReady"
Sep 16 15:50:34.822475 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.822142    2554 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 15:50:34.858269 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.858232    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-98b6c5d9f-b5qrx"]
Sep 16 15:50:34.862150 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.861700    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74cc858bfd-ngwlg"]
Sep 16 15:50:34.862150 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.861807    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx"
Sep 16 15:50:34.864093 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.864075    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-b5557cb49-pp9jc"]
Sep 16 15:50:34.864233 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.864211    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74cc858bfd-ngwlg"
Sep 16 15:50:34.864881 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.864837    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 15:50:34.864980 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.864885    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 15:50:34.864980 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.864886    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-c92pf\""
Sep 16 15:50:34.865100 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.865002    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 15:50:34.866351 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.866330    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg"]
Sep 16 15:50:34.866457 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.866413    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b5557cb49-pp9jc"
Sep 16 15:50:34.867910 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.867887    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 15:50:34.868017 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.867888    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-82xh7\""
Sep 16 15:50:34.868076 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.868037    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 16 15:50:34.869001 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.868979    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 15:50:34.869089 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.869029    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 15:50:34.869089 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.869056    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg"
Sep 16 15:50:34.871490 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.871124    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 16 15:50:34.872044 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.872024    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 15:50:34.872453 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.872430    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 16 15:50:34.872636 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.872619    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 16 15:50:34.872751 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.872734    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 16 15:50:34.874437 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.874416    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-98b6c5d9f-b5qrx"]
Sep 16 15:50:34.875002 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.874980    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 16 15:50:34.877022 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.876996    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74cc858bfd-ngwlg"]
Sep 16 15:50:34.891824 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.888928    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg"]
Sep 16 15:50:34.891824 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.889875    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gb9f6"]
Sep 16 15:50:34.894426 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.894402    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-g826v"]
Sep 16 15:50:34.894539 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.894519    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gb9f6"
Sep 16 15:50:34.896913 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.896893    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-b5557cb49-pp9jc"]
Sep 16 15:50:34.897010 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.896977    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-g826v"
Sep 16 15:50:34.897010 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.896997    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 15:50:34.897274 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.897181    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 15:50:34.897351 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.897282    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-thq8g\""
Sep 16 15:50:34.898809 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.898778    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-g826v"]
Sep 16 15:50:34.899546 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.899530    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 15:50:34.899743 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.899723    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-j6n8k\""
Sep 16 15:50:34.899949 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.899932    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 15:50:34.900048 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.899961    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 15:50:34.903263 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.903222    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gb9f6"]
Sep 16 15:50:34.942029 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.942003    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5a899343-9e86-4def-b525-4586cb7b2afc-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg\" (UID: \"5a899343-9e86-4def-b525-4586cb7b2afc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg"
Sep 16 15:50:34.942241 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.942041    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bb5d1a75-f8fc-4f48-9969-33e5126047f8-klusterlet-config\") pod \"klusterlet-addon-workmgr-b5557cb49-pp9jc\" (UID: \"bb5d1a75-f8fc-4f48-9969-33e5126047f8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b5557cb49-pp9jc"
Sep 16 15:50:34.942241 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.942074    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/5a899343-9e86-4def-b525-4586cb7b2afc-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg\" (UID: \"5a899343-9e86-4def-b525-4586cb7b2afc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg"
Sep 16 15:50:34.942241 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.942160    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-registry-tls\") pod \"image-registry-98b6c5d9f-b5qrx\" (UID: \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\") " pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx"
Sep 16 15:50:34.942241 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.942189    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqvct\" (UniqueName: \"kubernetes.io/projected/5a899343-9e86-4def-b525-4586cb7b2afc-kube-api-access-fqvct\") pod \"cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg\" (UID: \"5a899343-9e86-4def-b525-4586cb7b2afc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg"
Sep 16 15:50:34.942241 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.942218    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-registry-certificates\") pod \"image-registry-98b6c5d9f-b5qrx\" (UID: \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\") " pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx"
Sep 16 15:50:34.942620 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.942245    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6bfr\" (UniqueName: \"kubernetes.io/projected/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-kube-api-access-f6bfr\") pod \"image-registry-98b6c5d9f-b5qrx\" (UID: \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\") " pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx"
Sep 16 15:50:34.942620 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.942269    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wd2n\" (UniqueName: \"kubernetes.io/projected/bb5d1a75-f8fc-4f48-9969-33e5126047f8-kube-api-access-6wd2n\") pod \"klusterlet-addon-workmgr-b5557cb49-pp9jc\" (UID: \"bb5d1a75-f8fc-4f48-9969-33e5126047f8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b5557cb49-pp9jc"
Sep 16 15:50:34.942620 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.942297    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-installation-pull-secrets\") pod \"image-registry-98b6c5d9f-b5qrx\" (UID: \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\") " pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx"
Sep 16 15:50:34.942620 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.942317    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/5a899343-9e86-4def-b525-4586cb7b2afc-hub\") pod \"cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg\" (UID: \"5a899343-9e86-4def-b525-4586cb7b2afc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg"
Sep 16 15:50:34.942620 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.942337    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ecdd46d5-8e17-4b6e-a93e-db42d0530f67-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-74cc858bfd-ngwlg\" (UID: \"ecdd46d5-8e17-4b6e-a93e-db42d0530f67\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74cc858bfd-ngwlg"
Sep 16 15:50:34.942620 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.942393    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnxz4\" (UniqueName: \"kubernetes.io/projected/ecdd46d5-8e17-4b6e-a93e-db42d0530f67-kube-api-access-pnxz4\") pod \"managed-serviceaccount-addon-agent-74cc858bfd-ngwlg\" (UID: \"ecdd46d5-8e17-4b6e-a93e-db42d0530f67\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74cc858bfd-ngwlg"
Sep 16 15:50:34.942620 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.942424    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-trusted-ca\") pod \"image-registry-98b6c5d9f-b5qrx\" (UID: \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\") " pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx"
Sep 16 15:50:34.942620 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.942448    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a899343-9e86-4def-b525-4586cb7b2afc-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg\" (UID: \"5a899343-9e86-4def-b525-4586cb7b2afc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg"
Sep 16 15:50:34.942620 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.942468    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-ca-trust-extracted\") pod \"image-registry-98b6c5d9f-b5qrx\" (UID: \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\") " pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx"
Sep 16 15:50:34.942620 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.942493    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-image-registry-private-configuration\") pod \"image-registry-98b6c5d9f-b5qrx\" (UID: \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\") " pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx"
Sep 16 15:50:34.942620 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.942526    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-bound-sa-token\") pod \"image-registry-98b6c5d9f-b5qrx\" (UID: \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\") " pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx"
Sep 16 15:50:34.942620 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.942550    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb5d1a75-f8fc-4f48-9969-33e5126047f8-tmp\") pod \"klusterlet-addon-workmgr-b5557cb49-pp9jc\" (UID: \"bb5d1a75-f8fc-4f48-9969-33e5126047f8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b5557cb49-pp9jc"
Sep 16 15:50:34.942620 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:34.942581    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/5a899343-9e86-4def-b525-4586cb7b2afc-ca\") pod \"cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg\" (UID: \"5a899343-9e86-4def-b525-4586cb7b2afc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg"
Sep 16 15:50:35.043361 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.043329    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6bfr\" (UniqueName: \"kubernetes.io/projected/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-kube-api-access-f6bfr\") pod \"image-registry-98b6c5d9f-b5qrx\" (UID: \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\") " pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx"
Sep 16 15:50:35.043528 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.043376    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wd2n\" (UniqueName: \"kubernetes.io/projected/bb5d1a75-f8fc-4f48-9969-33e5126047f8-kube-api-access-6wd2n\") pod \"klusterlet-addon-workmgr-b5557cb49-pp9jc\" (UID: \"bb5d1a75-f8fc-4f48-9969-33e5126047f8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b5557cb49-pp9jc"
Sep 16 15:50:35.043528 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.043409    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-installation-pull-secrets\") pod \"image-registry-98b6c5d9f-b5qrx\" (UID: \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\") " pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx"
Sep 16 15:50:35.043651 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.043556    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/5a899343-9e86-4def-b525-4586cb7b2afc-hub\") pod \"cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg\" (UID: \"5a899343-9e86-4def-b525-4586cb7b2afc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg"
Sep 16 15:50:35.043651 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.043597    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ecdd46d5-8e17-4b6e-a93e-db42d0530f67-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-74cc858bfd-ngwlg\" (UID: \"ecdd46d5-8e17-4b6e-a93e-db42d0530f67\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74cc858bfd-ngwlg"
Sep 16 15:50:35.043651 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.043643    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnxz4\" (UniqueName: \"kubernetes.io/projected/ecdd46d5-8e17-4b6e-a93e-db42d0530f67-kube-api-access-pnxz4\") pod \"managed-serviceaccount-addon-agent-74cc858bfd-ngwlg\" (UID: \"ecdd46d5-8e17-4b6e-a93e-db42d0530f67\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74cc858bfd-ngwlg"
Sep 16 15:50:35.043838 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.043674    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhwph\" (UniqueName: \"kubernetes.io/projected/8f22950d-c9c3-44bd-bd0c-56ae5fb22821-kube-api-access-mhwph\") pod \"dns-default-gb9f6\" (UID: \"8f22950d-c9c3-44bd-bd0c-56ae5fb22821\") " pod="openshift-dns/dns-default-gb9f6"
Sep 16 15:50:35.043838 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.043703    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-trusted-ca\") pod \"image-registry-98b6c5d9f-b5qrx\" (UID: \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\") " pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx"
Sep 16 15:50:35.043838 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.043749    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a899343-9e86-4def-b525-4586cb7b2afc-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg\" (UID: \"5a899343-9e86-4def-b525-4586cb7b2afc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg"
Sep 16 15:50:35.043838 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.043808    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-ca-trust-extracted\") pod \"image-registry-98b6c5d9f-b5qrx\" (UID: \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\") " pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx"
Sep 16 15:50:35.044069 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.043841    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjjv7\" (UniqueName: \"kubernetes.io/projected/83d036c1-52ce-4513-8ffa-d2e74def24c6-kube-api-access-cjjv7\") pod \"ingress-canary-g826v\" (UID: \"83d036c1-52ce-4513-8ffa-d2e74def24c6\") " pod="openshift-ingress-canary/ingress-canary-g826v"
Sep 16 15:50:35.044069 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.043886    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f22950d-c9c3-44bd-bd0c-56ae5fb22821-metrics-tls\") pod \"dns-default-gb9f6\" (UID: \"8f22950d-c9c3-44bd-bd0c-56ae5fb22821\") " pod="openshift-dns/dns-default-gb9f6"
Sep 16 15:50:35.044069 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.043908    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f22950d-c9c3-44bd-bd0c-56ae5fb22821-tmp-dir\") pod \"dns-default-gb9f6\" (UID: \"8f22950d-c9c3-44bd-bd0c-56ae5fb22821\") " pod="openshift-dns/dns-default-gb9f6"
Sep 16 15:50:35.044069 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.043936    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-image-registry-private-configuration\") pod \"image-registry-98b6c5d9f-b5qrx\" (UID: \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\") " pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx"
Sep 16 15:50:35.044069 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.043963    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-bound-sa-token\") pod \"image-registry-98b6c5d9f-b5qrx\" (UID: \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\") " pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx"
Sep 16 15:50:35.044069 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.043986    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb5d1a75-f8fc-4f48-9969-33e5126047f8-tmp\") pod \"klusterlet-addon-workmgr-b5557cb49-pp9jc\" (UID: \"bb5d1a75-f8fc-4f48-9969-33e5126047f8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b5557cb49-pp9jc"
Sep 16 15:50:35.044069 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.044014    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8f22950d-c9c3-44bd-bd0c-56ae5fb22821-config-volume\") pod \"dns-default-gb9f6\" (UID: \"8f22950d-c9c3-44bd-bd0c-56ae5fb22821\") " pod="openshift-dns/dns-default-gb9f6"
Sep 16 15:50:35.044069 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.044041    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/5a899343-9e86-4def-b525-4586cb7b2afc-ca\") pod \"cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg\" (UID: \"5a899343-9e86-4def-b525-4586cb7b2afc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg"
Sep 16 15:50:35.044439 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.044088    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5a899343-9e86-4def-b525-4586cb7b2afc-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg\" (UID: \"5a899343-9e86-4def-b525-4586cb7b2afc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg"
Sep 16 15:50:35.044439 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.044112    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bb5d1a75-f8fc-4f48-9969-33e5126047f8-klusterlet-config\") pod \"klusterlet-addon-workmgr-b5557cb49-pp9jc\" (UID: \"bb5d1a75-f8fc-4f48-9969-33e5126047f8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b5557cb49-pp9jc"
Sep 16 15:50:35.044439 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.044137    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83d036c1-52ce-4513-8ffa-d2e74def24c6-cert\") pod \"ingress-canary-g826v\" (UID: \"83d036c1-52ce-4513-8ffa-d2e74def24c6\") " pod="openshift-ingress-canary/ingress-canary-g826v"
Sep 16 15:50:35.044439 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.044167    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-ca-trust-extracted\") pod \"image-registry-98b6c5d9f-b5qrx\" (UID: \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\") " pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx"
Sep 16 15:50:35.044439 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.044171    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/5a899343-9e86-4def-b525-4586cb7b2afc-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg\" (UID: \"5a899343-9e86-4def-b525-4586cb7b2afc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg"
Sep 16 15:50:35.044439 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.044368    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-registry-tls\") pod \"image-registry-98b6c5d9f-b5qrx\" (UID: \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\") " pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx"
Sep 16 15:50:35.044439 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.044396    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqvct\" (UniqueName: \"kubernetes.io/projected/5a899343-9e86-4def-b525-4586cb7b2afc-kube-api-access-fqvct\") pod \"cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg\" (UID: \"5a899343-9e86-4def-b525-4586cb7b2afc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg"
Sep 16 15:50:35.044439 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.044438    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-registry-certificates\") pod \"image-registry-98b6c5d9f-b5qrx\" (UID: \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\") " pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx"
Sep 16 15:50:35.044869 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.044710    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-trusted-ca\") pod \"image-registry-98b6c5d9f-b5qrx\" (UID: \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\") " pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx"
Sep 16 15:50:35.044869 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.044775    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a899343-9e86-4def-b525-4586cb7b2afc-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg\" (UID: \"5a899343-9e86-4def-b525-4586cb7b2afc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg"
Sep 16 15:50:35.044869 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:35.044824    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 15:50:35.044869 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:35.044841    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-98b6c5d9f-b5qrx: secret "image-registry-tls" not found
Sep 16 15:50:35.045028 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:35.044908    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-registry-tls podName:bd87ab33-4283-417b-8da9-c1e86ad0fa9e nodeName:}" failed. No retries permitted until 2026-09-16 15:50:35.544890519 +0000 UTC m=+35.989582692 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-registry-tls") pod "image-registry-98b6c5d9f-b5qrx" (UID: "bd87ab33-4283-417b-8da9-c1e86ad0fa9e") : secret "image-registry-tls" not found
Sep 16 15:50:35.045028 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.044966    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-registry-certificates\") pod \"image-registry-98b6c5d9f-b5qrx\" (UID: \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\") " pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx"
Sep 16 15:50:35.045504 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.045405    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb5d1a75-f8fc-4f48-9969-33e5126047f8-tmp\") pod \"klusterlet-addon-workmgr-b5557cb49-pp9jc\" (UID: \"bb5d1a75-f8fc-4f48-9969-33e5126047f8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b5557cb49-pp9jc"
Sep 16 15:50:35.048893 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.048867    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/5a899343-9e86-4def-b525-4586cb7b2afc-ca\") pod \"cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg\" (UID: \"5a899343-9e86-4def-b525-4586cb7b2afc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg"
Sep 16 15:50:35.048992 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.048905    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ecdd46d5-8e17-4b6e-a93e-db42d0530f67-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-74cc858bfd-ngwlg\" (UID: \"ecdd46d5-8e17-4b6e-a93e-db42d0530f67\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74cc858bfd-ngwlg"
Sep 16 15:50:35.048992 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.048961    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-installation-pull-secrets\") pod \"image-registry-98b6c5d9f-b5qrx\" (UID: \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\") " pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx"
Sep 16 15:50:35.049296 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.049253    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/5a899343-9e86-4def-b525-4586cb7b2afc-hub\") pod \"cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg\" (UID: \"5a899343-9e86-4def-b525-4586cb7b2afc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg"
Sep 16 15:50:35.049913 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.049590    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-image-registry-private-configuration\") pod \"image-registry-98b6c5d9f-b5qrx\" (UID: \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\") " pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx"
Sep 16 15:50:35.049913 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.049776    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5a899343-9e86-4def-b525-4586cb7b2afc-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg\" (UID: \"5a899343-9e86-4def-b525-4586cb7b2afc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg"
Sep 16 15:50:35.050066 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.050020    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bb5d1a75-f8fc-4f48-9969-33e5126047f8-klusterlet-config\") pod \"klusterlet-addon-workmgr-b5557cb49-pp9jc\" (UID: \"bb5d1a75-f8fc-4f48-9969-33e5126047f8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b5557cb49-pp9jc"
Sep 16 15:50:35.050470 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.050446    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/5a899343-9e86-4def-b525-4586cb7b2afc-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg\" (UID: \"5a899343-9e86-4def-b525-4586cb7b2afc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg"
Sep 16 15:50:35.052828 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.052804    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnxz4\" (UniqueName: \"kubernetes.io/projected/ecdd46d5-8e17-4b6e-a93e-db42d0530f67-kube-api-access-pnxz4\") pod \"managed-serviceaccount-addon-agent-74cc858bfd-ngwlg\" (UID: \"ecdd46d5-8e17-4b6e-a93e-db42d0530f67\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74cc858bfd-ngwlg"
Sep 16 15:50:35.053517 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.053478    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wd2n\" (UniqueName: \"kubernetes.io/projected/bb5d1a75-f8fc-4f48-9969-33e5126047f8-kube-api-access-6wd2n\") pod \"klusterlet-addon-workmgr-b5557cb49-pp9jc\" (UID: \"bb5d1a75-f8fc-4f48-9969-33e5126047f8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b5557cb49-pp9jc"
Sep 16 15:50:35.053637 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.053614    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6bfr\" (UniqueName: \"kubernetes.io/projected/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-kube-api-access-f6bfr\") pod \"image-registry-98b6c5d9f-b5qrx\" (UID: \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\") " pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx"
Sep 16 15:50:35.054003 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.053983    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqvct\" (UniqueName: \"kubernetes.io/projected/5a899343-9e86-4def-b525-4586cb7b2afc-kube-api-access-fqvct\") pod \"cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg\" (UID: \"5a899343-9e86-4def-b525-4586cb7b2afc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg"
Sep 16 15:50:35.054428 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.054406    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-bound-sa-token\") pod \"image-registry-98b6c5d9f-b5qrx\" (UID: \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\") " pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx"
Sep 16 15:50:35.071411 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.071387    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vlhth"
Sep 16 15:50:35.071518 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.071412    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtdsz"
Sep 16 15:50:35.071586 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.071543    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wcrfk"
Sep 16 15:50:35.074494 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.074474    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 15:50:35.074649 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.074630    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 15:50:35.074649 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.074647    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mmzn4\""
Sep 16 15:50:35.074785 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.074765    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 15:50:35.074932 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.074918    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 15:50:35.074979 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.074957    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hlg9r\""
Sep 16 15:50:35.145595 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.145565    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f22950d-c9c3-44bd-bd0c-56ae5fb22821-metrics-tls\") pod \"dns-default-gb9f6\" (UID: \"8f22950d-c9c3-44bd-bd0c-56ae5fb22821\") " pod="openshift-dns/dns-default-gb9f6"
Sep 16 15:50:35.145742 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.145610    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f22950d-c9c3-44bd-bd0c-56ae5fb22821-tmp-dir\") pod \"dns-default-gb9f6\" (UID: \"8f22950d-c9c3-44bd-bd0c-56ae5fb22821\") " pod="openshift-dns/dns-default-gb9f6"
Sep 16 15:50:35.145742 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:35.145680    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 15:50:35.145903 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:35.145745    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f22950d-c9c3-44bd-bd0c-56ae5fb22821-metrics-tls podName:8f22950d-c9c3-44bd-bd0c-56ae5fb22821 nodeName:}" failed. No retries permitted until 2026-09-16 15:50:35.645723731 +0000 UTC m=+36.090415887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f22950d-c9c3-44bd-bd0c-56ae5fb22821-metrics-tls") pod "dns-default-gb9f6" (UID: "8f22950d-c9c3-44bd-bd0c-56ae5fb22821") : secret "dns-default-metrics-tls" not found
Sep 16 15:50:35.145903 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.145770    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8f22950d-c9c3-44bd-bd0c-56ae5fb22821-config-volume\") pod \"dns-default-gb9f6\" (UID: \"8f22950d-c9c3-44bd-bd0c-56ae5fb22821\") " pod="openshift-dns/dns-default-gb9f6"
Sep 16 15:50:35.145903 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.145855    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83d036c1-52ce-4513-8ffa-d2e74def24c6-cert\") pod \"ingress-canary-g826v\" (UID: \"83d036c1-52ce-4513-8ffa-d2e74def24c6\") " pod="openshift-ingress-canary/ingress-canary-g826v"
Sep 16 15:50:35.146056 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.145924    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f22950d-c9c3-44bd-bd0c-56ae5fb22821-tmp-dir\") pod \"dns-default-gb9f6\" (UID: \"8f22950d-c9c3-44bd-bd0c-56ae5fb22821\") " pod="openshift-dns/dns-default-gb9f6"
Sep 16 15:50:35.146056 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.145964    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhwph\" (UniqueName: \"kubernetes.io/projected/8f22950d-c9c3-44bd-bd0c-56ae5fb22821-kube-api-access-mhwph\") pod \"dns-default-gb9f6\" (UID: \"8f22950d-c9c3-44bd-bd0c-56ae5fb22821\") " pod="openshift-dns/dns-default-gb9f6"
Sep 16 15:50:35.146056 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.145992    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjjv7\" (UniqueName: \"kubernetes.io/projected/83d036c1-52ce-4513-8ffa-d2e74def24c6-kube-api-access-cjjv7\") pod \"ingress-canary-g826v\" (UID: \"83d036c1-52ce-4513-8ffa-d2e74def24c6\") " pod="openshift-ingress-canary/ingress-canary-g826v"
Sep 16 15:50:35.146276 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.146257    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8f22950d-c9c3-44bd-bd0c-56ae5fb22821-config-volume\") pod \"dns-default-gb9f6\" (UID: \"8f22950d-c9c3-44bd-bd0c-56ae5fb22821\") " pod="openshift-dns/dns-default-gb9f6"
Sep 16 15:50:35.148414 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.148396    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83d036c1-52ce-4513-8ffa-d2e74def24c6-cert\") pod \"ingress-canary-g826v\" (UID: \"83d036c1-52ce-4513-8ffa-d2e74def24c6\") " pod="openshift-ingress-canary/ingress-canary-g826v"
Sep 16 15:50:35.155627 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.155607    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhwph\" (UniqueName: \"kubernetes.io/projected/8f22950d-c9c3-44bd-bd0c-56ae5fb22821-kube-api-access-mhwph\") pod \"dns-default-gb9f6\" (UID: \"8f22950d-c9c3-44bd-bd0c-56ae5fb22821\") " pod="openshift-dns/dns-default-gb9f6"
Sep 16 15:50:35.155721 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.155667    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjjv7\" (UniqueName: \"kubernetes.io/projected/83d036c1-52ce-4513-8ffa-d2e74def24c6-kube-api-access-cjjv7\") pod \"ingress-canary-g826v\" (UID: \"83d036c1-52ce-4513-8ffa-d2e74def24c6\") " pod="openshift-ingress-canary/ingress-canary-g826v"
Sep 16 15:50:35.189743 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.189721    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74cc858bfd-ngwlg"
Sep 16 15:50:35.199979 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.199958    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b5557cb49-pp9jc"
Sep 16 15:50:35.207952 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.207786    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg"
Sep 16 15:50:35.230126 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.230089    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-g826v"
Sep 16 15:50:35.489412 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.489234    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74cc858bfd-ngwlg"]
Sep 16 15:50:35.493524 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.493497    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-b5557cb49-pp9jc"]
Sep 16 15:50:35.494372 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.494339    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg"]
Sep 16 15:50:35.494496 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:50:35.494476    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podecdd46d5_8e17_4b6e_a93e_db42d0530f67.slice/crio-cf45a92db0bf5207d4ba080a3dd6e510e677c8fa719237e0ce951950e8f59914 WatchSource:0}: Error finding container cf45a92db0bf5207d4ba080a3dd6e510e677c8fa719237e0ce951950e8f59914: Status 404 returned error can't find the container with id cf45a92db0bf5207d4ba080a3dd6e510e677c8fa719237e0ce951950e8f59914
Sep 16 15:50:35.497585 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:50:35.497566    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a899343_9e86_4def_b525_4586cb7b2afc.slice/crio-8eca84578af0c3a7cd1102546f3611deab949dfbed72ee3078d4ca21f1e67547 WatchSource:0}: Error finding container 8eca84578af0c3a7cd1102546f3611deab949dfbed72ee3078d4ca21f1e67547: Status 404 returned error can't find the container with id 8eca84578af0c3a7cd1102546f3611deab949dfbed72ee3078d4ca21f1e67547
Sep 16 15:50:35.498276 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:50:35.498250    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb5d1a75_f8fc_4f48_9969_33e5126047f8.slice/crio-a31c88d0da5214bd459bd48bf37a89e3eb42b50abdc049f42daf8698f0685070 WatchSource:0}: Error finding container a31c88d0da5214bd459bd48bf37a89e3eb42b50abdc049f42daf8698f0685070: Status 404 returned error can't find the container with id a31c88d0da5214bd459bd48bf37a89e3eb42b50abdc049f42daf8698f0685070
Sep 16 15:50:35.530850 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.530816    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-g826v"]
Sep 16 15:50:35.534006 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:50:35.533980    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83d036c1_52ce_4513_8ffa_d2e74def24c6.slice/crio-6176205d087c145cebcbd0ccdaa9ba416c9ab5b65f3750069567896738b3a1a3 WatchSource:0}: Error finding container 6176205d087c145cebcbd0ccdaa9ba416c9ab5b65f3750069567896738b3a1a3: Status 404 returned error can't find the container with id 6176205d087c145cebcbd0ccdaa9ba416c9ab5b65f3750069567896738b3a1a3
Sep 16 15:50:35.549538 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.549517    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-registry-tls\") pod \"image-registry-98b6c5d9f-b5qrx\" (UID: \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\") " pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx"
Sep 16 15:50:35.549652 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:35.549639    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 15:50:35.549694 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:35.549656    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-98b6c5d9f-b5qrx: secret "image-registry-tls" not found
Sep 16 15:50:35.549725 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:35.549704    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-registry-tls podName:bd87ab33-4283-417b-8da9-c1e86ad0fa9e nodeName:}" failed. No retries permitted until 2026-09-16 15:50:36.549686217 +0000 UTC m=+36.994378392 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-registry-tls") pod "image-registry-98b6c5d9f-b5qrx" (UID: "bd87ab33-4283-417b-8da9-c1e86ad0fa9e") : secret "image-registry-tls" not found
Sep 16 15:50:35.650955 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:35.650881    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f22950d-c9c3-44bd-bd0c-56ae5fb22821-metrics-tls\") pod \"dns-default-gb9f6\" (UID: \"8f22950d-c9c3-44bd-bd0c-56ae5fb22821\") " pod="openshift-dns/dns-default-gb9f6"
Sep 16 15:50:35.651117 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:35.651050    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 15:50:35.651176 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:35.651123    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f22950d-c9c3-44bd-bd0c-56ae5fb22821-metrics-tls podName:8f22950d-c9c3-44bd-bd0c-56ae5fb22821 nodeName:}" failed. No retries permitted until 2026-09-16 15:50:36.651103333 +0000 UTC m=+37.095795493 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f22950d-c9c3-44bd-bd0c-56ae5fb22821-metrics-tls") pod "dns-default-gb9f6" (UID: "8f22950d-c9c3-44bd-bd0c-56ae5fb22821") : secret "dns-default-metrics-tls" not found
Sep 16 15:50:36.281963 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:36.281899    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b5557cb49-pp9jc" event={"ID":"bb5d1a75-f8fc-4f48-9969-33e5126047f8","Type":"ContainerStarted","Data":"a31c88d0da5214bd459bd48bf37a89e3eb42b50abdc049f42daf8698f0685070"}
Sep 16 15:50:36.283253 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:36.283219    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg" event={"ID":"5a899343-9e86-4def-b525-4586cb7b2afc","Type":"ContainerStarted","Data":"8eca84578af0c3a7cd1102546f3611deab949dfbed72ee3078d4ca21f1e67547"}
Sep 16 15:50:36.284501 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:36.284474    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74cc858bfd-ngwlg" event={"ID":"ecdd46d5-8e17-4b6e-a93e-db42d0530f67","Type":"ContainerStarted","Data":"cf45a92db0bf5207d4ba080a3dd6e510e677c8fa719237e0ce951950e8f59914"}
Sep 16 15:50:36.285650 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:36.285625    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-g826v" event={"ID":"83d036c1-52ce-4513-8ffa-d2e74def24c6","Type":"ContainerStarted","Data":"6176205d087c145cebcbd0ccdaa9ba416c9ab5b65f3750069567896738b3a1a3"}
Sep 16 15:50:36.559279 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:36.559181    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-registry-tls\") pod \"image-registry-98b6c5d9f-b5qrx\" (UID: \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\") " pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx"
Sep 16 15:50:36.559426 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:36.559350    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 15:50:36.559426 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:36.559371    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-98b6c5d9f-b5qrx: secret "image-registry-tls" not found
Sep 16 15:50:36.559519 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:36.559440    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-registry-tls podName:bd87ab33-4283-417b-8da9-c1e86ad0fa9e nodeName:}" failed. No retries permitted until 2026-09-16 15:50:38.559423336 +0000 UTC m=+39.004115495 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-registry-tls") pod "image-registry-98b6c5d9f-b5qrx" (UID: "bd87ab33-4283-417b-8da9-c1e86ad0fa9e") : secret "image-registry-tls" not found
Sep 16 15:50:36.659979 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:36.659944    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f22950d-c9c3-44bd-bd0c-56ae5fb22821-metrics-tls\") pod \"dns-default-gb9f6\" (UID: \"8f22950d-c9c3-44bd-bd0c-56ae5fb22821\") " pod="openshift-dns/dns-default-gb9f6"
Sep 16 15:50:36.660138 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:36.660114    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 15:50:36.660204 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:36.660173    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f22950d-c9c3-44bd-bd0c-56ae5fb22821-metrics-tls podName:8f22950d-c9c3-44bd-bd0c-56ae5fb22821 nodeName:}" failed. No retries permitted until 2026-09-16 15:50:38.660158321 +0000 UTC m=+39.104850497 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f22950d-c9c3-44bd-bd0c-56ae5fb22821-metrics-tls") pod "dns-default-gb9f6" (UID: "8f22950d-c9c3-44bd-bd0c-56ae5fb22821") : secret "dns-default-metrics-tls" not found
Sep 16 15:50:37.140491 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:37.140456    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:50:37.227714 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:37.227683    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:50:37.290742 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:37.290711    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:50:38.295992 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:38.295782    2554 generic.go:356] "Generic (PLEG): container finished" podID="d8e64660-b4ae-45ae-b31f-1faee114f212" containerID="afae5e299070cab8852a42f685d50852085ecb3769763d94712e14f1f421a3ba" exitCode=0
Sep 16 15:50:38.295992 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:38.295869    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-65c8p" event={"ID":"d8e64660-b4ae-45ae-b31f-1faee114f212","Type":"ContainerDied","Data":"afae5e299070cab8852a42f685d50852085ecb3769763d94712e14f1f421a3ba"}
Sep 16 15:50:38.578703 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:38.578074    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-registry-tls\") pod \"image-registry-98b6c5d9f-b5qrx\" (UID: \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\") " pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx"
Sep 16 15:50:38.578703 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:38.578247    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 15:50:38.578703 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:38.578264    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-98b6c5d9f-b5qrx: secret "image-registry-tls" not found
Sep 16 15:50:38.578703 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:38.578324    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-registry-tls podName:bd87ab33-4283-417b-8da9-c1e86ad0fa9e nodeName:}" failed. No retries permitted until 2026-09-16 15:50:42.578307005 +0000 UTC m=+43.022999165 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-registry-tls") pod "image-registry-98b6c5d9f-b5qrx" (UID: "bd87ab33-4283-417b-8da9-c1e86ad0fa9e") : secret "image-registry-tls" not found
Sep 16 15:50:38.679764 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:38.679144    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f22950d-c9c3-44bd-bd0c-56ae5fb22821-metrics-tls\") pod \"dns-default-gb9f6\" (UID: \"8f22950d-c9c3-44bd-bd0c-56ae5fb22821\") " pod="openshift-dns/dns-default-gb9f6"
Sep 16 15:50:38.679764 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:38.679351    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 15:50:38.679764 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:38.679411    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f22950d-c9c3-44bd-bd0c-56ae5fb22821-metrics-tls podName:8f22950d-c9c3-44bd-bd0c-56ae5fb22821 nodeName:}" failed. No retries permitted until 2026-09-16 15:50:42.679393353 +0000 UTC m=+43.124085512 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f22950d-c9c3-44bd-bd0c-56ae5fb22821-metrics-tls") pod "dns-default-gb9f6" (UID: "8f22950d-c9c3-44bd-bd0c-56ae5fb22821") : secret "dns-default-metrics-tls" not found
Sep 16 15:50:39.303423 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:39.303321    2554 generic.go:356] "Generic (PLEG): container finished" podID="d8e64660-b4ae-45ae-b31f-1faee114f212" containerID="f9b82059b67a180062d83f50d5cb31a169ea5e8fa75d8877271885af30a3e245" exitCode=0
Sep 16 15:50:39.303423 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:39.303367    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-65c8p" event={"ID":"d8e64660-b4ae-45ae-b31f-1faee114f212","Type":"ContainerDied","Data":"f9b82059b67a180062d83f50d5cb31a169ea5e8fa75d8877271885af30a3e245"}
Sep 16 15:50:40.351720 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:40.351688    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:50:40.682462 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:40.682434    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:50:40.783029 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:40.782996    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:50:40.839435 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:40.839400    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:50:42.368544 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:42.368511    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:50:42.464669 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:42.464633    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:50:42.611543 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:42.611512    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-registry-tls\") pod \"image-registry-98b6c5d9f-b5qrx\" (UID: \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\") " pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx"
Sep 16 15:50:42.611719 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:42.611666    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 15:50:42.611719 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:42.611685    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-98b6c5d9f-b5qrx: secret "image-registry-tls" not found
Sep 16 15:50:42.611852 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:42.611751    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-registry-tls podName:bd87ab33-4283-417b-8da9-c1e86ad0fa9e nodeName:}" failed. No retries permitted until 2026-09-16 15:50:50.611731075 +0000 UTC m=+51.056423250 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-registry-tls") pod "image-registry-98b6c5d9f-b5qrx" (UID: "bd87ab33-4283-417b-8da9-c1e86ad0fa9e") : secret "image-registry-tls" not found
Sep 16 15:50:42.712617 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:42.712535    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f22950d-c9c3-44bd-bd0c-56ae5fb22821-metrics-tls\") pod \"dns-default-gb9f6\" (UID: \"8f22950d-c9c3-44bd-bd0c-56ae5fb22821\") " pod="openshift-dns/dns-default-gb9f6"
Sep 16 15:50:42.712829 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:42.712696    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 15:50:42.712829 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:42.712767    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f22950d-c9c3-44bd-bd0c-56ae5fb22821-metrics-tls podName:8f22950d-c9c3-44bd-bd0c-56ae5fb22821 nodeName:}" failed. No retries permitted until 2026-09-16 15:50:50.712746515 +0000 UTC m=+51.157438686 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f22950d-c9c3-44bd-bd0c-56ae5fb22821-metrics-tls") pod "dns-default-gb9f6" (UID: "8f22950d-c9c3-44bd-bd0c-56ae5fb22821") : secret "dns-default-metrics-tls" not found
Sep 16 15:50:45.319821 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:45.319721    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-g826v" event={"ID":"83d036c1-52ce-4513-8ffa-d2e74def24c6","Type":"ContainerStarted","Data":"d8b8da55c5011246af5b063e47940d0bf0f182aacb5fe988015983b34667897b"}
Sep 16 15:50:45.321777 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:45.321747    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b5557cb49-pp9jc" event={"ID":"bb5d1a75-f8fc-4f48-9969-33e5126047f8","Type":"ContainerStarted","Data":"0f9ec4b167185a527d43fae259d9bedfb9d4956d64a596a7b13327adda352b12"}
Sep 16 15:50:45.321945 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:45.321932    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b5557cb49-pp9jc"
Sep 16 15:50:45.323451 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:45.323430    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b5557cb49-pp9jc"
Sep 16 15:50:45.325904 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:45.325882    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-65c8p" event={"ID":"d8e64660-b4ae-45ae-b31f-1faee114f212","Type":"ContainerStarted","Data":"8dac9df961a403e61f468421a07fb9d0b2647aacdb944463d992a9d9814dd048"}
Sep 16 15:50:45.327324 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:45.327306    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74cc858bfd-ngwlg" event={"ID":"ecdd46d5-8e17-4b6e-a93e-db42d0530f67","Type":"ContainerStarted","Data":"cd6f962f047d3aceb535d4b445d1e29d9b2319362b9eb8dfd1b49a5048ccb2e2"}
Sep 16 15:50:45.335478 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:45.335442    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-g826v" podStartSLOduration=1.807044849 podStartE2EDuration="11.335431309s" totalImagesPullingTime="9.52838646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:50:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:50:45.33492587 +0000 UTC m=+45.779618049" watchObservedRunningTime="2026-09-16 15:50:45.335431309 +0000 UTC m=+45.780123484"
Sep 16 15:50:45.350752 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:45.350701    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74cc858bfd-ngwlg" podStartSLOduration=19.78342395 podStartE2EDuration="29.3506881s" totalImagesPullingTime="9.56726415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:50:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:50:45.349886991 +0000 UTC m=+45.794579172" watchObservedRunningTime="2026-09-16 15:50:45.3506881 +0000 UTC m=+45.795380278"
Sep 16 15:50:45.369768 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:45.369719    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-65c8p" podStartSLOduration=11.939817507 podStartE2EDuration="44.369699782s" totalImagesPullingTime="32.429882275s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 15:50:01 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:50:45.368639579 +0000 UTC m=+45.813331758" watchObservedRunningTime="2026-09-16 15:50:45.369699782 +0000 UTC m=+45.814391963"
Sep 16 15:50:45.389448 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:45.389399    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b5557cb49-pp9jc" podStartSLOduration=19.825713051 podStartE2EDuration="29.389382933s" totalImagesPullingTime="9.563669882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:50:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:50:45.388354262 +0000 UTC m=+45.833046441" watchObservedRunningTime="2026-09-16 15:50:45.389382933 +0000 UTC m=+45.834075110"
Sep 16 15:50:45.488986 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:45.488953    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:50:45.589547 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:45.589520    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:50:46.334099 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:46.334062    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg" event={"ID":"5a899343-9e86-4def-b525-4586cb7b2afc","Type":"ContainerStarted","Data":"9d572f586dcf748d0b5dae5a4dfc4f4c876f243bcafef6e90bbbc52958572c42"}
Sep 16 15:50:46.334547 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:46.334114    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg" event={"ID":"5a899343-9e86-4def-b525-4586cb7b2afc","Type":"ContainerStarted","Data":"e5eb7540c02bca625f509d50cfd37104a4ffbd68f34f5949e2b791da26f2c3c0"}
Sep 16 15:50:46.334547 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:46.334135    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg" event={"ID":"5a899343-9e86-4def-b525-4586cb7b2afc","Type":"ContainerStarted","Data":"0b7888fe9999c9df34f33edb7e556904cc56d3386e1fbf5efceb18472d19c0d2"}
Sep 16 15:50:46.360929 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:46.360884    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg" podStartSLOduration=20.285673507 podStartE2EDuration="30.360868352s" totalImagesPullingTime="10.075194845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:50:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:50:46.360107791 +0000 UTC m=+46.804799970" watchObservedRunningTime="2026-09-16 15:50:46.360868352 +0000 UTC m=+46.805560530"
Sep 16 15:50:50.672515 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:50.672477    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-registry-tls\") pod \"image-registry-98b6c5d9f-b5qrx\" (UID: \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\") " pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx"
Sep 16 15:50:50.672979 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:50.672655    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 15:50:50.672979 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:50.672683    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-98b6c5d9f-b5qrx: secret "image-registry-tls" not found
Sep 16 15:50:50.672979 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:50.672788    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-registry-tls podName:bd87ab33-4283-417b-8da9-c1e86ad0fa9e nodeName:}" failed. No retries permitted until 2026-09-16 15:51:06.672765809 +0000 UTC m=+67.117457980 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-registry-tls") pod "image-registry-98b6c5d9f-b5qrx" (UID: "bd87ab33-4283-417b-8da9-c1e86ad0fa9e") : secret "image-registry-tls" not found
Sep 16 15:50:50.773337 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:50:50.773309    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f22950d-c9c3-44bd-bd0c-56ae5fb22821-metrics-tls\") pod \"dns-default-gb9f6\" (UID: \"8f22950d-c9c3-44bd-bd0c-56ae5fb22821\") " pod="openshift-dns/dns-default-gb9f6"
Sep 16 15:50:50.773461 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:50.773413    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 15:50:50.773504 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:50:50.773469    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f22950d-c9c3-44bd-bd0c-56ae5fb22821-metrics-tls podName:8f22950d-c9c3-44bd-bd0c-56ae5fb22821 nodeName:}" failed. No retries permitted until 2026-09-16 15:51:06.773456229 +0000 UTC m=+67.218148384 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f22950d-c9c3-44bd-bd0c-56ae5fb22821-metrics-tls") pod "dns-default-gb9f6" (UID: "8f22950d-c9c3-44bd-bd0c-56ae5fb22821") : secret "dns-default-metrics-tls" not found
Sep 16 15:51:01.270277 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:51:01.270250    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w2qgw"
Sep 16 15:51:01.430382 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:51:01.430344    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:51:01.483841 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:51:01.483811    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:51:03.355673 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:51:03.355642    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:51:03.422387 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:51:03.422350    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:51:03.639287 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:51:03.639166    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:51:04.469892 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:51:04.469864    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:51:05.882926 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:51:05.882896    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2e625e0-e1fd-4733-97c0-2219660a6700-metrics-certs\") pod \"network-metrics-daemon-wtdsz\" (UID: \"b2e625e0-e1fd-4733-97c0-2219660a6700\") " pod="openshift-multus/network-metrics-daemon-wtdsz"
Sep 16 15:51:05.883348 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:51:05.882980    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0d31180-c1e0-48f6-a5a2-d1eb9ae02886-original-pull-secret\") pod \"global-pull-secret-syncer-wcrfk\" (UID: \"c0d31180-c1e0-48f6-a5a2-d1eb9ae02886\") " pod="kube-system/global-pull-secret-syncer-wcrfk"
Sep 16 15:51:05.886039 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:51:05.886021    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 15:51:05.886095 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:51:05.886085    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 15:51:05.893510 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:51:05.893493    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 15:51:05.893575 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:51:05.893561    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2e625e0-e1fd-4733-97c0-2219660a6700-metrics-certs podName:b2e625e0-e1fd-4733-97c0-2219660a6700 nodeName:}" failed. No retries permitted until 2026-09-16 15:52:09.893539812 +0000 UTC m=+130.338231982 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b2e625e0-e1fd-4733-97c0-2219660a6700-metrics-certs") pod "network-metrics-daemon-wtdsz" (UID: "b2e625e0-e1fd-4733-97c0-2219660a6700") : secret "metrics-daemon-secret" not found
Sep 16 15:51:05.896538 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:51:05.896520    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0d31180-c1e0-48f6-a5a2-d1eb9ae02886-original-pull-secret\") pod \"global-pull-secret-syncer-wcrfk\" (UID: \"c0d31180-c1e0-48f6-a5a2-d1eb9ae02886\") " pod="kube-system/global-pull-secret-syncer-wcrfk"
Sep 16 15:51:05.983844 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:51:05.983811    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q45qm\" (UniqueName: \"kubernetes.io/projected/25af0988-5e7d-4ee0-bdcb-cc956c6789ef-kube-api-access-q45qm\") pod \"network-check-target-vlhth\" (UID: \"25af0988-5e7d-4ee0-bdcb-cc956c6789ef\") " pod="openshift-network-diagnostics/network-check-target-vlhth"
Sep 16 15:51:05.986731 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:51:05.986714    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 15:51:05.990042 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:51:05.990028    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wcrfk"
Sep 16 15:51:05.996862 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:51:05.996844    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 15:51:06.007644 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:51:06.007620    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q45qm\" (UniqueName: \"kubernetes.io/projected/25af0988-5e7d-4ee0-bdcb-cc956c6789ef-kube-api-access-q45qm\") pod \"network-check-target-vlhth\" (UID: \"25af0988-5e7d-4ee0-bdcb-cc956c6789ef\") " pod="openshift-network-diagnostics/network-check-target-vlhth"
Sep 16 15:51:06.257890 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:51:06.257816    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wcrfk"]
Sep 16 15:51:06.261089 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:51:06.261065    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d31180_c1e0_48f6_a5a2_d1eb9ae02886.slice/crio-cbe294088ead6a40b597e312a85250adf61f617209e71230b4efb9c22fb60691 WatchSource:0}: Error finding container cbe294088ead6a40b597e312a85250adf61f617209e71230b4efb9c22fb60691: Status 404 returned error can't find the container with id cbe294088ead6a40b597e312a85250adf61f617209e71230b4efb9c22fb60691
Sep 16 15:51:06.285660 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:51:06.285639    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hlg9r\""
Sep 16 15:51:06.292953 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:51:06.292935    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vlhth"
Sep 16 15:51:06.395524 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:51:06.395487    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wcrfk" event={"ID":"c0d31180-c1e0-48f6-a5a2-d1eb9ae02886","Type":"ContainerStarted","Data":"cbe294088ead6a40b597e312a85250adf61f617209e71230b4efb9c22fb60691"}
Sep 16 15:51:06.558556 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:51:06.558524    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vlhth"]
Sep 16 15:51:06.563107 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:51:06.563078    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25af0988_5e7d_4ee0_bdcb_cc956c6789ef.slice/crio-bb1358fc47dc1c3c1b781f5907af5741f4e57dadb95503977e54d02d4def2b96 WatchSource:0}: Error finding container bb1358fc47dc1c3c1b781f5907af5741f4e57dadb95503977e54d02d4def2b96: Status 404 returned error can't find the container with id bb1358fc47dc1c3c1b781f5907af5741f4e57dadb95503977e54d02d4def2b96
Sep 16 15:51:06.688734 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:51:06.688694    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-registry-tls\") pod \"image-registry-98b6c5d9f-b5qrx\" (UID: \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\") " pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx"
Sep 16 15:51:06.688938 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:51:06.688885    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 15:51:06.688938 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:51:06.688904    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-98b6c5d9f-b5qrx: secret "image-registry-tls" not found
Sep 16 15:51:06.689018 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:51:06.688975    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-registry-tls podName:bd87ab33-4283-417b-8da9-c1e86ad0fa9e nodeName:}" failed. No retries permitted until 2026-09-16 15:51:38.688957559 +0000 UTC m=+99.133649725 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-registry-tls") pod "image-registry-98b6c5d9f-b5qrx" (UID: "bd87ab33-4283-417b-8da9-c1e86ad0fa9e") : secret "image-registry-tls" not found
Sep 16 15:51:06.789929 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:51:06.789896    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f22950d-c9c3-44bd-bd0c-56ae5fb22821-metrics-tls\") pod \"dns-default-gb9f6\" (UID: \"8f22950d-c9c3-44bd-bd0c-56ae5fb22821\") " pod="openshift-dns/dns-default-gb9f6"
Sep 16 15:51:06.790079 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:51:06.790026    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 15:51:06.790123 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:51:06.790089    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f22950d-c9c3-44bd-bd0c-56ae5fb22821-metrics-tls podName:8f22950d-c9c3-44bd-bd0c-56ae5fb22821 nodeName:}" failed. No retries permitted until 2026-09-16 15:51:38.790072611 +0000 UTC m=+99.234764769 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f22950d-c9c3-44bd-bd0c-56ae5fb22821-metrics-tls") pod "dns-default-gb9f6" (UID: "8f22950d-c9c3-44bd-bd0c-56ae5fb22821") : secret "dns-default-metrics-tls" not found
Sep 16 15:51:07.399226 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:51:07.399195    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vlhth" event={"ID":"25af0988-5e7d-4ee0-bdcb-cc956c6789ef","Type":"ContainerStarted","Data":"bb1358fc47dc1c3c1b781f5907af5741f4e57dadb95503977e54d02d4def2b96"}
Sep 16 15:51:12.418307 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:51:12.418267    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wcrfk" event={"ID":"c0d31180-c1e0-48f6-a5a2-d1eb9ae02886","Type":"ContainerStarted","Data":"29841b28a177bf027892fb35d9b16c51f02c2ef068060f8dfb35d00261b42f3f"}
Sep 16 15:51:12.419747 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:51:12.419723    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vlhth" event={"ID":"25af0988-5e7d-4ee0-bdcb-cc956c6789ef","Type":"ContainerStarted","Data":"989b7f55fddf675320c472a2e0a08cb0a92757f54e612bcc9dfc155ceadec3d7"}
Sep 16 15:51:12.419886 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:51:12.419854    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vlhth"
Sep 16 15:51:12.435925 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:51:12.435880    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-wcrfk" podStartSLOduration=66.323512211 podStartE2EDuration="1m11.435866155s" totalImagesPullingTime="5.112353944s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 15:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:51:12.434153745 +0000 UTC m=+72.878845924" watchObservedRunningTime="2026-09-16 15:51:12.435866155 +0000 UTC m=+72.880558332"
Sep 16 15:51:12.452965 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:51:12.452546    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vlhth" podStartSLOduration=66.64577039 podStartE2EDuration="1m11.45252504s" totalImagesPullingTime="4.80675465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:51:12.452087916 +0000 UTC m=+72.896780095" watchObservedRunningTime="2026-09-16 15:51:12.45252504 +0000 UTC m=+72.897217218"
Sep 16 15:51:21.173478 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:51:21.173442    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:51:38.709033 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:51:38.708904    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-registry-tls\") pod \"image-registry-98b6c5d9f-b5qrx\" (UID: \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\") " pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx"
Sep 16 15:51:38.709508 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:51:38.709064    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 15:51:38.709508 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:51:38.709084    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-98b6c5d9f-b5qrx: secret "image-registry-tls" not found
Sep 16 15:51:38.709508 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:51:38.709151    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-registry-tls podName:bd87ab33-4283-417b-8da9-c1e86ad0fa9e nodeName:}" failed. No retries permitted until 2026-09-16 15:52:42.709133478 +0000 UTC m=+163.153825634 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-registry-tls") pod "image-registry-98b6c5d9f-b5qrx" (UID: "bd87ab33-4283-417b-8da9-c1e86ad0fa9e") : secret "image-registry-tls" not found
Sep 16 15:51:38.809963 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:51:38.809928    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f22950d-c9c3-44bd-bd0c-56ae5fb22821-metrics-tls\") pod \"dns-default-gb9f6\" (UID: \"8f22950d-c9c3-44bd-bd0c-56ae5fb22821\") " pod="openshift-dns/dns-default-gb9f6"
Sep 16 15:51:38.810081 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:51:38.810036    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 15:51:38.810117 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:51:38.810108    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f22950d-c9c3-44bd-bd0c-56ae5fb22821-metrics-tls podName:8f22950d-c9c3-44bd-bd0c-56ae5fb22821 nodeName:}" failed. No retries permitted until 2026-09-16 15:52:42.810094835 +0000 UTC m=+163.254786991 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f22950d-c9c3-44bd-bd0c-56ae5fb22821-metrics-tls") pod "dns-default-gb9f6" (UID: "8f22950d-c9c3-44bd-bd0c-56ae5fb22821") : secret "dns-default-metrics-tls" not found
Sep 16 15:51:43.425609 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:51:43.425574    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vlhth"
Sep 16 15:52:06.187956 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:06.187919    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:07.552140 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:07.552109    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:08.850951 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:08.850916    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:09.933387 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:09.933344    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2e625e0-e1fd-4733-97c0-2219660a6700-metrics-certs\") pod \"network-metrics-daemon-wtdsz\" (UID: \"b2e625e0-e1fd-4733-97c0-2219660a6700\") " pod="openshift-multus/network-metrics-daemon-wtdsz"
Sep 16 15:52:09.933870 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:52:09.933528    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 15:52:09.933870 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:52:09.933607    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2e625e0-e1fd-4733-97c0-2219660a6700-metrics-certs podName:b2e625e0-e1fd-4733-97c0-2219660a6700 nodeName:}" failed. No retries permitted until 2026-09-16 15:54:11.933588463 +0000 UTC m=+252.378280639 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b2e625e0-e1fd-4733-97c0-2219660a6700-metrics-certs") pod "network-metrics-daemon-wtdsz" (UID: "b2e625e0-e1fd-4733-97c0-2219660a6700") : secret "metrics-daemon-secret" not found
Sep 16 15:52:10.563910 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:10.563875    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:10.662886 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:10.662853    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:10.763221 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:10.763192    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:11.057096 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:11.057067    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-62g56"]
Sep 16 15:52:11.059035 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:11.059018    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-62g56"
Sep 16 15:52:11.062071 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:11.062048    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 15:52:11.062180 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:11.062107    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 15:52:11.062246 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:11.062231    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 15:52:11.062298 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:11.062243    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 15:52:11.062365 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:11.062348    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ckzgw\""
Sep 16 15:52:11.072228 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:11.072206    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-62g56"]
Sep 16 15:52:11.076951 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:11.076910    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:11.134909 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:11.134879    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:11.200491 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:11.200463    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:11.242720 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:11.242691    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-data-volume\") pod \"insights-runtime-extractor-62g56\" (UID: \"bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a\") " pod="openshift-insights/insights-runtime-extractor-62g56"
Sep 16 15:52:11.242860 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:11.242734    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-62g56\" (UID: \"bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a\") " pod="openshift-insights/insights-runtime-extractor-62g56"
Sep 16 15:52:11.242860 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:11.242778    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c897p\" (UniqueName: \"kubernetes.io/projected/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-kube-api-access-c897p\") pod \"insights-runtime-extractor-62g56\" (UID: \"bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a\") " pod="openshift-insights/insights-runtime-extractor-62g56"
Sep 16 15:52:11.242860 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:11.242811    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-crio-socket\") pod \"insights-runtime-extractor-62g56\" (UID: \"bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a\") " pod="openshift-insights/insights-runtime-extractor-62g56"
Sep 16 15:52:11.242860 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:11.242843    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-62g56\" (UID: \"bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a\") " pod="openshift-insights/insights-runtime-extractor-62g56"
Sep 16 15:52:11.344241 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:11.344173    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-data-volume\") pod \"insights-runtime-extractor-62g56\" (UID: \"bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a\") " pod="openshift-insights/insights-runtime-extractor-62g56"
Sep 16 15:52:11.344241 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:11.344213    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-62g56\" (UID: \"bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a\") " pod="openshift-insights/insights-runtime-extractor-62g56"
Sep 16 15:52:11.344387 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:11.344263    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c897p\" (UniqueName: \"kubernetes.io/projected/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-kube-api-access-c897p\") pod \"insights-runtime-extractor-62g56\" (UID: \"bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a\") " pod="openshift-insights/insights-runtime-extractor-62g56"
Sep 16 15:52:11.344387 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:11.344285    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-crio-socket\") pod \"insights-runtime-extractor-62g56\" (UID: \"bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a\") " pod="openshift-insights/insights-runtime-extractor-62g56"
Sep 16 15:52:11.344387 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:11.344327    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-62g56\" (UID: \"bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a\") " pod="openshift-insights/insights-runtime-extractor-62g56"
Sep 16 15:52:11.344532 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:52:11.344396    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 15:52:11.344532 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:11.344408    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-crio-socket\") pod \"insights-runtime-extractor-62g56\" (UID: \"bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a\") " pod="openshift-insights/insights-runtime-extractor-62g56"
Sep 16 15:52:11.344532 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:52:11.344418    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 15:52:11.344532 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:52:11.344514    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-kube-rbac-proxy-cm podName:bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a nodeName:}" failed. No retries permitted until 2026-09-16 15:52:11.844491017 +0000 UTC m=+132.289183198 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-62g56" (UID: "bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a") : configmap "kube-rbac-proxy" not found
Sep 16 15:52:11.344715 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:52:11.344535    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-insights-runtime-extractor-tls podName:bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a nodeName:}" failed. No retries permitted until 2026-09-16 15:52:11.844525427 +0000 UTC m=+132.289217586 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-62g56" (UID: "bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a") : secret "insights-runtime-extractor-tls" not found
Sep 16 15:52:11.344715 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:11.344576    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-data-volume\") pod \"insights-runtime-extractor-62g56\" (UID: \"bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a\") " pod="openshift-insights/insights-runtime-extractor-62g56"
Sep 16 15:52:11.353088 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:11.353060    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c897p\" (UniqueName: \"kubernetes.io/projected/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-kube-api-access-c897p\") pod \"insights-runtime-extractor-62g56\" (UID: \"bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a\") " pod="openshift-insights/insights-runtime-extractor-62g56"
Sep 16 15:52:11.434668 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:11.434615    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:11.487401 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:11.487372    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:11.599572 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:11.599505    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:11.654824 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:11.654778    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:11.848688 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:11.848655    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-62g56\" (UID: \"bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a\") " pod="openshift-insights/insights-runtime-extractor-62g56"
Sep 16 15:52:11.848923 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:11.848735    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-62g56\" (UID: \"bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a\") " pod="openshift-insights/insights-runtime-extractor-62g56"
Sep 16 15:52:11.848923 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:52:11.848770    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 15:52:11.848923 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:52:11.848876    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-kube-rbac-proxy-cm podName:bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a nodeName:}" failed. No retries permitted until 2026-09-16 15:52:12.848860083 +0000 UTC m=+133.293552240 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-62g56" (UID: "bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a") : configmap "kube-rbac-proxy" not found
Sep 16 15:52:11.848923 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:52:11.848876    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 15:52:11.848923 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:52:11.848909    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-insights-runtime-extractor-tls podName:bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a nodeName:}" failed. No retries permitted until 2026-09-16 15:52:12.848903375 +0000 UTC m=+133.293595563 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-62g56" (UID: "bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a") : secret "insights-runtime-extractor-tls" not found
Sep 16 15:52:12.855924 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:12.855872    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-62g56\" (UID: \"bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a\") " pod="openshift-insights/insights-runtime-extractor-62g56"
Sep 16 15:52:12.856332 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:12.855988    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-62g56\" (UID: \"bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a\") " pod="openshift-insights/insights-runtime-extractor-62g56"
Sep 16 15:52:12.856332 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:52:12.856039    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 15:52:12.856332 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:52:12.856106    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-insights-runtime-extractor-tls podName:bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a nodeName:}" failed. No retries permitted until 2026-09-16 15:52:14.856090448 +0000 UTC m=+135.300782604 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-62g56" (UID: "bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a") : secret "insights-runtime-extractor-tls" not found
Sep 16 15:52:12.856332 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:52:12.856122    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 15:52:12.856332 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:52:12.856182    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-kube-rbac-proxy-cm podName:bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a nodeName:}" failed. No retries permitted until 2026-09-16 15:52:14.856167173 +0000 UTC m=+135.300859349 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-62g56" (UID: "bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a") : configmap "kube-rbac-proxy" not found
Sep 16 15:52:13.031286 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:13.031260    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ffp62_ef023c4c-0a32-416e-8a44-f8e8e4302e5f/dns-node-resolver/0.log"
Sep 16 15:52:13.058301 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:13.058275    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:13.113147 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:13.113087    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:13.829607 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:13.829581    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fzl79_7b3420ee-3d02-4cbf-a6b2-fc58c2da0298/node-ca/0.log"
Sep 16 15:52:13.873777 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:13.873749    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:14.629425 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:14.629389    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-g826v_83d036c1-52ce-4513-8ffa-d2e74def24c6/serve-healthcheck-canary/0.log"
Sep 16 15:52:14.871517 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:14.871479    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-62g56\" (UID: \"bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a\") " pod="openshift-insights/insights-runtime-extractor-62g56"
Sep 16 15:52:14.871703 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:14.871565    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-62g56\" (UID: \"bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a\") " pod="openshift-insights/insights-runtime-extractor-62g56"
Sep 16 15:52:14.871703 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:52:14.871639    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 15:52:14.871854 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:52:14.871716    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-insights-runtime-extractor-tls podName:bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a nodeName:}" failed. No retries permitted until 2026-09-16 15:52:18.871698582 +0000 UTC m=+139.316390766 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-62g56" (UID: "bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a") : secret "insights-runtime-extractor-tls" not found
Sep 16 15:52:14.871854 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:52:14.871643    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 15:52:14.871854 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:52:14.871828    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-kube-rbac-proxy-cm podName:bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a nodeName:}" failed. No retries permitted until 2026-09-16 15:52:18.871788722 +0000 UTC m=+139.316480879 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-62g56" (UID: "bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a") : configmap "kube-rbac-proxy" not found
Sep 16 15:52:15.473703 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:15.473672    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:15.630084 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:15.630034    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:15.832221 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:15.832192    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:16.092985 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:16.092900    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:16.183139 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:16.183111    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:16.592644 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:16.592618    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:16.651191 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:16.651163    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:16.992134 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:16.992062    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:17.358746 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:17.358715    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:17.960200 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:17.960167    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:18.164943 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:18.164909    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:18.566621 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:18.566589    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:18.759976 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:18.759946    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:18.902092 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:18.902005    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-62g56\" (UID: \"bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a\") " pod="openshift-insights/insights-runtime-extractor-62g56"
Sep 16 15:52:18.902092 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:18.902063    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-62g56\" (UID: \"bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a\") " pod="openshift-insights/insights-runtime-extractor-62g56"
Sep 16 15:52:18.902281 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:52:18.902130    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 15:52:18.902281 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:52:18.902172    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 15:52:18.902281 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:52:18.902191    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-kube-rbac-proxy-cm podName:bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a nodeName:}" failed. No retries permitted until 2026-09-16 15:52:26.902176319 +0000 UTC m=+147.346868502 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-62g56" (UID: "bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a") : configmap "kube-rbac-proxy" not found
Sep 16 15:52:18.902281 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:52:18.902211    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-insights-runtime-extractor-tls podName:bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a nodeName:}" failed. No retries permitted until 2026-09-16 15:52:26.902200503 +0000 UTC m=+147.346892679 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-62g56" (UID: "bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a") : secret "insights-runtime-extractor-tls" not found
Sep 16 15:52:19.341552 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:19.341519    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:19.441614 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:19.441583    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:19.563586 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:19.563553    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:20.158502 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:20.158471    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:20.213565 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:20.213531    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:20.767803 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:20.767766    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:21.342529 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:21.342500    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:21.403532 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:21.403478    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:21.556264 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:21.556234    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:21.868747 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:21.868719    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:22.159155 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:22.159082    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:22.763492 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:22.763463    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:22.962598 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:22.962568    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:23.155063 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:23.155038    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:23.561635 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:23.561610    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:23.747203 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:23.747175    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:23.959655 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:23.959561    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:24.775923 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:24.775895    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:24.956339 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:24.956295    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:25.355331 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:25.355298    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:25.566022 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:25.565985    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:25.799620 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:25.799583    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:25.860462 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:25.860434    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:25.990259 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:25.990217    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:26.045904 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:26.045872    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:26.159335 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:26.159242    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:26.976909 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:26.976864    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-62g56\" (UID: \"bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a\") " pod="openshift-insights/insights-runtime-extractor-62g56"
Sep 16 15:52:26.977295 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:26.976936    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-62g56\" (UID: \"bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a\") " pod="openshift-insights/insights-runtime-extractor-62g56"
Sep 16 15:52:26.977397 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:26.977379    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-62g56\" (UID: \"bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a\") " pod="openshift-insights/insights-runtime-extractor-62g56"
Sep 16 15:52:26.979187 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:26.979162    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-62g56\" (UID: \"bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a\") " pod="openshift-insights/insights-runtime-extractor-62g56"
Sep 16 15:52:27.267840 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:27.267738    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-62g56"
Sep 16 15:52:27.537885 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:27.537855    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-62g56"]
Sep 16 15:52:27.540980 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:52:27.540953    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc0e5d11_ed3b_4c32_996a_35bbc8ffc88a.slice/crio-2f2cef69489b477f4a3b9e66c7c69aeb00cbbdc7414e97608c992e4bd5ca9c17 WatchSource:0}: Error finding container 2f2cef69489b477f4a3b9e66c7c69aeb00cbbdc7414e97608c992e4bd5ca9c17: Status 404 returned error can't find the container with id 2f2cef69489b477f4a3b9e66c7c69aeb00cbbdc7414e97608c992e4bd5ca9c17
Sep 16 15:52:27.639694 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:27.639661    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-62g56" event={"ID":"bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a","Type":"ContainerStarted","Data":"42d53319d0dc06cae517531f91899c7fb78bdb985b2ed891504f109a2d87666c"}
Sep 16 15:52:27.639845 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:27.639743    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-62g56" event={"ID":"bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a","Type":"ContainerStarted","Data":"2f2cef69489b477f4a3b9e66c7c69aeb00cbbdc7414e97608c992e4bd5ca9c17"}
Sep 16 15:52:28.645010 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:28.644976    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-62g56" event={"ID":"bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a","Type":"ContainerStarted","Data":"3868a1e6ec5b0d4c6bcf8a439928d9c355af2645b563a1f06bd45cc6a212951b"}
Sep 16 15:52:29.650104 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:29.650024    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-62g56" event={"ID":"bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a","Type":"ContainerStarted","Data":"e4010fea31afa24743a5c5253cc00cb0f5f0f404b8ba39067bae1ac2be5a757b"}
Sep 16 15:52:29.668373 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:29.668325    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-62g56" podStartSLOduration=16.927209153 podStartE2EDuration="18.668311579s" totalImagesPullingTime="1.741102426s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 15:52:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:52:29.66808692 +0000 UTC m=+150.112779097" watchObservedRunningTime="2026-09-16 15:52:29.668311579 +0000 UTC m=+150.113003757"
Sep 16 15:52:36.434215 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:36.434180    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:36.524699 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:36.524666    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:37.876232 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:52:37.876191    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx" podUID="bd87ab33-4283-417b-8da9-c1e86ad0fa9e"
Sep 16 15:52:37.923692 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:52:37.923651    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-gb9f6" podUID="8f22950d-c9c3-44bd-bd0c-56ae5fb22821"
Sep 16 15:52:38.094966 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:52:38.094926    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-wtdsz" podUID="b2e625e0-e1fd-4733-97c0-2219660a6700"
Sep 16 15:52:38.677675 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:38.677639    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx"
Sep 16 15:52:38.677675 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:38.677681    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gb9f6"
Sep 16 15:52:39.010542 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:39.010510    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:39.081220 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:39.081191    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:40.314667 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:40.314632    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:40.415112 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:40.415080    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:40.486047 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:40.486017    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:40.714580 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:40.714507    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:42.809281 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:42.809250    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-registry-tls\") pod \"image-registry-98b6c5d9f-b5qrx\" (UID: \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\") " pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx"
Sep 16 15:52:42.811487 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:42.811459    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-registry-tls\") pod \"image-registry-98b6c5d9f-b5qrx\" (UID: \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\") " pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx"
Sep 16 15:52:42.881143 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:42.881116    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-c92pf\""
Sep 16 15:52:42.889417 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:42.889387    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx"
Sep 16 15:52:42.910519 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:42.910494    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f22950d-c9c3-44bd-bd0c-56ae5fb22821-metrics-tls\") pod \"dns-default-gb9f6\" (UID: \"8f22950d-c9c3-44bd-bd0c-56ae5fb22821\") " pod="openshift-dns/dns-default-gb9f6"
Sep 16 15:52:42.912665 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:42.912644    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f22950d-c9c3-44bd-bd0c-56ae5fb22821-metrics-tls\") pod \"dns-default-gb9f6\" (UID: \"8f22950d-c9c3-44bd-bd0c-56ae5fb22821\") " pod="openshift-dns/dns-default-gb9f6"
Sep 16 15:52:43.160946 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:43.160875    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-98b6c5d9f-b5qrx"]
Sep 16 15:52:43.164339 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:52:43.164309    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd87ab33_4283_417b_8da9_c1e86ad0fa9e.slice/crio-58f70556102d606808d0f5c21fa8be9ad9c7b82c6c690f8ece1e63d37ae7701f WatchSource:0}: Error finding container 58f70556102d606808d0f5c21fa8be9ad9c7b82c6c690f8ece1e63d37ae7701f: Status 404 returned error can't find the container with id 58f70556102d606808d0f5c21fa8be9ad9c7b82c6c690f8ece1e63d37ae7701f
Sep 16 15:52:43.181486 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:43.181467    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-thq8g\""
Sep 16 15:52:43.189232 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:43.189214    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gb9f6"
Sep 16 15:52:43.459479 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:43.459401    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gb9f6"]
Sep 16 15:52:43.462662 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:52:43.462631    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f22950d_c9c3_44bd_bd0c_56ae5fb22821.slice/crio-3bec1a28afdde8a5b681e504556fe39212aa38395fa0268a530562eace57d3dd WatchSource:0}: Error finding container 3bec1a28afdde8a5b681e504556fe39212aa38395fa0268a530562eace57d3dd: Status 404 returned error can't find the container with id 3bec1a28afdde8a5b681e504556fe39212aa38395fa0268a530562eace57d3dd
Sep 16 15:52:43.694492 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:43.694458    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gb9f6" event={"ID":"8f22950d-c9c3-44bd-bd0c-56ae5fb22821","Type":"ContainerStarted","Data":"3bec1a28afdde8a5b681e504556fe39212aa38395fa0268a530562eace57d3dd"}
Sep 16 15:52:43.695922 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:43.695899    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx" event={"ID":"bd87ab33-4283-417b-8da9-c1e86ad0fa9e","Type":"ContainerStarted","Data":"43903512fb9c1d24ab632f1b7f846a9d0b0923b81793a7947b63f8db738662d5"}
Sep 16 15:52:43.696022 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:43.695940    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx" event={"ID":"bd87ab33-4283-417b-8da9-c1e86ad0fa9e","Type":"ContainerStarted","Data":"58f70556102d606808d0f5c21fa8be9ad9c7b82c6c690f8ece1e63d37ae7701f"}
Sep 16 15:52:43.696062 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:43.696038    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx"
Sep 16 15:52:43.719872 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:43.719780    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx" podStartSLOduration=162.719767062 podStartE2EDuration="2m42.719767062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:50:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:52:43.718276059 +0000 UTC m=+164.162968255" watchObservedRunningTime="2026-09-16 15:52:43.719767062 +0000 UTC m=+164.164459239"
Sep 16 15:52:44.517064 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:44.517029    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:44.593781 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:44.593745    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:45.322854 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:45.322777    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b5557cb49-pp9jc" podUID="bb5d1a75-f8fc-4f48-9969-33e5126047f8" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.9:8000/readyz\": dial tcp 10.132.0.9:8000: connect: connection refused"
Sep 16 15:52:45.705147 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:45.705117    2554 generic.go:356] "Generic (PLEG): container finished" podID="ecdd46d5-8e17-4b6e-a93e-db42d0530f67" containerID="cd6f962f047d3aceb535d4b445d1e29d9b2319362b9eb8dfd1b49a5048ccb2e2" exitCode=255
Sep 16 15:52:45.705528 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:45.705190    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74cc858bfd-ngwlg" event={"ID":"ecdd46d5-8e17-4b6e-a93e-db42d0530f67","Type":"ContainerDied","Data":"cd6f962f047d3aceb535d4b445d1e29d9b2319362b9eb8dfd1b49a5048ccb2e2"}
Sep 16 15:52:45.705588 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:45.705559    2554 scope.go:117] "RemoveContainer" containerID="cd6f962f047d3aceb535d4b445d1e29d9b2319362b9eb8dfd1b49a5048ccb2e2"
Sep 16 15:52:45.707080 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:45.707057    2554 generic.go:356] "Generic (PLEG): container finished" podID="bb5d1a75-f8fc-4f48-9969-33e5126047f8" containerID="0f9ec4b167185a527d43fae259d9bedfb9d4956d64a596a7b13327adda352b12" exitCode=1
Sep 16 15:52:45.707180 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:45.707089    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b5557cb49-pp9jc" event={"ID":"bb5d1a75-f8fc-4f48-9969-33e5126047f8","Type":"ContainerDied","Data":"0f9ec4b167185a527d43fae259d9bedfb9d4956d64a596a7b13327adda352b12"}
Sep 16 15:52:45.707467 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:45.707452    2554 scope.go:117] "RemoveContainer" containerID="0f9ec4b167185a527d43fae259d9bedfb9d4956d64a596a7b13327adda352b12"
Sep 16 15:52:46.712424 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:46.712383    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74cc858bfd-ngwlg" event={"ID":"ecdd46d5-8e17-4b6e-a93e-db42d0530f67","Type":"ContainerStarted","Data":"33f0ca03ec857f58a117e28727a11331c96fe8c403b5378d383e06cceb9766d4"}
Sep 16 15:52:46.714178 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:46.714156    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b5557cb49-pp9jc" event={"ID":"bb5d1a75-f8fc-4f48-9969-33e5126047f8","Type":"ContainerStarted","Data":"3aeeeeab566d8de34c34019c1d5b0617029fb800e3342c8ca46cb024babe5c9b"}
Sep 16 15:52:46.714450 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:46.714435    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b5557cb49-pp9jc"
Sep 16 15:52:46.715106 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:46.715087    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b5557cb49-pp9jc"
Sep 16 15:52:46.715966 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:46.715948    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gb9f6" event={"ID":"8f22950d-c9c3-44bd-bd0c-56ae5fb22821","Type":"ContainerStarted","Data":"c0e71e52693e148edb79fbfd9cf7e8f666cbca39d73ab65c76de437225315c75"}
Sep 16 15:52:46.716050 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:46.715980    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gb9f6" event={"ID":"8f22950d-c9c3-44bd-bd0c-56ae5fb22821","Type":"ContainerStarted","Data":"78e6617dcf7ab933d897e43da8b5a06d5459db6026d245a812950e1fc29cfe95"}
Sep 16 15:52:46.716096 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:46.716078    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gb9f6"
Sep 16 15:52:46.743874 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:46.743825    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-gb9f6" podStartSLOduration=130.534632223 podStartE2EDuration="2m12.743809234s" totalImagesPullingTime="2.209177011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:50:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:52:46.743153597 +0000 UTC m=+167.187845786" watchObservedRunningTime="2026-09-16 15:52:46.743809234 +0000 UTC m=+167.188501403"
Sep 16 15:52:46.832951 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:46.832898    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:48.655766 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:48.655732    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:48.873890 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:48.873859    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:48.979882 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:48.977786    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fsvlc"]
Sep 16 15:52:48.984007 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:48.983983    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fsvlc"
Sep 16 15:52:48.987368 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:48.987347    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 15:52:48.987687 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:48.987667    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 15:52:48.988757 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:48.988735    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 15:52:48.988890 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:48.988755    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fzfxs\""
Sep 16 15:52:48.988890 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:48.988786    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 15:52:48.988890 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:48.988819    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 15:52:48.989071 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:48.989059    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 15:52:49.020293 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:49.020268    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:49.064850 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:49.064825    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23df07fb-e55c-416b-a43e-adb9b81fe3b3-sys\") pod \"node-exporter-fsvlc\" (UID: \"23df07fb-e55c-416b-a43e-adb9b81fe3b3\") " pod="openshift-monitoring/node-exporter-fsvlc"
Sep 16 15:52:49.064995 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:49.064863    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/23df07fb-e55c-416b-a43e-adb9b81fe3b3-node-exporter-textfile\") pod \"node-exporter-fsvlc\" (UID: \"23df07fb-e55c-416b-a43e-adb9b81fe3b3\") " pod="openshift-monitoring/node-exporter-fsvlc"
Sep 16 15:52:49.064995 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:49.064885    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/23df07fb-e55c-416b-a43e-adb9b81fe3b3-node-exporter-accelerators-collector-config\") pod \"node-exporter-fsvlc\" (UID: \"23df07fb-e55c-416b-a43e-adb9b81fe3b3\") " pod="openshift-monitoring/node-exporter-fsvlc"
Sep 16 15:52:49.064995 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:49.064907    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/23df07fb-e55c-416b-a43e-adb9b81fe3b3-root\") pod \"node-exporter-fsvlc\" (UID: \"23df07fb-e55c-416b-a43e-adb9b81fe3b3\") " pod="openshift-monitoring/node-exporter-fsvlc"
Sep 16 15:52:49.064995 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:49.064944    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/23df07fb-e55c-416b-a43e-adb9b81fe3b3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fsvlc\" (UID: \"23df07fb-e55c-416b-a43e-adb9b81fe3b3\") " pod="openshift-monitoring/node-exporter-fsvlc"
Sep 16 15:52:49.065127 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:49.064994    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23df07fb-e55c-416b-a43e-adb9b81fe3b3-metrics-client-ca\") pod \"node-exporter-fsvlc\" (UID: \"23df07fb-e55c-416b-a43e-adb9b81fe3b3\") " pod="openshift-monitoring/node-exporter-fsvlc"
Sep 16 15:52:49.065127 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:49.065032    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq9zh\" (UniqueName: \"kubernetes.io/projected/23df07fb-e55c-416b-a43e-adb9b81fe3b3-kube-api-access-hq9zh\") pod \"node-exporter-fsvlc\" (UID: \"23df07fb-e55c-416b-a43e-adb9b81fe3b3\") " pod="openshift-monitoring/node-exporter-fsvlc"
Sep 16 15:52:49.065127 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:49.065046    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/23df07fb-e55c-416b-a43e-adb9b81fe3b3-node-exporter-wtmp\") pod \"node-exporter-fsvlc\" (UID: \"23df07fb-e55c-416b-a43e-adb9b81fe3b3\") " pod="openshift-monitoring/node-exporter-fsvlc"
Sep 16 15:52:49.065127 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:49.065104    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/23df07fb-e55c-416b-a43e-adb9b81fe3b3-node-exporter-tls\") pod \"node-exporter-fsvlc\" (UID: \"23df07fb-e55c-416b-a43e-adb9b81fe3b3\") " pod="openshift-monitoring/node-exporter-fsvlc"
Sep 16 15:52:49.096536 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:49.096509    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:49.166021 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:49.165989    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/23df07fb-e55c-416b-a43e-adb9b81fe3b3-node-exporter-tls\") pod \"node-exporter-fsvlc\" (UID: \"23df07fb-e55c-416b-a43e-adb9b81fe3b3\") " pod="openshift-monitoring/node-exporter-fsvlc"
Sep 16 15:52:49.166156 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:49.166038    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23df07fb-e55c-416b-a43e-adb9b81fe3b3-sys\") pod \"node-exporter-fsvlc\" (UID: \"23df07fb-e55c-416b-a43e-adb9b81fe3b3\") " pod="openshift-monitoring/node-exporter-fsvlc"
Sep 16 15:52:49.166156 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:49.166062    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/23df07fb-e55c-416b-a43e-adb9b81fe3b3-node-exporter-textfile\") pod \"node-exporter-fsvlc\" (UID: \"23df07fb-e55c-416b-a43e-adb9b81fe3b3\") " pod="openshift-monitoring/node-exporter-fsvlc"
Sep 16 15:52:49.166156 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:49.166095    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/23df07fb-e55c-416b-a43e-adb9b81fe3b3-node-exporter-accelerators-collector-config\") pod \"node-exporter-fsvlc\" (UID: \"23df07fb-e55c-416b-a43e-adb9b81fe3b3\") " pod="openshift-monitoring/node-exporter-fsvlc"
Sep 16 15:52:49.166156 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:49.166119    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23df07fb-e55c-416b-a43e-adb9b81fe3b3-sys\") pod \"node-exporter-fsvlc\" (UID: \"23df07fb-e55c-416b-a43e-adb9b81fe3b3\") " pod="openshift-monitoring/node-exporter-fsvlc"
Sep 16 15:52:49.166156 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:49.166121    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/23df07fb-e55c-416b-a43e-adb9b81fe3b3-root\") pod \"node-exporter-fsvlc\" (UID: \"23df07fb-e55c-416b-a43e-adb9b81fe3b3\") " pod="openshift-monitoring/node-exporter-fsvlc"
Sep 16 15:52:49.166366 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:49.166156    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/23df07fb-e55c-416b-a43e-adb9b81fe3b3-root\") pod \"node-exporter-fsvlc\" (UID: \"23df07fb-e55c-416b-a43e-adb9b81fe3b3\") " pod="openshift-monitoring/node-exporter-fsvlc"
Sep 16 15:52:49.166366 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:49.166169    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/23df07fb-e55c-416b-a43e-adb9b81fe3b3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fsvlc\" (UID: \"23df07fb-e55c-416b-a43e-adb9b81fe3b3\") " pod="openshift-monitoring/node-exporter-fsvlc"
Sep 16 15:52:49.166366 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:49.166195    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23df07fb-e55c-416b-a43e-adb9b81fe3b3-metrics-client-ca\") pod \"node-exporter-fsvlc\" (UID: \"23df07fb-e55c-416b-a43e-adb9b81fe3b3\") " pod="openshift-monitoring/node-exporter-fsvlc"
Sep 16 15:52:49.166366 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:49.166222    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq9zh\" (UniqueName: \"kubernetes.io/projected/23df07fb-e55c-416b-a43e-adb9b81fe3b3-kube-api-access-hq9zh\") pod \"node-exporter-fsvlc\" (UID: \"23df07fb-e55c-416b-a43e-adb9b81fe3b3\") " pod="openshift-monitoring/node-exporter-fsvlc"
Sep 16 15:52:49.166366 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:49.166245    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/23df07fb-e55c-416b-a43e-adb9b81fe3b3-node-exporter-wtmp\") pod \"node-exporter-fsvlc\" (UID: \"23df07fb-e55c-416b-a43e-adb9b81fe3b3\") " pod="openshift-monitoring/node-exporter-fsvlc"
Sep 16 15:52:49.166598 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:49.166387    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/23df07fb-e55c-416b-a43e-adb9b81fe3b3-node-exporter-wtmp\") pod \"node-exporter-fsvlc\" (UID: \"23df07fb-e55c-416b-a43e-adb9b81fe3b3\") " pod="openshift-monitoring/node-exporter-fsvlc"
Sep 16 15:52:49.166598 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:49.166495    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/23df07fb-e55c-416b-a43e-adb9b81fe3b3-node-exporter-textfile\") pod \"node-exporter-fsvlc\" (UID: \"23df07fb-e55c-416b-a43e-adb9b81fe3b3\") " pod="openshift-monitoring/node-exporter-fsvlc"
Sep 16 15:52:49.166758 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:49.166739    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/23df07fb-e55c-416b-a43e-adb9b81fe3b3-node-exporter-accelerators-collector-config\") pod \"node-exporter-fsvlc\" (UID: \"23df07fb-e55c-416b-a43e-adb9b81fe3b3\") " pod="openshift-monitoring/node-exporter-fsvlc"
Sep 16 15:52:49.166880 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:49.166860    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23df07fb-e55c-416b-a43e-adb9b81fe3b3-metrics-client-ca\") pod \"node-exporter-fsvlc\" (UID: \"23df07fb-e55c-416b-a43e-adb9b81fe3b3\") " pod="openshift-monitoring/node-exporter-fsvlc"
Sep 16 15:52:49.168375 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:49.168357    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/23df07fb-e55c-416b-a43e-adb9b81fe3b3-node-exporter-tls\") pod \"node-exporter-fsvlc\" (UID: \"23df07fb-e55c-416b-a43e-adb9b81fe3b3\") " pod="openshift-monitoring/node-exporter-fsvlc"
Sep 16 15:52:49.168463 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:49.168448    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/23df07fb-e55c-416b-a43e-adb9b81fe3b3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fsvlc\" (UID: \"23df07fb-e55c-416b-a43e-adb9b81fe3b3\") " pod="openshift-monitoring/node-exporter-fsvlc"
Sep 16 15:52:49.175501 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:49.175474    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq9zh\" (UniqueName: \"kubernetes.io/projected/23df07fb-e55c-416b-a43e-adb9b81fe3b3-kube-api-access-hq9zh\") pod \"node-exporter-fsvlc\" (UID: \"23df07fb-e55c-416b-a43e-adb9b81fe3b3\") " pod="openshift-monitoring/node-exporter-fsvlc"
Sep 16 15:52:49.194921 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:49.194892    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:49.292595 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:49.292572    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fsvlc"
Sep 16 15:52:49.295041 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:49.295012    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:49.303535 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:52:49.303513    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23df07fb_e55c_416b_a43e_adb9b81fe3b3.slice/crio-045fa399a3b587a9d425e1ac8bb1accbe077d019f52eef5d884bc9d049afa4de WatchSource:0}: Error finding container 045fa399a3b587a9d425e1ac8bb1accbe077d019f52eef5d884bc9d049afa4de: Status 404 returned error can't find the container with id 045fa399a3b587a9d425e1ac8bb1accbe077d019f52eef5d884bc9d049afa4de
Sep 16 15:52:49.395745 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:49.395713    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:49.448120 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:49.448092    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:49.593382 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:49.593307    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:49.726423 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:49.726392    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fsvlc" event={"ID":"23df07fb-e55c-416b-a43e-adb9b81fe3b3","Type":"ContainerStarted","Data":"045fa399a3b587a9d425e1ac8bb1accbe077d019f52eef5d884bc9d049afa4de"}
Sep 16 15:52:49.949481 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:49.949404    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:50.018350 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:50.018319    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:50.073194 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:50.073168    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtdsz"
Sep 16 15:52:50.104165 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:50.104130    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:50.732227 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:50.732149    2554 generic.go:356] "Generic (PLEG): container finished" podID="23df07fb-e55c-416b-a43e-adb9b81fe3b3" containerID="75954a0941d236a73fa0a72b311229c26db679bd93da018c2bcdac1917c17751" exitCode=0
Sep 16 15:52:50.732227 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:50.732204    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fsvlc" event={"ID":"23df07fb-e55c-416b-a43e-adb9b81fe3b3","Type":"ContainerDied","Data":"75954a0941d236a73fa0a72b311229c26db679bd93da018c2bcdac1917c17751"}
Sep 16 15:52:50.929936 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:50.929904    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:51.026271 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:51.026200    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:51.083913 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:51.083880    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:51.320436 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:51.320403    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:51.602808 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:51.602714    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:51.672074 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:51.672041    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:51.737968 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:51.737936    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fsvlc" event={"ID":"23df07fb-e55c-416b-a43e-adb9b81fe3b3","Type":"ContainerStarted","Data":"c2ff8d5e3ecadaf8935324bc04bf2afb0a23f2ec0725daf4e7f2f4cc03a0a6d5"}
Sep 16 15:52:51.738309 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:51.737983    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fsvlc" event={"ID":"23df07fb-e55c-416b-a43e-adb9b81fe3b3","Type":"ContainerStarted","Data":"ad8b822c202fb25fe62dc142a82ff4447d6a8c5b165fcab3d9d940e2580c49c9"}
Sep 16 15:52:51.758368 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:51.758323    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fsvlc" podStartSLOduration=2.680406347 podStartE2EDuration="3.758309394s" totalImagesPullingTime="1.016659476s" totalInitContainerRuntime="61.243571ms" isStatefulPod=true podCreationTimestamp="2026-09-16 15:52:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:52:51.757959942 +0000 UTC m=+172.202652121" watchObservedRunningTime="2026-09-16 15:52:51.758309394 +0000 UTC m=+172.203001571"
Sep 16 15:52:51.816482 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:51.816456    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:51.914002 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:51.913909    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:52.037363 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:52.037329    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:52.254621 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:52.254512    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:52.602477 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:52.602448    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:52.795773 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:52.795745    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:53.015497 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:53.015405    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:53.138508 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:53.138477    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:53.193636 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:53.193609    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:53.744323 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:53.744266    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:53.811586 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:53.811554    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:53.873085 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:53.873057    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:54.097599 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:54.097568    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:55.117574 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:55.117545    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:56.721348 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:56.721317    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gb9f6"
Sep 16 15:52:58.937140 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:58.937107    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-98b6c5d9f-b5qrx"]
Sep 16 15:52:58.942843 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:58.942808    2554 patch_prober.go:28] interesting pod/image-registry-98b6c5d9f-b5qrx container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 15:52:58.942984 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:58.942885    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx" podUID="bd87ab33-4283-417b-8da9-c1e86ad0fa9e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 15:52:59.029780 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:59.029746    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:52:59.250607 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:52:59.250522    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:53:05.861967 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:05.861938    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:53:05.962715 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:05.962683    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:53:06.043740 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:06.043706    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:53:06.103771 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:06.103743    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:53:07.112136 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:07.112100    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:53:07.212760 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:07.212728    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:53:07.868923 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:07.868890    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:53:07.969400 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:07.969368    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:53:08.941808 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:08.941761    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx"
Sep 16 15:53:09.824187 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:09.824157    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:53:09.895548 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:09.895521    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:53:14.225923 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:14.225888    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:53:14.326501 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:14.326472    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:53:14.887317 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:14.887284    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:53:15.593329 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:15.593298    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:53:15.692428 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:15.692400    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:53:16.039761 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:16.039730    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:53:16.137313 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:16.137282    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:53:16.929988 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:16.929954    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:53:17.030566 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:17.030532    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:53:17.646201 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:17.646174    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:53:17.747494 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:17.747464    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:53:19.717219 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:19.717186    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:53:19.783690 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:19.783660    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:53:21.241520 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:21.241489    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:53:21.342618 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:21.342590    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:53:21.397810 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:21.397764    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:53:22.105425 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:22.105381    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:53:22.209444 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:22.209412    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:53:23.956395 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:23.956352    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx" podUID="bd87ab33-4283-417b-8da9-c1e86ad0fa9e" containerName="registry" containerID="cri-o://43903512fb9c1d24ab632f1b7f846a9d0b0923b81793a7947b63f8db738662d5" gracePeriod=30
Sep 16 15:53:24.198880 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:24.198858    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx"
Sep 16 15:53:24.223106 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:24.223031    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-image-registry-private-configuration\") pod \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\" (UID: \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\") "
Sep 16 15:53:24.223268 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:24.223076    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-installation-pull-secrets\") pod \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\" (UID: \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\") "
Sep 16 15:53:24.223343 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:24.223330    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-trusted-ca\") pod \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\" (UID: \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\") "
Sep 16 15:53:24.223410 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:24.223387    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-bound-sa-token\") pod \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\" (UID: \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\") "
Sep 16 15:53:24.223465 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:24.223434    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6bfr\" (UniqueName: \"kubernetes.io/projected/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-kube-api-access-f6bfr\") pod \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\" (UID: \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\") "
Sep 16 15:53:24.223597 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:24.223480    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-registry-certificates\") pod \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\" (UID: \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\") "
Sep 16 15:53:24.223597 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:24.223517    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-ca-trust-extracted\") pod \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\" (UID: \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\") "
Sep 16 15:53:24.223597 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:24.223562    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-registry-tls\") pod \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\" (UID: \"bd87ab33-4283-417b-8da9-c1e86ad0fa9e\") "
Sep 16 15:53:24.225093 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:24.224287    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "bd87ab33-4283-417b-8da9-c1e86ad0fa9e" (UID: "bd87ab33-4283-417b-8da9-c1e86ad0fa9e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 15:53:24.225093 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:24.224813    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "bd87ab33-4283-417b-8da9-c1e86ad0fa9e" (UID: "bd87ab33-4283-417b-8da9-c1e86ad0fa9e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 15:53:24.226352 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:24.226305    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "bd87ab33-4283-417b-8da9-c1e86ad0fa9e" (UID: "bd87ab33-4283-417b-8da9-c1e86ad0fa9e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 15:53:24.226468 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:24.226442    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-kube-api-access-f6bfr" (OuterVolumeSpecName: "kube-api-access-f6bfr") pod "bd87ab33-4283-417b-8da9-c1e86ad0fa9e" (UID: "bd87ab33-4283-417b-8da9-c1e86ad0fa9e"). InnerVolumeSpecName "kube-api-access-f6bfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 15:53:24.226548 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:24.226472    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "bd87ab33-4283-417b-8da9-c1e86ad0fa9e" (UID: "bd87ab33-4283-417b-8da9-c1e86ad0fa9e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 15:53:24.226620 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:24.226604    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "bd87ab33-4283-417b-8da9-c1e86ad0fa9e" (UID: "bd87ab33-4283-417b-8da9-c1e86ad0fa9e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 15:53:24.227242 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:24.227216    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "bd87ab33-4283-417b-8da9-c1e86ad0fa9e" (UID: "bd87ab33-4283-417b-8da9-c1e86ad0fa9e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 15:53:24.233442 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:24.233412    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "bd87ab33-4283-417b-8da9-c1e86ad0fa9e" (UID: "bd87ab33-4283-417b-8da9-c1e86ad0fa9e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:53:24.324347 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:24.324309    2554 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-registry-certificates\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 15:53:24.324347 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:24.324339    2554 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-ca-trust-extracted\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 15:53:24.324347 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:24.324355    2554 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-registry-tls\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 15:53:24.324586 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:24.324368    2554 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-image-registry-private-configuration\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 15:53:24.324586 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:24.324382    2554 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-installation-pull-secrets\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 15:53:24.324586 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:24.324393    2554 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-trusted-ca\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 15:53:24.324586 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:24.324404    2554 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-bound-sa-token\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 15:53:24.324586 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:24.324414    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6bfr\" (UniqueName: \"kubernetes.io/projected/bd87ab33-4283-417b-8da9-c1e86ad0fa9e-kube-api-access-f6bfr\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 15:53:24.848408 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:24.848375    2554 generic.go:356] "Generic (PLEG): container finished" podID="bd87ab33-4283-417b-8da9-c1e86ad0fa9e" containerID="43903512fb9c1d24ab632f1b7f846a9d0b0923b81793a7947b63f8db738662d5" exitCode=0
Sep 16 15:53:24.848581 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:24.848422    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx"
Sep 16 15:53:24.848581 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:24.848431    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx" event={"ID":"bd87ab33-4283-417b-8da9-c1e86ad0fa9e","Type":"ContainerDied","Data":"43903512fb9c1d24ab632f1b7f846a9d0b0923b81793a7947b63f8db738662d5"}
Sep 16 15:53:24.848581 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:24.848471    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-98b6c5d9f-b5qrx" event={"ID":"bd87ab33-4283-417b-8da9-c1e86ad0fa9e","Type":"ContainerDied","Data":"58f70556102d606808d0f5c21fa8be9ad9c7b82c6c690f8ece1e63d37ae7701f"}
Sep 16 15:53:24.848581 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:24.848507    2554 scope.go:117] "RemoveContainer" containerID="43903512fb9c1d24ab632f1b7f846a9d0b0923b81793a7947b63f8db738662d5"
Sep 16 15:53:24.856340 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:24.856311    2554 scope.go:117] "RemoveContainer" containerID="43903512fb9c1d24ab632f1b7f846a9d0b0923b81793a7947b63f8db738662d5"
Sep 16 15:53:24.856645 ip-10-0-136-228 kubenswrapper[2554]: E0916 15:53:24.856625    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43903512fb9c1d24ab632f1b7f846a9d0b0923b81793a7947b63f8db738662d5\": container with ID starting with 43903512fb9c1d24ab632f1b7f846a9d0b0923b81793a7947b63f8db738662d5 not found: ID does not exist" containerID="43903512fb9c1d24ab632f1b7f846a9d0b0923b81793a7947b63f8db738662d5"
Sep 16 15:53:24.856711 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:24.856654    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43903512fb9c1d24ab632f1b7f846a9d0b0923b81793a7947b63f8db738662d5"} err="failed to get container status \"43903512fb9c1d24ab632f1b7f846a9d0b0923b81793a7947b63f8db738662d5\": rpc error: code = NotFound desc = could not find container \"43903512fb9c1d24ab632f1b7f846a9d0b0923b81793a7947b63f8db738662d5\": container with ID starting with 43903512fb9c1d24ab632f1b7f846a9d0b0923b81793a7947b63f8db738662d5 not found: ID does not exist"
Sep 16 15:53:24.869316 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:24.869293    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-98b6c5d9f-b5qrx"]
Sep 16 15:53:24.873255 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:24.873236    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-98b6c5d9f-b5qrx"]
Sep 16 15:53:26.075929 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:26.075901    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd87ab33-4283-417b-8da9-c1e86ad0fa9e" path="/var/lib/kubelet/pods/bd87ab33-4283-417b-8da9-c1e86ad0fa9e/volumes"
Sep 16 15:53:33.998149 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:33.998119    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:53:35.209631 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:35.209594    2554 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg" podUID="5a899343-9e86-4def-b525-4586cb7b2afc" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 16 15:53:37.109270 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:37.109223    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:53:37.196210 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:37.196182    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:53:45.209227 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:45.209190    2554 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg" podUID="5a899343-9e86-4def-b525-4586cb7b2afc" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 16 15:53:55.209483 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:55.209445    2554 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg" podUID="5a899343-9e86-4def-b525-4586cb7b2afc" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 16 15:53:55.209946 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:55.209517    2554 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg"
Sep 16 15:53:55.210014 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:55.209997    2554 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"9d572f586dcf748d0b5dae5a4dfc4f4c876f243bcafef6e90bbbc52958572c42"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 16 15:53:55.210051 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:55.210033    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg" podUID="5a899343-9e86-4def-b525-4586cb7b2afc" containerName="service-proxy" containerID="cri-o://9d572f586dcf748d0b5dae5a4dfc4f4c876f243bcafef6e90bbbc52958572c42" gracePeriod=30
Sep 16 15:53:55.950354 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:55.950319    2554 generic.go:356] "Generic (PLEG): container finished" podID="5a899343-9e86-4def-b525-4586cb7b2afc" containerID="9d572f586dcf748d0b5dae5a4dfc4f4c876f243bcafef6e90bbbc52958572c42" exitCode=2
Sep 16 15:53:55.950520 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:55.950393    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg" event={"ID":"5a899343-9e86-4def-b525-4586cb7b2afc","Type":"ContainerDied","Data":"9d572f586dcf748d0b5dae5a4dfc4f4c876f243bcafef6e90bbbc52958572c42"}
Sep 16 15:53:55.950520 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:53:55.950436    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd9f99dc8-qdnlg" event={"ID":"5a899343-9e86-4def-b525-4586cb7b2afc","Type":"ContainerStarted","Data":"5b2389a23b4a4566eedaf09964f85edd2cbd4d5e89f2b7f2be0cebcc4cac9299"}
Sep 16 15:54:02.725541 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:54:02.725508    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:54:06.354527 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:54:06.354485    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:54:06.406370 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:54:06.406321    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:54:08.115388 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:54:08.115343    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:54:08.219806 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:54:08.219764    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:54:11.969745 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:54:11.969687    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2e625e0-e1fd-4733-97c0-2219660a6700-metrics-certs\") pod \"network-metrics-daemon-wtdsz\" (UID: \"b2e625e0-e1fd-4733-97c0-2219660a6700\") " pod="openshift-multus/network-metrics-daemon-wtdsz"
Sep 16 15:54:11.972124 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:54:11.972098    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2e625e0-e1fd-4733-97c0-2219660a6700-metrics-certs\") pod \"network-metrics-daemon-wtdsz\" (UID: \"b2e625e0-e1fd-4733-97c0-2219660a6700\") " pod="openshift-multus/network-metrics-daemon-wtdsz"
Sep 16 15:54:11.976925 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:54:11.976907    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mmzn4\""
Sep 16 15:54:11.984426 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:54:11.984408    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtdsz"
Sep 16 15:54:12.255540 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:54:12.255459    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wtdsz"]
Sep 16 15:54:12.258277 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:54:12.258249    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e625e0_e1fd_4733_97c0_2219660a6700.slice/crio-0fe0b842202bbf2298b11578aa919e973a298c4a0147b79a72ad1cfb5f445a95 WatchSource:0}: Error finding container 0fe0b842202bbf2298b11578aa919e973a298c4a0147b79a72ad1cfb5f445a95: Status 404 returned error can't find the container with id 0fe0b842202bbf2298b11578aa919e973a298c4a0147b79a72ad1cfb5f445a95
Sep 16 15:54:13.006499 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:54:13.006465    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wtdsz" event={"ID":"b2e625e0-e1fd-4733-97c0-2219660a6700","Type":"ContainerStarted","Data":"0fe0b842202bbf2298b11578aa919e973a298c4a0147b79a72ad1cfb5f445a95"}
Sep 16 15:54:14.011858 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:54:14.011760    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wtdsz" event={"ID":"b2e625e0-e1fd-4733-97c0-2219660a6700","Type":"ContainerStarted","Data":"b36632bfb34a0bef9a368c6be51b0eb69474c022840222c3fa0732962618ffe4"}
Sep 16 15:54:14.011858 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:54:14.011835    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wtdsz" event={"ID":"b2e625e0-e1fd-4733-97c0-2219660a6700","Type":"ContainerStarted","Data":"27358090393d27616515493d3515236ac3b89a62bac867c101ab271b79cb4f93"}
Sep 16 15:54:14.150038 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:54:14.150000    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:54:14.211704 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:54:14.211673    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:54:16.217886 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:54:16.217848    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:54:16.275260 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:54:16.275234    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:54:17.432842 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:54:17.432811    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:54:17.487955 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:54:17.487924    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:54:17.551000 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:54:17.550975    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:54:18.175273 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:54:18.175241    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:54:18.233710 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:54:18.233687    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:54:27.496160 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:54:27.496082    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:54:27.597490 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:54:27.597463    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:54:27.660725 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:54:27.660698    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:54:51.319424 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:54:51.319391    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:54:52.053501 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:54:52.053469    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:54:52.111601 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:54:52.111570    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:54:59.978851 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:54:59.978823    2554 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 15:55:10.913697 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:55:10.913661    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:55:23.376698 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:55:23.376669    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:55:23.432296 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:55:23.432265    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:55:35.295496 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:55:35.295460    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:55:35.349670 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:55:35.349626    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:55:51.170487 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:55:51.170451    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:55:51.226772 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:55:51.226744    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:55:52.110992 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:55:52.110960    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:55:52.211967 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:55:52.211894    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:56:01.857975 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:56:01.857940    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:56:01.958026 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:56:01.957997    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:56:02.058439 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:56:02.058409    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:57:00.450034 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:57:00.449996    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:57:00.550382 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:57:00.550351    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:58:02.197915 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:58:02.197874    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:58:13.378910 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:58:13.378843    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:58:13.478272 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:58:13.478246    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:58:13.577829 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:58:13.577789    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:58:31.749252 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:58:31.749224    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:58:31.847488 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:58:31.847464    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:58:31.907447 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:58:31.907417    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:58:34.899365 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:58:34.899334    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:58:34.957768 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:58:34.957739    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:58:50.208355 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:58:50.208323    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:58:50.267400 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:58:50.267376    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:58:54.026072 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:58:54.025997    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:58:54.121769 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:58:54.121738    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:59:00.165232 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:59:00.165198    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:59:00.450090 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:59:00.450013    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:59:00.618673 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:59:00.618644    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:59:00.750990 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:59:00.750916    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:59:00.815107 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:59:00.815083    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:59:00.922589 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:59:00.922555    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:59:00.940686 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:59:00.940640    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wtdsz" podStartSLOduration=538.490448902 podStartE2EDuration="8m59.940623892s" totalImagesPullingTime="1.45017499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:54:14.028909663 +0000 UTC m=+254.473601861" watchObservedRunningTime="2026-09-16 15:59:00.940623892 +0000 UTC m=+541.385316071"
Sep 16 15:59:00.941501 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:59:00.941483    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-5tcxb"]
Sep 16 15:59:00.941707 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:59:00.941696    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd87ab33-4283-417b-8da9-c1e86ad0fa9e" containerName="registry"
Sep 16 15:59:00.941749 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:59:00.941710    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd87ab33-4283-417b-8da9-c1e86ad0fa9e" containerName="registry"
Sep 16 15:59:00.941749 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:59:00.941743    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd87ab33-4283-417b-8da9-c1e86ad0fa9e" containerName="registry"
Sep 16 15:59:00.944541 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:59:00.944526    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-5tcxb"
Sep 16 15:59:00.947751 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:59:00.947727    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 16 15:59:00.947863 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:59:00.947727    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 16 15:59:00.949101 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:59:00.949079    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 16 15:59:00.949201 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:59:00.949124    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 16 15:59:00.949201 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:59:00.949125    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-b4tx9\""
Sep 16 15:59:00.954125 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:59:00.954103    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-5tcxb"]
Sep 16 15:59:01.022659 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:59:01.022597    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:59:01.066787 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:59:01.066762    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7cwm\" (UniqueName: \"kubernetes.io/projected/9bf8aa69-2b2c-49ab-a2cb-4dd7afd1cc14-kube-api-access-k7cwm\") pod \"keda-admission-6c4d864b86-5tcxb\" (UID: \"9bf8aa69-2b2c-49ab-a2cb-4dd7afd1cc14\") " pod="openshift-keda/keda-admission-6c4d864b86-5tcxb"
Sep 16 15:59:01.066896 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:59:01.066824    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9bf8aa69-2b2c-49ab-a2cb-4dd7afd1cc14-certificates\") pod \"keda-admission-6c4d864b86-5tcxb\" (UID: \"9bf8aa69-2b2c-49ab-a2cb-4dd7afd1cc14\") " pod="openshift-keda/keda-admission-6c4d864b86-5tcxb"
Sep 16 15:59:01.123161 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:59:01.123133    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:59:01.168034 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:59:01.168007    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7cwm\" (UniqueName: \"kubernetes.io/projected/9bf8aa69-2b2c-49ab-a2cb-4dd7afd1cc14-kube-api-access-k7cwm\") pod \"keda-admission-6c4d864b86-5tcxb\" (UID: \"9bf8aa69-2b2c-49ab-a2cb-4dd7afd1cc14\") " pod="openshift-keda/keda-admission-6c4d864b86-5tcxb"
Sep 16 15:59:01.168417 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:59:01.168045    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9bf8aa69-2b2c-49ab-a2cb-4dd7afd1cc14-certificates\") pod \"keda-admission-6c4d864b86-5tcxb\" (UID: \"9bf8aa69-2b2c-49ab-a2cb-4dd7afd1cc14\") " pod="openshift-keda/keda-admission-6c4d864b86-5tcxb"
Sep 16 15:59:01.170472 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:59:01.170453    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9bf8aa69-2b2c-49ab-a2cb-4dd7afd1cc14-certificates\") pod \"keda-admission-6c4d864b86-5tcxb\" (UID: \"9bf8aa69-2b2c-49ab-a2cb-4dd7afd1cc14\") " pod="openshift-keda/keda-admission-6c4d864b86-5tcxb"
Sep 16 15:59:01.177830 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:59:01.177809    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7cwm\" (UniqueName: \"kubernetes.io/projected/9bf8aa69-2b2c-49ab-a2cb-4dd7afd1cc14-kube-api-access-k7cwm\") pod \"keda-admission-6c4d864b86-5tcxb\" (UID: \"9bf8aa69-2b2c-49ab-a2cb-4dd7afd1cc14\") " pod="openshift-keda/keda-admission-6c4d864b86-5tcxb"
Sep 16 15:59:01.223388 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:59:01.223359    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:59:01.253994 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:59:01.253969    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-5tcxb"
Sep 16 15:59:01.327552 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:59:01.327526    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:59:01.483993 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:59:01.483959    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:59:01.521260 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:59:01.521103    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-5tcxb"]
Sep 16 15:59:01.523778 ip-10-0-136-228 kubenswrapper[2554]: W0916 15:59:01.523746    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bf8aa69_2b2c_49ab_a2cb_4dd7afd1cc14.slice/crio-ae2e3a671ac3a78f633f53e8227ddce793f306577695dd6c8e0e2ca5777467d9 WatchSource:0}: Error finding container ae2e3a671ac3a78f633f53e8227ddce793f306577695dd6c8e0e2ca5777467d9: Status 404 returned error can't find the container with id ae2e3a671ac3a78f633f53e8227ddce793f306577695dd6c8e0e2ca5777467d9
Sep 16 15:59:01.525143 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:59:01.525126    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 15:59:01.583570 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:59:01.583485    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:59:01.668340 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:59:01.668308    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:59:01.927017 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:59:01.926938    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-5tcxb" event={"ID":"9bf8aa69-2b2c-49ab-a2cb-4dd7afd1cc14","Type":"ContainerStarted","Data":"ae2e3a671ac3a78f633f53e8227ddce793f306577695dd6c8e0e2ca5777467d9"}
Sep 16 15:59:04.938926 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:59:04.938885    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-5tcxb" event={"ID":"9bf8aa69-2b2c-49ab-a2cb-4dd7afd1cc14","Type":"ContainerStarted","Data":"c687876c3ec48cf20d51288d9b9391abd6a1b1cd24e18a07f048ef100bf70a21"}
Sep 16 15:59:04.939354 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:59:04.939031    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-5tcxb"
Sep 16 15:59:04.954626 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:59:04.954585    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-5tcxb" podStartSLOduration=2.28164983 podStartE2EDuration="4.954567364s" totalImagesPullingTime="2.672917534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:59:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:59:04.954541348 +0000 UTC m=+545.399233537" watchObservedRunningTime="2026-09-16 15:59:04.954567364 +0000 UTC m=+545.399259521"
Sep 16 15:59:25.944515 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:59:25.944486    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-5tcxb"
Sep 16 15:59:52.904624 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:59:52.904584    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 15:59:53.005403 ip-10-0-136-228 kubenswrapper[2554]: I0916 15:59:53.005373    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:00:04.405093 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:04.405046    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:00:05.380600 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:05.380543    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:00:08.286964 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:08.286902    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:00:08.387047 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:08.387003    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:00:09.115453 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:09.115422    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:00:09.229223 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:09.229178    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:00:11.510751 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:11.510703    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:00:11.766334 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:11.766250    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:00:11.798708 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:11.798681    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-mg5zp"]
Sep 16 16:00:11.801467 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:11.801449    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mg5zp"
Sep 16 16:00:11.804254 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:11.804234    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 16 16:00:11.805439 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:11.805420    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-k594n\""
Sep 16 16:00:11.805529 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:11.805449    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 16 16:00:11.812061 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:11.812041    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-mg5zp"]
Sep 16 16:00:11.824664 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:11.824612    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwtnh\" (UniqueName: \"kubernetes.io/projected/873d3696-3e54-4c42-8481-4047542aa54c-kube-api-access-fwtnh\") pod \"cert-manager-cainjector-559cdd8c87-mg5zp\" (UID: \"873d3696-3e54-4c42-8481-4047542aa54c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mg5zp"
Sep 16 16:00:11.824894 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:11.824668    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/873d3696-3e54-4c42-8481-4047542aa54c-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-mg5zp\" (UID: \"873d3696-3e54-4c42-8481-4047542aa54c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mg5zp"
Sep 16 16:00:11.925621 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:11.925589    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwtnh\" (UniqueName: \"kubernetes.io/projected/873d3696-3e54-4c42-8481-4047542aa54c-kube-api-access-fwtnh\") pod \"cert-manager-cainjector-559cdd8c87-mg5zp\" (UID: \"873d3696-3e54-4c42-8481-4047542aa54c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mg5zp"
Sep 16 16:00:11.925755 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:11.925626    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/873d3696-3e54-4c42-8481-4047542aa54c-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-mg5zp\" (UID: \"873d3696-3e54-4c42-8481-4047542aa54c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mg5zp"
Sep 16 16:00:11.933653 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:11.933630    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/873d3696-3e54-4c42-8481-4047542aa54c-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-mg5zp\" (UID: \"873d3696-3e54-4c42-8481-4047542aa54c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mg5zp"
Sep 16 16:00:11.933873 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:11.933856    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwtnh\" (UniqueName: \"kubernetes.io/projected/873d3696-3e54-4c42-8481-4047542aa54c-kube-api-access-fwtnh\") pod \"cert-manager-cainjector-559cdd8c87-mg5zp\" (UID: \"873d3696-3e54-4c42-8481-4047542aa54c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mg5zp"
Sep 16 16:00:12.109685 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:12.109654    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mg5zp"
Sep 16 16:00:12.378062 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:12.377960    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-mg5zp"]
Sep 16 16:00:12.381425 ip-10-0-136-228 kubenswrapper[2554]: W0916 16:00:12.381396    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod873d3696_3e54_4c42_8481_4047542aa54c.slice/crio-3fd9f08dabb7e5830202b85a3539aeae07334dffded6586a2104a0d257a4f75e WatchSource:0}: Error finding container 3fd9f08dabb7e5830202b85a3539aeae07334dffded6586a2104a0d257a4f75e: Status 404 returned error can't find the container with id 3fd9f08dabb7e5830202b85a3539aeae07334dffded6586a2104a0d257a4f75e
Sep 16 16:00:12.474283 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:12.474254    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:00:12.531502 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:12.531470    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:00:13.154405 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:13.154374    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mg5zp" event={"ID":"873d3696-3e54-4c42-8481-4047542aa54c","Type":"ContainerStarted","Data":"3fd9f08dabb7e5830202b85a3539aeae07334dffded6586a2104a0d257a4f75e"}
Sep 16 16:00:14.612120 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:14.612026    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:00:14.710576 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:14.710543    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:00:14.780011 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:14.779974    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:00:18.174836 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:18.174784    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mg5zp" event={"ID":"873d3696-3e54-4c42-8481-4047542aa54c","Type":"ContainerStarted","Data":"c77163da566ecf3993c4665e31b4b0cce4b5a47255ef72de19bee3a65140613b"}
Sep 16 16:00:18.193424 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:18.193376    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mg5zp" podStartSLOduration=1.928897894 podStartE2EDuration="7.193360882s" totalImagesPullingTime="5.264462988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:00:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:00:18.191997447 +0000 UTC m=+618.636689626" watchObservedRunningTime="2026-09-16 16:00:18.193360882 +0000 UTC m=+618.638053060"
Sep 16 16:00:18.417122 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:18.417079    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:00:18.476377 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:18.476304    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:00:19.096814 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:19.096777    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:00:27.087770 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:27.087692    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:00:27.187255 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:27.187222    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:00:38.142141 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:38.142099    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:00:42.853903 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:42.853871    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:00:42.950363 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:42.950329    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:00:43.932518 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:43.932484    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:00:46.365512 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:46.365476    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:00:46.446325 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:46.446291    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-75c8bf98c8-k8hnj"]
Sep 16 16:00:46.448161 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:46.448143    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-75c8bf98c8-k8hnj"
Sep 16 16:00:46.449750 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:46.449249    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:00:46.453149 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:46.451685    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 16 16:00:46.453149 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:46.452002    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 16 16:00:46.453395 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:46.453377    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-rxqwh\""
Sep 16 16:00:46.453478 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:46.453399    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 16 16:00:46.453478 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:46.453411    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 16 16:00:46.453929 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:46.453907    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 16 16:00:46.461366 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:46.461232    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-75c8bf98c8-k8hnj"]
Sep 16 16:00:46.548088 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:46.548052    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1cca4de1-23f3-4a52-8439-6d5ab51428de-cert\") pod \"lws-controller-manager-75c8bf98c8-k8hnj\" (UID: \"1cca4de1-23f3-4a52-8439-6d5ab51428de\") " pod="openshift-lws-operator/lws-controller-manager-75c8bf98c8-k8hnj"
Sep 16 16:00:46.548257 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:46.548094    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1cca4de1-23f3-4a52-8439-6d5ab51428de-metrics-cert\") pod \"lws-controller-manager-75c8bf98c8-k8hnj\" (UID: \"1cca4de1-23f3-4a52-8439-6d5ab51428de\") " pod="openshift-lws-operator/lws-controller-manager-75c8bf98c8-k8hnj"
Sep 16 16:00:46.548257 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:46.548123    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1cca4de1-23f3-4a52-8439-6d5ab51428de-manager-config\") pod \"lws-controller-manager-75c8bf98c8-k8hnj\" (UID: \"1cca4de1-23f3-4a52-8439-6d5ab51428de\") " pod="openshift-lws-operator/lws-controller-manager-75c8bf98c8-k8hnj"
Sep 16 16:00:46.548257 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:46.548212    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk256\" (UniqueName: \"kubernetes.io/projected/1cca4de1-23f3-4a52-8439-6d5ab51428de-kube-api-access-rk256\") pod \"lws-controller-manager-75c8bf98c8-k8hnj\" (UID: \"1cca4de1-23f3-4a52-8439-6d5ab51428de\") " pod="openshift-lws-operator/lws-controller-manager-75c8bf98c8-k8hnj"
Sep 16 16:00:46.649448 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:46.649369    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk256\" (UniqueName: \"kubernetes.io/projected/1cca4de1-23f3-4a52-8439-6d5ab51428de-kube-api-access-rk256\") pod \"lws-controller-manager-75c8bf98c8-k8hnj\" (UID: \"1cca4de1-23f3-4a52-8439-6d5ab51428de\") " pod="openshift-lws-operator/lws-controller-manager-75c8bf98c8-k8hnj"
Sep 16 16:00:46.649448 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:46.649429    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1cca4de1-23f3-4a52-8439-6d5ab51428de-cert\") pod \"lws-controller-manager-75c8bf98c8-k8hnj\" (UID: \"1cca4de1-23f3-4a52-8439-6d5ab51428de\") " pod="openshift-lws-operator/lws-controller-manager-75c8bf98c8-k8hnj"
Sep 16 16:00:46.649448 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:46.649448    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1cca4de1-23f3-4a52-8439-6d5ab51428de-metrics-cert\") pod \"lws-controller-manager-75c8bf98c8-k8hnj\" (UID: \"1cca4de1-23f3-4a52-8439-6d5ab51428de\") " pod="openshift-lws-operator/lws-controller-manager-75c8bf98c8-k8hnj"
Sep 16 16:00:46.649651 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:46.649468    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1cca4de1-23f3-4a52-8439-6d5ab51428de-manager-config\") pod \"lws-controller-manager-75c8bf98c8-k8hnj\" (UID: \"1cca4de1-23f3-4a52-8439-6d5ab51428de\") " pod="openshift-lws-operator/lws-controller-manager-75c8bf98c8-k8hnj"
Sep 16 16:00:46.650062 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:46.650030    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1cca4de1-23f3-4a52-8439-6d5ab51428de-manager-config\") pod \"lws-controller-manager-75c8bf98c8-k8hnj\" (UID: \"1cca4de1-23f3-4a52-8439-6d5ab51428de\") " pod="openshift-lws-operator/lws-controller-manager-75c8bf98c8-k8hnj"
Sep 16 16:00:46.651979 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:46.651958    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1cca4de1-23f3-4a52-8439-6d5ab51428de-cert\") pod \"lws-controller-manager-75c8bf98c8-k8hnj\" (UID: \"1cca4de1-23f3-4a52-8439-6d5ab51428de\") " pod="openshift-lws-operator/lws-controller-manager-75c8bf98c8-k8hnj"
Sep 16 16:00:46.652063 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:46.651977    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1cca4de1-23f3-4a52-8439-6d5ab51428de-metrics-cert\") pod \"lws-controller-manager-75c8bf98c8-k8hnj\" (UID: \"1cca4de1-23f3-4a52-8439-6d5ab51428de\") " pod="openshift-lws-operator/lws-controller-manager-75c8bf98c8-k8hnj"
Sep 16 16:00:46.657960 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:46.657940    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk256\" (UniqueName: \"kubernetes.io/projected/1cca4de1-23f3-4a52-8439-6d5ab51428de-kube-api-access-rk256\") pod \"lws-controller-manager-75c8bf98c8-k8hnj\" (UID: \"1cca4de1-23f3-4a52-8439-6d5ab51428de\") " pod="openshift-lws-operator/lws-controller-manager-75c8bf98c8-k8hnj"
Sep 16 16:00:46.761916 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:46.761884    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-75c8bf98c8-k8hnj"
Sep 16 16:00:47.037451 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:47.037420    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-75c8bf98c8-k8hnj"]
Sep 16 16:00:47.040314 ip-10-0-136-228 kubenswrapper[2554]: W0916 16:00:47.040289    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cca4de1_23f3_4a52_8439_6d5ab51428de.slice/crio-468224e32c758212d97e4a5998357b0264e15de78edb5407a9fb723ad2c14d03 WatchSource:0}: Error finding container 468224e32c758212d97e4a5998357b0264e15de78edb5407a9fb723ad2c14d03: Status 404 returned error can't find the container with id 468224e32c758212d97e4a5998357b0264e15de78edb5407a9fb723ad2c14d03
Sep 16 16:00:47.266588 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:47.266547    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-75c8bf98c8-k8hnj" event={"ID":"1cca4de1-23f3-4a52-8439-6d5ab51428de","Type":"ContainerStarted","Data":"468224e32c758212d97e4a5998357b0264e15de78edb5407a9fb723ad2c14d03"}
Sep 16 16:00:49.316251 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:49.316213    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:00:49.415701 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:49.415666    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:00:50.465635 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:50.465603    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:00:50.563146 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:50.563116    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:00:53.668275 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:53.668238    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:00:55.295344 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:55.295307    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-75c8bf98c8-k8hnj" event={"ID":"1cca4de1-23f3-4a52-8439-6d5ab51428de","Type":"ContainerStarted","Data":"db4ff1f011f63fd1fb8a4e9cfdd0254e1027ebb62f2a1a1597f9afd4cf26453c"}
Sep 16 16:00:55.295727 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:55.295441    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-75c8bf98c8-k8hnj"
Sep 16 16:00:55.311539 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:55.311488    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-75c8bf98c8-k8hnj" podStartSLOduration=1.856920989 podStartE2EDuration="9.311468625s" totalImagesPullingTime="7.454547636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:00:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:00:55.311434563 +0000 UTC m=+655.756126760" watchObservedRunningTime="2026-09-16 16:00:55.311468625 +0000 UTC m=+655.756160804"
Sep 16 16:00:57.947638 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:57.947608    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:00:58.051146 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:00:58.051117    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:01:06.303296 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:06.303255    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-75c8bf98c8-k8hnj"
Sep 16 16:01:25.541282 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:25.541238    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:01:26.341505 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:26.341470    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:01:29.756764 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:29.756736    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:01:30.180540 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:30.180511    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:01:30.577708 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:30.577675    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:01:31.226273 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:31.226239    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:01:31.315873 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:31.315845    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:01:32.794911 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:32.794878    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:01:34.505867 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:34.505833    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:01:34.611452 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:34.611420    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:01:35.064918 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:35.064885    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:01:35.166205 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:35.166171    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:01:36.804825 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:36.804775    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-2qjkj"]
Sep 16 16:01:36.811469 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:36.811448    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-2qjkj"
Sep 16 16:01:36.814633 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:36.814609    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-m7xdj\""
Sep 16 16:01:36.814633 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:36.814623    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 16:01:36.814816 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:36.814670    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 16:01:36.814865 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:36.814847    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 16 16:01:36.819253 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:36.819230    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-2qjkj"]
Sep 16 16:01:36.912609 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:36.912564    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlb8m\" (UniqueName: \"kubernetes.io/projected/93232e18-073e-43c6-b4ce-0b88d0a675d5-kube-api-access-hlb8m\") pod \"dns-operator-controller-manager-65dc8f76f5-2qjkj\" (UID: \"93232e18-073e-43c6-b4ce-0b88d0a675d5\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-2qjkj"
Sep 16 16:01:37.013434 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:37.013404    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlb8m\" (UniqueName: \"kubernetes.io/projected/93232e18-073e-43c6-b4ce-0b88d0a675d5-kube-api-access-hlb8m\") pod \"dns-operator-controller-manager-65dc8f76f5-2qjkj\" (UID: \"93232e18-073e-43c6-b4ce-0b88d0a675d5\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-2qjkj"
Sep 16 16:01:37.025102 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:37.025078    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlb8m\" (UniqueName: \"kubernetes.io/projected/93232e18-073e-43c6-b4ce-0b88d0a675d5-kube-api-access-hlb8m\") pod \"dns-operator-controller-manager-65dc8f76f5-2qjkj\" (UID: \"93232e18-073e-43c6-b4ce-0b88d0a675d5\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-2qjkj"
Sep 16 16:01:37.121425 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:37.121339    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-2qjkj"
Sep 16 16:01:37.391852 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:37.391761    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-2qjkj"]
Sep 16 16:01:37.394991 ip-10-0-136-228 kubenswrapper[2554]: W0916 16:01:37.394969    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93232e18_073e_43c6_b4ce_0b88d0a675d5.slice/crio-14bb9cb397149ba809f4482a168bfd9e075ea49a2f49fb0955d0bbc3395d9ef0 WatchSource:0}: Error finding container 14bb9cb397149ba809f4482a168bfd9e075ea49a2f49fb0955d0bbc3395d9ef0: Status 404 returned error can't find the container with id 14bb9cb397149ba809f4482a168bfd9e075ea49a2f49fb0955d0bbc3395d9ef0
Sep 16 16:01:37.437528 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:37.437496    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-2qjkj" event={"ID":"93232e18-073e-43c6-b4ce-0b88d0a675d5","Type":"ContainerStarted","Data":"14bb9cb397149ba809f4482a168bfd9e075ea49a2f49fb0955d0bbc3395d9ef0"}
Sep 16 16:01:37.622612 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:37.622578    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:01:38.779314 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:38.779284    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:01:38.783709 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:38.783684    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-2cglz"]
Sep 16 16:01:38.786845 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:38.786830    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-2cglz"
Sep 16 16:01:38.790041 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:38.790014    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 16 16:01:38.790041 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:38.790027    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-frpm4\""
Sep 16 16:01:38.790187 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:38.790063    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 16 16:01:38.796971 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:38.796948    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-2cglz"]
Sep 16 16:01:38.924320 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:38.924290    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:01:38.927380 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:38.927358    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcwcd\" (UniqueName: \"kubernetes.io/projected/5b483164-2cb4-419d-a6ff-e726301444ed-kube-api-access-wcwcd\") pod \"kuadrant-console-plugin-fbdd4f445-2cglz\" (UID: \"5b483164-2cb4-419d-a6ff-e726301444ed\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-2cglz"
Sep 16 16:01:38.927455 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:38.927401    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b483164-2cb4-419d-a6ff-e726301444ed-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-2cglz\" (UID: \"5b483164-2cb4-419d-a6ff-e726301444ed\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-2cglz"
Sep 16 16:01:38.927455 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:38.927421    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5b483164-2cb4-419d-a6ff-e726301444ed-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-2cglz\" (UID: \"5b483164-2cb4-419d-a6ff-e726301444ed\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-2cglz"
Sep 16 16:01:39.028618 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:39.028577    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcwcd\" (UniqueName: \"kubernetes.io/projected/5b483164-2cb4-419d-a6ff-e726301444ed-kube-api-access-wcwcd\") pod \"kuadrant-console-plugin-fbdd4f445-2cglz\" (UID: \"5b483164-2cb4-419d-a6ff-e726301444ed\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-2cglz"
Sep 16 16:01:39.028820 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:39.028630    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b483164-2cb4-419d-a6ff-e726301444ed-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-2cglz\" (UID: \"5b483164-2cb4-419d-a6ff-e726301444ed\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-2cglz"
Sep 16 16:01:39.028820 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:39.028651    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5b483164-2cb4-419d-a6ff-e726301444ed-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-2cglz\" (UID: \"5b483164-2cb4-419d-a6ff-e726301444ed\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-2cglz"
Sep 16 16:01:39.029445 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:39.029383    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5b483164-2cb4-419d-a6ff-e726301444ed-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-2cglz\" (UID: \"5b483164-2cb4-419d-a6ff-e726301444ed\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-2cglz"
Sep 16 16:01:39.031156 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:39.031136    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b483164-2cb4-419d-a6ff-e726301444ed-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-2cglz\" (UID: \"5b483164-2cb4-419d-a6ff-e726301444ed\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-2cglz"
Sep 16 16:01:39.037884 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:39.037860    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcwcd\" (UniqueName: \"kubernetes.io/projected/5b483164-2cb4-419d-a6ff-e726301444ed-kube-api-access-wcwcd\") pod \"kuadrant-console-plugin-fbdd4f445-2cglz\" (UID: \"5b483164-2cb4-419d-a6ff-e726301444ed\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-2cglz"
Sep 16 16:01:39.096147 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:39.096107    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-2cglz"
Sep 16 16:01:39.368851 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:39.368820    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-2cglz"]
Sep 16 16:01:39.371444 ip-10-0-136-228 kubenswrapper[2554]: W0916 16:01:39.371416    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b483164_2cb4_419d_a6ff_e726301444ed.slice/crio-3af64515da738e9af4f9169b535f0b388eb32232bc29bcf63b3c20ee624b491c WatchSource:0}: Error finding container 3af64515da738e9af4f9169b535f0b388eb32232bc29bcf63b3c20ee624b491c: Status 404 returned error can't find the container with id 3af64515da738e9af4f9169b535f0b388eb32232bc29bcf63b3c20ee624b491c
Sep 16 16:01:39.451481 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:39.451445    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-2cglz" event={"ID":"5b483164-2cb4-419d-a6ff-e726301444ed","Type":"ContainerStarted","Data":"3af64515da738e9af4f9169b535f0b388eb32232bc29bcf63b3c20ee624b491c"}
Sep 16 16:01:40.456216 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:40.456187    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-2qjkj" event={"ID":"93232e18-073e-43c6-b4ce-0b88d0a675d5","Type":"ContainerStarted","Data":"f541ec76ecad27e680c70928e379d978ae52729ba8533d77bfa89f078122aa78"}
Sep 16 16:01:40.456559 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:40.456293    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-2qjkj"
Sep 16 16:01:40.477647 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:40.477599    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-2qjkj" podStartSLOduration=1.63580619 podStartE2EDuration="4.477584148s" totalImagesPullingTime="2.841777958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:01:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:01:40.475953296 +0000 UTC m=+700.920645473" watchObservedRunningTime="2026-09-16 16:01:40.477584148 +0000 UTC m=+700.922276325"
Sep 16 16:01:40.557809 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:40.557766    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:01:40.655087 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:40.655062    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:01:41.351746 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:41.351672    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:01:41.412715 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:41.412684    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:01:41.759681 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:41.759600    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:01:41.823826 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:41.823782    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:01:45.477384 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:45.477348    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-2cglz" event={"ID":"5b483164-2cb4-419d-a6ff-e726301444ed","Type":"ContainerStarted","Data":"bec49494cb2c61f37333043082c5180e6e197b20a88b8a456c60e089e9065b09"}
Sep 16 16:01:45.495188 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:45.495141    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-2cglz" podStartSLOduration=2.210671818 podStartE2EDuration="7.495125991s" totalImagesPullingTime="5.284454173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:01:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:01:45.4935278 +0000 UTC m=+705.938219994" watchObservedRunningTime="2026-09-16 16:01:45.495125991 +0000 UTC m=+705.939818166"
Sep 16 16:01:49.960357 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:49.960324    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:01:51.462255 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:51.462223    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-2qjkj"
Sep 16 16:01:52.313544 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:52.313468    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:01:52.418306 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:52.418275    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:01:54.445217 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:54.445187    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:01:54.500619 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:54.500591    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:01:55.403024 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:55.402984    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:01:55.458788 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:01:55.458758    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:02:03.162161 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:03.162125    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:02:03.260543 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:03.260509    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:02:06.915978 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:06.915946    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:02:10.768340 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:10.768309    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:02:10.868452 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:10.868420    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:02:13.173430 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:13.173402    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:02:13.766159 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:13.766124    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:02:13.867576 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:13.867541    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:02:16.215670 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:16.215634    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:02:16.268636 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:16.268607    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:02:20.757936 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:20.757906    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:02:20.856998 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:20.856970    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:02:20.911325 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:20.911295    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:02:21.137965 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:21.137928    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-q7kx5"]
Sep 16 16:02:21.157345 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:21.157318    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-q7kx5"]
Sep 16 16:02:21.157512 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:21.157405    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-q7kx5"
Sep 16 16:02:21.160429 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:21.160408    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 16 16:02:21.171829 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:21.171787    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-q7kx5"]
Sep 16 16:02:21.222702 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:21.222672    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fc5c5232-5455-488f-9b79-ed0048e637c0-config-file\") pod \"limitador-limitador-84994f7ffc-q7kx5\" (UID: \"fc5c5232-5455-488f-9b79-ed0048e637c0\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-q7kx5"
Sep 16 16:02:21.222853 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:21.222711    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bddfh\" (UniqueName: \"kubernetes.io/projected/fc5c5232-5455-488f-9b79-ed0048e637c0-kube-api-access-bddfh\") pod \"limitador-limitador-84994f7ffc-q7kx5\" (UID: \"fc5c5232-5455-488f-9b79-ed0048e637c0\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-q7kx5"
Sep 16 16:02:21.323113 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:21.323088    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fc5c5232-5455-488f-9b79-ed0048e637c0-config-file\") pod \"limitador-limitador-84994f7ffc-q7kx5\" (UID: \"fc5c5232-5455-488f-9b79-ed0048e637c0\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-q7kx5"
Sep 16 16:02:21.323268 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:21.323125    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bddfh\" (UniqueName: \"kubernetes.io/projected/fc5c5232-5455-488f-9b79-ed0048e637c0-kube-api-access-bddfh\") pod \"limitador-limitador-84994f7ffc-q7kx5\" (UID: \"fc5c5232-5455-488f-9b79-ed0048e637c0\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-q7kx5"
Sep 16 16:02:21.323671 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:21.323651    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fc5c5232-5455-488f-9b79-ed0048e637c0-config-file\") pod \"limitador-limitador-84994f7ffc-q7kx5\" (UID: \"fc5c5232-5455-488f-9b79-ed0048e637c0\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-q7kx5"
Sep 16 16:02:21.332374 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:21.332354    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bddfh\" (UniqueName: \"kubernetes.io/projected/fc5c5232-5455-488f-9b79-ed0048e637c0-kube-api-access-bddfh\") pod \"limitador-limitador-84994f7ffc-q7kx5\" (UID: \"fc5c5232-5455-488f-9b79-ed0048e637c0\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-q7kx5"
Sep 16 16:02:21.466712 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:21.466623    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-q7kx5"
Sep 16 16:02:21.591681 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:21.591650    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:02:21.693276 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:21.693244    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:02:21.732748 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:21.732669    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-q7kx5"]
Sep 16 16:02:21.735738 ip-10-0-136-228 kubenswrapper[2554]: W0916 16:02:21.735713    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc5c5232_5455_488f_9b79_ed0048e637c0.slice/crio-2d7b7a62faf72b2b178c744ee9dcc75b562e78c2dc26d76331e056ead9015a60 WatchSource:0}: Error finding container 2d7b7a62faf72b2b178c744ee9dcc75b562e78c2dc26d76331e056ead9015a60: Status 404 returned error can't find the container with id 2d7b7a62faf72b2b178c744ee9dcc75b562e78c2dc26d76331e056ead9015a60
Sep 16 16:02:22.608511 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:22.608486    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-q7kx5" event={"ID":"fc5c5232-5455-488f-9b79-ed0048e637c0","Type":"ContainerStarted","Data":"2d7b7a62faf72b2b178c744ee9dcc75b562e78c2dc26d76331e056ead9015a60"}
Sep 16 16:02:22.977483 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:22.977400    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:02:23.080274 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:23.080244    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:02:23.614455 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:23.614422    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-q7kx5" event={"ID":"fc5c5232-5455-488f-9b79-ed0048e637c0","Type":"ContainerStarted","Data":"e5d733bb73f35ddbe8f9a397937a2ce652afad115c8baf677e4747e29353a643"}
Sep 16 16:02:23.614889 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:23.614614    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-q7kx5"
Sep 16 16:02:23.632167 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:23.632117    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-q7kx5" podStartSLOduration=1.806792174 podStartE2EDuration="2.632102459s" totalImagesPullingTime="825.310285ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:02:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:02:23.631728463 +0000 UTC m=+744.076420662" watchObservedRunningTime="2026-09-16 16:02:23.632102459 +0000 UTC m=+744.076794684"
Sep 16 16:02:24.841298 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:24.841265    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:02:24.896520 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:24.896495    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:02:32.581687 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:32.581653    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:02:32.681549 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:32.681515    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:02:34.620007 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:34.619973    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-q7kx5"
Sep 16 16:02:35.586035 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:35.586007    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:02:36.791697 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:36.791669    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:02:38.038127 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:38.038093    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:02:38.096986 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:38.096954    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:02:40.692108 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:40.692073    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:02:40.821359 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:40.821328    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:02:48.646036 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:48.646001    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:02:48.764306 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:02:48.764276    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:03:05.355340 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:03:05.355304    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:03:13.998606 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:03:13.998446    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:03:14.088369 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:03:14.088313    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:03:14.178568 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:03:14.178533    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:03:14.322846 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:03:14.322786    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:03:15.791591 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:03:15.791547    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:03:15.857064 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:03:15.857033    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:03:16.545181 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:03:16.545147    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:04:34.275996 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:04:34.275960    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:04:34.347276 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:04:34.347242    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:04:34.411719 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:04:34.411694    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:04:34.533457 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:04:34.533429    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:04:34.625451 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:04:34.625423    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:04:35.679158 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:04:35.679125    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:04:35.736174 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:04:35.736145    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:04:51.082423 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:04:51.082385    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-87g7k"]
Sep 16 16:04:51.086607 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:04:51.086582    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-87g7k"
Sep 16 16:04:51.089602 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:04:51.089578    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 16:04:51.089834 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:04:51.089621    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 16:04:51.089952 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:04:51.089678    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 16 16:04:51.090920 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:04:51.090896    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 16 16:04:51.091029 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:04:51.090927    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-75bhf\""
Sep 16 16:04:51.092468 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:04:51.092447    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-87g7k"]
Sep 16 16:04:51.191324 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:04:51.191288    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/295f4d72-af70-4a3b-ad51-79874ee0345b-model-cache\") pod \"s3-init-87g7k\" (UID: \"295f4d72-af70-4a3b-ad51-79874ee0345b\") " pod="kserve/s3-init-87g7k"
Sep 16 16:04:51.191613 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:04:51.191595    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgw88\" (UniqueName: \"kubernetes.io/projected/295f4d72-af70-4a3b-ad51-79874ee0345b-kube-api-access-xgw88\") pod \"s3-init-87g7k\" (UID: \"295f4d72-af70-4a3b-ad51-79874ee0345b\") " pod="kserve/s3-init-87g7k"
Sep 16 16:04:51.291951 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:04:51.291917    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgw88\" (UniqueName: \"kubernetes.io/projected/295f4d72-af70-4a3b-ad51-79874ee0345b-kube-api-access-xgw88\") pod \"s3-init-87g7k\" (UID: \"295f4d72-af70-4a3b-ad51-79874ee0345b\") " pod="kserve/s3-init-87g7k"
Sep 16 16:04:51.292079 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:04:51.291962    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/295f4d72-af70-4a3b-ad51-79874ee0345b-model-cache\") pod \"s3-init-87g7k\" (UID: \"295f4d72-af70-4a3b-ad51-79874ee0345b\") " pod="kserve/s3-init-87g7k"
Sep 16 16:04:51.292287 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:04:51.292272    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/295f4d72-af70-4a3b-ad51-79874ee0345b-model-cache\") pod \"s3-init-87g7k\" (UID: \"295f4d72-af70-4a3b-ad51-79874ee0345b\") " pod="kserve/s3-init-87g7k"
Sep 16 16:04:51.301309 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:04:51.301274    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgw88\" (UniqueName: \"kubernetes.io/projected/295f4d72-af70-4a3b-ad51-79874ee0345b-kube-api-access-xgw88\") pod \"s3-init-87g7k\" (UID: \"295f4d72-af70-4a3b-ad51-79874ee0345b\") " pod="kserve/s3-init-87g7k"
Sep 16 16:04:51.412903 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:04:51.412816    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-87g7k"
Sep 16 16:04:51.681840 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:04:51.681814    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-87g7k"]
Sep 16 16:04:51.684318 ip-10-0-136-228 kubenswrapper[2554]: W0916 16:04:51.684297    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod295f4d72_af70_4a3b_ad51_79874ee0345b.slice/crio-4ec5ade49487c4b808a51ceb53a4a77c60c7aeacc4eec149a51773e17dcad63d WatchSource:0}: Error finding container 4ec5ade49487c4b808a51ceb53a4a77c60c7aeacc4eec149a51773e17dcad63d: Status 404 returned error can't find the container with id 4ec5ade49487c4b808a51ceb53a4a77c60c7aeacc4eec149a51773e17dcad63d
Sep 16 16:04:51.686180 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:04:51.686166    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 16:04:52.142881 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:04:52.142781    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-87g7k" event={"ID":"295f4d72-af70-4a3b-ad51-79874ee0345b","Type":"ContainerStarted","Data":"4ec5ade49487c4b808a51ceb53a4a77c60c7aeacc4eec149a51773e17dcad63d"}
Sep 16 16:04:56.160494 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:04:56.160461    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-87g7k" event={"ID":"295f4d72-af70-4a3b-ad51-79874ee0345b","Type":"ContainerStarted","Data":"34bbe03f6d4d1886637510b8e91ca790b8c7c2eec9488b8ca246d574679541ff"}
Sep 16 16:04:59.172922 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:04:59.172888    2554 generic.go:356] "Generic (PLEG): container finished" podID="295f4d72-af70-4a3b-ad51-79874ee0345b" containerID="34bbe03f6d4d1886637510b8e91ca790b8c7c2eec9488b8ca246d574679541ff" exitCode=0
Sep 16 16:04:59.173392 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:04:59.172962    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-87g7k" event={"ID":"295f4d72-af70-4a3b-ad51-79874ee0345b","Type":"ContainerDied","Data":"34bbe03f6d4d1886637510b8e91ca790b8c7c2eec9488b8ca246d574679541ff"}
Sep 16 16:05:02.783328 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:05:02.783289    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:05:03.595226 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:05:03.595206    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 16 16:05:04.198098 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:05:04.198059    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-87g7k" event={"ID":"295f4d72-af70-4a3b-ad51-79874ee0345b","Type":"ContainerStarted","Data":"96a5444dd953e4111a62d1d860fbe69a7a945c2e4b13e49da78f6a3b7ff1c190"}
Sep 16 16:05:04.215480 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:05:04.215428    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-87g7k" podStartSLOduration=2.280372636 podStartE2EDuration="13.215412363s" totalImagesPullingTime="8.187778082s" totalInitContainerRuntime="2.747261645s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:04:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:05:04.214386582 +0000 UTC m=+904.659078759" watchObservedRunningTime="2026-09-16 16:05:04.215412363 +0000 UTC m=+904.660104540"
Sep 16 16:05:29.004599 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:05:29.004540    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:06:09.437576 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:09.437538    2554 generic.go:356] "Generic (PLEG): container finished" podID="295f4d72-af70-4a3b-ad51-79874ee0345b" containerID="96a5444dd953e4111a62d1d860fbe69a7a945c2e4b13e49da78f6a3b7ff1c190" exitCode=0
Sep 16 16:06:09.437988 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:09.437586    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-87g7k" event={"ID":"295f4d72-af70-4a3b-ad51-79874ee0345b","Type":"ContainerDied","Data":"96a5444dd953e4111a62d1d860fbe69a7a945c2e4b13e49da78f6a3b7ff1c190"}
Sep 16 16:06:10.563211 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:10.563186    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-87g7k"
Sep 16 16:06:10.594661 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:10.594634    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgw88\" (UniqueName: \"kubernetes.io/projected/295f4d72-af70-4a3b-ad51-79874ee0345b-kube-api-access-xgw88\") pod \"295f4d72-af70-4a3b-ad51-79874ee0345b\" (UID: \"295f4d72-af70-4a3b-ad51-79874ee0345b\") "
Sep 16 16:06:10.594822 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:10.594727    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/295f4d72-af70-4a3b-ad51-79874ee0345b-model-cache\") pod \"295f4d72-af70-4a3b-ad51-79874ee0345b\" (UID: \"295f4d72-af70-4a3b-ad51-79874ee0345b\") "
Sep 16 16:06:10.596916 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:10.596889    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/295f4d72-af70-4a3b-ad51-79874ee0345b-kube-api-access-xgw88" (OuterVolumeSpecName: "kube-api-access-xgw88") pod "295f4d72-af70-4a3b-ad51-79874ee0345b" (UID: "295f4d72-af70-4a3b-ad51-79874ee0345b"). InnerVolumeSpecName "kube-api-access-xgw88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:06:10.613407 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:10.613366    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/295f4d72-af70-4a3b-ad51-79874ee0345b-model-cache" (OuterVolumeSpecName: "model-cache") pod "295f4d72-af70-4a3b-ad51-79874ee0345b" (UID: "295f4d72-af70-4a3b-ad51-79874ee0345b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:06:10.695618 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:10.695530    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/295f4d72-af70-4a3b-ad51-79874ee0345b-model-cache\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:06:10.695618 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:10.695563    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgw88\" (UniqueName: \"kubernetes.io/projected/295f4d72-af70-4a3b-ad51-79874ee0345b-kube-api-access-xgw88\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:06:11.446113 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:11.446076    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-87g7k" event={"ID":"295f4d72-af70-4a3b-ad51-79874ee0345b","Type":"ContainerDied","Data":"4ec5ade49487c4b808a51ceb53a4a77c60c7aeacc4eec149a51773e17dcad63d"}
Sep 16 16:06:11.446113 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:11.446117    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-87g7k"
Sep 16 16:06:11.446321 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:11.446121    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ec5ade49487c4b808a51ceb53a4a77c60c7aeacc4eec149a51773e17dcad63d"
Sep 16 16:06:11.826642 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:11.826609    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:06:11.924095 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:11.924069    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:06:24.572701 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:24.572627    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:06:24.677264 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:24.677230    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:06:28.902439 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:28.902403    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:06:29.158010 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:29.157932    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:06:29.368302 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:29.368267    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 16:06:29.368546 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:29.368531    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="295f4d72-af70-4a3b-ad51-79874ee0345b" containerName="download-hf-model"
Sep 16 16:06:29.368588 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:29.368548    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="295f4d72-af70-4a3b-ad51-79874ee0345b" containerName="download-hf-model"
Sep 16 16:06:29.368588 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:29.368558    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="295f4d72-af70-4a3b-ad51-79874ee0345b" containerName="s3-init"
Sep 16 16:06:29.368588 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:29.368565    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="295f4d72-af70-4a3b-ad51-79874ee0345b" containerName="s3-init"
Sep 16 16:06:29.368680 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:29.368624    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="295f4d72-af70-4a3b-ad51-79874ee0345b" containerName="s3-init"
Sep 16 16:06:29.371756 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:29.371731    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 16:06:29.374499 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:29.374476    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 16 16:06:29.375849 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:29.375825    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 16 16:06:29.375849 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:29.375839    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-mfjsx\""
Sep 16 16:06:29.376014 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:29.375830    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 16 16:06:29.376014 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:29.375831    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 16:06:29.376014 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:29.375840    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-j795p\""
Sep 16 16:06:29.376014 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:29.375828    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 16 16:06:29.386127 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:29.386102    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 16:06:29.428651 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:29.428566    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75ecbd33-d755-4900-ac4d-3cf9e7954507-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"75ecbd33-d755-4900-ac4d-3cf9e7954507\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 16:06:29.428651 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:29.428600    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75ecbd33-d755-4900-ac4d-3cf9e7954507-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"75ecbd33-d755-4900-ac4d-3cf9e7954507\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 16:06:29.428651 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:29.428625    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt8kp\" (UniqueName: \"kubernetes.io/projected/75ecbd33-d755-4900-ac4d-3cf9e7954507-kube-api-access-pt8kp\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"75ecbd33-d755-4900-ac4d-3cf9e7954507\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 16:06:29.428651 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:29.428648    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75ecbd33-d755-4900-ac4d-3cf9e7954507-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"75ecbd33-d755-4900-ac4d-3cf9e7954507\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 16:06:29.428993 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:29.428705    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75ecbd33-d755-4900-ac4d-3cf9e7954507-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"75ecbd33-d755-4900-ac4d-3cf9e7954507\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 16:06:29.428993 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:29.428731    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75ecbd33-d755-4900-ac4d-3cf9e7954507-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"75ecbd33-d755-4900-ac4d-3cf9e7954507\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 16:06:29.428993 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:29.428851    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75ecbd33-d755-4900-ac4d-3cf9e7954507-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"75ecbd33-d755-4900-ac4d-3cf9e7954507\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 16:06:29.428993 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:29.428895    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75ecbd33-d755-4900-ac4d-3cf9e7954507-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"75ecbd33-d755-4900-ac4d-3cf9e7954507\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 16:06:29.530086 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:29.530052    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75ecbd33-d755-4900-ac4d-3cf9e7954507-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"75ecbd33-d755-4900-ac4d-3cf9e7954507\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 16:06:29.530086 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:29.530086    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75ecbd33-d755-4900-ac4d-3cf9e7954507-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"75ecbd33-d755-4900-ac4d-3cf9e7954507\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 16:06:29.530327 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:29.530109    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt8kp\" (UniqueName: \"kubernetes.io/projected/75ecbd33-d755-4900-ac4d-3cf9e7954507-kube-api-access-pt8kp\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"75ecbd33-d755-4900-ac4d-3cf9e7954507\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 16:06:29.530327 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:29.530134    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75ecbd33-d755-4900-ac4d-3cf9e7954507-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"75ecbd33-d755-4900-ac4d-3cf9e7954507\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 16:06:29.530327 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:29.530247    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75ecbd33-d755-4900-ac4d-3cf9e7954507-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"75ecbd33-d755-4900-ac4d-3cf9e7954507\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 16:06:29.530327 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:29.530289    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75ecbd33-d755-4900-ac4d-3cf9e7954507-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"75ecbd33-d755-4900-ac4d-3cf9e7954507\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 16:06:29.530533 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:29.530369    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75ecbd33-d755-4900-ac4d-3cf9e7954507-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"75ecbd33-d755-4900-ac4d-3cf9e7954507\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 16:06:29.530533 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:29.530424    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75ecbd33-d755-4900-ac4d-3cf9e7954507-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"75ecbd33-d755-4900-ac4d-3cf9e7954507\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 16:06:29.530533 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:29.530429    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75ecbd33-d755-4900-ac4d-3cf9e7954507-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"75ecbd33-d755-4900-ac4d-3cf9e7954507\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 16:06:29.530678 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:29.530566    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75ecbd33-d755-4900-ac4d-3cf9e7954507-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"75ecbd33-d755-4900-ac4d-3cf9e7954507\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 16:06:29.530732 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:29.530684    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75ecbd33-d755-4900-ac4d-3cf9e7954507-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"75ecbd33-d755-4900-ac4d-3cf9e7954507\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 16:06:29.530732 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:29.530719    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75ecbd33-d755-4900-ac4d-3cf9e7954507-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"75ecbd33-d755-4900-ac4d-3cf9e7954507\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 16:06:29.530732 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:29.530724    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75ecbd33-d755-4900-ac4d-3cf9e7954507-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"75ecbd33-d755-4900-ac4d-3cf9e7954507\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 16:06:29.532371 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:29.532351    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75ecbd33-d755-4900-ac4d-3cf9e7954507-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"75ecbd33-d755-4900-ac4d-3cf9e7954507\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 16:06:29.532571 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:29.532551    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75ecbd33-d755-4900-ac4d-3cf9e7954507-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"75ecbd33-d755-4900-ac4d-3cf9e7954507\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 16:06:29.539675 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:29.539653    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt8kp\" (UniqueName: \"kubernetes.io/projected/75ecbd33-d755-4900-ac4d-3cf9e7954507-kube-api-access-pt8kp\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"75ecbd33-d755-4900-ac4d-3cf9e7954507\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 16:06:29.681957 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:29.681877    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 16:06:29.734674 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:29.734640    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:06:29.837705 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:29.837669    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:06:29.958679 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:29.958486    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 16:06:29.961152 ip-10-0-136-228 kubenswrapper[2554]: W0916 16:06:29.961127    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75ecbd33_d755_4900_ac4d_3cf9e7954507.slice/crio-deca0bf2a6ebac492a2d6cbd14340e7f9bf37ab34538e750d373dd5c15b59c37 WatchSource:0}: Error finding container deca0bf2a6ebac492a2d6cbd14340e7f9bf37ab34538e750d373dd5c15b59c37: Status 404 returned error can't find the container with id deca0bf2a6ebac492a2d6cbd14340e7f9bf37ab34538e750d373dd5c15b59c37
Sep 16 16:06:30.514182 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:30.514143    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"75ecbd33-d755-4900-ac4d-3cf9e7954507","Type":"ContainerStarted","Data":"c25ad27d7921e6ed55fd01c593cf13455247049014e6943cce0580df7d2e0851"}
Sep 16 16:06:30.514352 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:30.514226    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"75ecbd33-d755-4900-ac4d-3cf9e7954507","Type":"ContainerStarted","Data":"deca0bf2a6ebac492a2d6cbd14340e7f9bf37ab34538e750d373dd5c15b59c37"}
Sep 16 16:06:32.523181 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:32.523096    2554 generic.go:356] "Generic (PLEG): container finished" podID="75ecbd33-d755-4900-ac4d-3cf9e7954507" containerID="c25ad27d7921e6ed55fd01c593cf13455247049014e6943cce0580df7d2e0851" exitCode=0
Sep 16 16:06:32.523181 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:32.523150    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"75ecbd33-d755-4900-ac4d-3cf9e7954507","Type":"ContainerDied","Data":"c25ad27d7921e6ed55fd01c593cf13455247049014e6943cce0580df7d2e0851"}
Sep 16 16:06:44.087164 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:44.087108    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bb4d7"]
Sep 16 16:06:44.092461 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:44.092420    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bb4d7"
Sep 16 16:06:44.095698 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:44.095594    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 16 16:06:44.096931 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:44.096911    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 16 16:06:44.097164 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:44.097140    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-fd8sp\""
Sep 16 16:06:44.097393 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:44.097357    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 16 16:06:44.098987 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:44.098964    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bb4d7"]
Sep 16 16:06:44.260827 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:44.260766    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f46d911-5d34-4305-857a-42212946fd62-home\") pod \"fc-smoke-test-kserve-7f4687897-bb4d7\" (UID: \"3f46d911-5d34-4305-857a-42212946fd62\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bb4d7"
Sep 16 16:06:44.261015 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:44.260835    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f46d911-5d34-4305-857a-42212946fd62-dshm\") pod \"fc-smoke-test-kserve-7f4687897-bb4d7\" (UID: \"3f46d911-5d34-4305-857a-42212946fd62\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bb4d7"
Sep 16 16:06:44.261015 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:44.260872    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f46d911-5d34-4305-857a-42212946fd62-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-bb4d7\" (UID: \"3f46d911-5d34-4305-857a-42212946fd62\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bb4d7"
Sep 16 16:06:44.261015 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:44.260903    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf6wr\" (UniqueName: \"kubernetes.io/projected/3f46d911-5d34-4305-857a-42212946fd62-kube-api-access-kf6wr\") pod \"fc-smoke-test-kserve-7f4687897-bb4d7\" (UID: \"3f46d911-5d34-4305-857a-42212946fd62\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bb4d7"
Sep 16 16:06:44.261015 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:44.260984    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f46d911-5d34-4305-857a-42212946fd62-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-bb4d7\" (UID: \"3f46d911-5d34-4305-857a-42212946fd62\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bb4d7"
Sep 16 16:06:44.261222 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:44.261023    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f46d911-5d34-4305-857a-42212946fd62-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-bb4d7\" (UID: \"3f46d911-5d34-4305-857a-42212946fd62\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bb4d7"
Sep 16 16:06:44.329148 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:44.329075    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:06:44.362292 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:44.362211    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f46d911-5d34-4305-857a-42212946fd62-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-bb4d7\" (UID: \"3f46d911-5d34-4305-857a-42212946fd62\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bb4d7"
Sep 16 16:06:44.362292 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:44.362267    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f46d911-5d34-4305-857a-42212946fd62-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-bb4d7\" (UID: \"3f46d911-5d34-4305-857a-42212946fd62\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bb4d7"
Sep 16 16:06:44.362503 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:44.362321    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f46d911-5d34-4305-857a-42212946fd62-home\") pod \"fc-smoke-test-kserve-7f4687897-bb4d7\" (UID: \"3f46d911-5d34-4305-857a-42212946fd62\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bb4d7"
Sep 16 16:06:44.362503 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:44.362348    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f46d911-5d34-4305-857a-42212946fd62-dshm\") pod \"fc-smoke-test-kserve-7f4687897-bb4d7\" (UID: \"3f46d911-5d34-4305-857a-42212946fd62\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bb4d7"
Sep 16 16:06:44.362503 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:44.362377    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f46d911-5d34-4305-857a-42212946fd62-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-bb4d7\" (UID: \"3f46d911-5d34-4305-857a-42212946fd62\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bb4d7"
Sep 16 16:06:44.362503 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:44.362405    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf6wr\" (UniqueName: \"kubernetes.io/projected/3f46d911-5d34-4305-857a-42212946fd62-kube-api-access-kf6wr\") pod \"fc-smoke-test-kserve-7f4687897-bb4d7\" (UID: \"3f46d911-5d34-4305-857a-42212946fd62\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bb4d7"
Sep 16 16:06:44.363146 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:44.363113    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f46d911-5d34-4305-857a-42212946fd62-home\") pod \"fc-smoke-test-kserve-7f4687897-bb4d7\" (UID: \"3f46d911-5d34-4305-857a-42212946fd62\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bb4d7"
Sep 16 16:06:44.363386 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:44.363343    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f46d911-5d34-4305-857a-42212946fd62-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-bb4d7\" (UID: \"3f46d911-5d34-4305-857a-42212946fd62\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bb4d7"
Sep 16 16:06:44.363496 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:44.363445    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f46d911-5d34-4305-857a-42212946fd62-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-bb4d7\" (UID: \"3f46d911-5d34-4305-857a-42212946fd62\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bb4d7"
Sep 16 16:06:44.366100 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:44.366074    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f46d911-5d34-4305-857a-42212946fd62-dshm\") pod \"fc-smoke-test-kserve-7f4687897-bb4d7\" (UID: \"3f46d911-5d34-4305-857a-42212946fd62\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bb4d7"
Sep 16 16:06:44.367007 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:44.366965    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f46d911-5d34-4305-857a-42212946fd62-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-bb4d7\" (UID: \"3f46d911-5d34-4305-857a-42212946fd62\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bb4d7"
Sep 16 16:06:44.371718 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:44.371673    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf6wr\" (UniqueName: \"kubernetes.io/projected/3f46d911-5d34-4305-857a-42212946fd62-kube-api-access-kf6wr\") pod \"fc-smoke-test-kserve-7f4687897-bb4d7\" (UID: \"3f46d911-5d34-4305-857a-42212946fd62\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bb4d7"
Sep 16 16:06:44.385668 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:44.385632    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:06:44.406859 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:44.406715    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bb4d7"
Sep 16 16:06:45.266040 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:45.265990    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bb4d7"]
Sep 16 16:06:45.268140 ip-10-0-136-228 kubenswrapper[2554]: W0916 16:06:45.268109    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f46d911_5d34_4305_857a_42212946fd62.slice/crio-4b875cf2ad691d1a517db0aeab6df37458f520c5162a96a05f17f2a64ac65c77 WatchSource:0}: Error finding container 4b875cf2ad691d1a517db0aeab6df37458f520c5162a96a05f17f2a64ac65c77: Status 404 returned error can't find the container with id 4b875cf2ad691d1a517db0aeab6df37458f520c5162a96a05f17f2a64ac65c77
Sep 16 16:06:45.588827 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:45.588768    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bb4d7" event={"ID":"3f46d911-5d34-4305-857a-42212946fd62","Type":"ContainerStarted","Data":"4b875cf2ad691d1a517db0aeab6df37458f520c5162a96a05f17f2a64ac65c77"}
Sep 16 16:06:55.212503 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:55.212466    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:06:59.660752 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:06:59.660720    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:07:00.657229 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:00.657192    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"75ecbd33-d755-4900-ac4d-3cf9e7954507","Type":"ContainerStarted","Data":"e6c6f445f2003564f2d72d4e66aff15fd9d130b39f0181794ab7c12c8df7c861"}
Sep 16 16:07:00.658861 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:00.658835    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bb4d7" event={"ID":"3f46d911-5d34-4305-857a-42212946fd62","Type":"ContainerStarted","Data":"e77fc8f521a2d102717535bf3f7caab22fea047b21cd00c52c95dc933a4487b6"}
Sep 16 16:07:00.682122 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:00.682089    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 16:07:00.693372 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:00.693322    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=1.851796368 podStartE2EDuration="31.693307648s" totalImagesPullingTime="27.666670013s" totalInitContainerRuntime="2.174841267s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:06:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:07:00.690960454 +0000 UTC m=+1021.135652656" watchObservedRunningTime="2026-09-16 16:07:00.693307648 +0000 UTC m=+1021.137999869"
Sep 16 16:07:00.707737 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:00.707673    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bb4d7" podStartSLOduration=1.69830752 podStartE2EDuration="16.707654151s" totalImagesPullingTime="15.009346631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:06:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:07:00.707447044 +0000 UTC m=+1021.152139243" watchObservedRunningTime="2026-09-16 16:07:00.707654151 +0000 UTC m=+1021.152346331"
Sep 16 16:07:01.406880 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:01.406844    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bb4d7"
Sep 16 16:07:04.407530 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:04.407484    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bb4d7"
Sep 16 16:07:04.420286 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:04.420261    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bb4d7"
Sep 16 16:07:04.687217 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:04.687136    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bb4d7"
Sep 16 16:07:09.682292 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:09.682251    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 16:07:09.683832 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:09.683782    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="75ecbd33-d755-4900-ac4d-3cf9e7954507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.20:8000/health\": dial tcp 10.132.0.20:8000: connect: connection refused"
Sep 16 16:07:19.682454 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:19.682410    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="75ecbd33-d755-4900-ac4d-3cf9e7954507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.20:8000/health\": dial tcp 10.132.0.20:8000: connect: connection refused"
Sep 16 16:07:23.310030 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:23.309994    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bb4d7"]
Sep 16 16:07:23.314006 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:23.313972    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:07:23.320302 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:07:23.320261    2554 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 16:07:23.320302 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:07:23.320302 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 16:07:23.320302 ip-10-0-136-228 kubenswrapper[2554]:  > containerID="e77fc8f521a2d102717535bf3f7caab22fea047b21cd00c52c95dc933a4487b6" cmd=["/bin/sleep","15"]
Sep 16 16:07:23.320604 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:23.320332    2554 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bb4d7" message="" err=<
Sep 16 16:07:23.320604 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:07:23.320604 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 16:07:23.320604 ip-10-0-136-228 kubenswrapper[2554]:  >
Sep 16 16:07:23.320604 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:07:23.320350    2554 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 16:07:23.320604 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:07:23.320604 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 16:07:23.320604 ip-10-0-136-228 kubenswrapper[2554]:  > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bb4d7" podUID="3f46d911-5d34-4305-857a-42212946fd62" containerName="main" containerID="cri-o://e77fc8f521a2d102717535bf3f7caab22fea047b21cd00c52c95dc933a4487b6"
Sep 16 16:07:23.320604 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:23.320400    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bb4d7" podUID="3f46d911-5d34-4305-857a-42212946fd62" containerName="main" containerID="cri-o://e77fc8f521a2d102717535bf3f7caab22fea047b21cd00c52c95dc933a4487b6" gracePeriod=60
Sep 16 16:07:23.371585 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:23.371544    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:07:23.658841 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:23.658817    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bb4d7"
Sep 16 16:07:23.757645 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:23.757613    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f46d911-5d34-4305-857a-42212946fd62-tls-certs\") pod \"3f46d911-5d34-4305-857a-42212946fd62\" (UID: \"3f46d911-5d34-4305-857a-42212946fd62\") "
Sep 16 16:07:23.757874 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:23.757660    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f46d911-5d34-4305-857a-42212946fd62-model-cache\") pod \"3f46d911-5d34-4305-857a-42212946fd62\" (UID: \"3f46d911-5d34-4305-857a-42212946fd62\") "
Sep 16 16:07:23.757874 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:23.757780    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kf6wr\" (UniqueName: \"kubernetes.io/projected/3f46d911-5d34-4305-857a-42212946fd62-kube-api-access-kf6wr\") pod \"3f46d911-5d34-4305-857a-42212946fd62\" (UID: \"3f46d911-5d34-4305-857a-42212946fd62\") "
Sep 16 16:07:23.758023 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:23.757877    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f46d911-5d34-4305-857a-42212946fd62-dshm\") pod \"3f46d911-5d34-4305-857a-42212946fd62\" (UID: \"3f46d911-5d34-4305-857a-42212946fd62\") "
Sep 16 16:07:23.758023 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:23.757917    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f46d911-5d34-4305-857a-42212946fd62-tmp-dir\") pod \"3f46d911-5d34-4305-857a-42212946fd62\" (UID: \"3f46d911-5d34-4305-857a-42212946fd62\") "
Sep 16 16:07:23.758023 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:23.757948    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f46d911-5d34-4305-857a-42212946fd62-home\") pod \"3f46d911-5d34-4305-857a-42212946fd62\" (UID: \"3f46d911-5d34-4305-857a-42212946fd62\") "
Sep 16 16:07:23.758023 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:23.757983    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f46d911-5d34-4305-857a-42212946fd62-model-cache" (OuterVolumeSpecName: "model-cache") pod "3f46d911-5d34-4305-857a-42212946fd62" (UID: "3f46d911-5d34-4305-857a-42212946fd62"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:07:23.758230 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:23.758153    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f46d911-5d34-4305-857a-42212946fd62-model-cache\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:07:23.758293 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:23.758264    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f46d911-5d34-4305-857a-42212946fd62-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3f46d911-5d34-4305-857a-42212946fd62" (UID: "3f46d911-5d34-4305-857a-42212946fd62"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:07:23.758401 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:23.758353    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f46d911-5d34-4305-857a-42212946fd62-home" (OuterVolumeSpecName: "home") pod "3f46d911-5d34-4305-857a-42212946fd62" (UID: "3f46d911-5d34-4305-857a-42212946fd62"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:07:23.758535 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:23.758509    2554 generic.go:356] "Generic (PLEG): container finished" podID="3f46d911-5d34-4305-857a-42212946fd62" containerID="e77fc8f521a2d102717535bf3f7caab22fea047b21cd00c52c95dc933a4487b6" exitCode=0
Sep 16 16:07:23.758604 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:23.758564    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bb4d7" event={"ID":"3f46d911-5d34-4305-857a-42212946fd62","Type":"ContainerDied","Data":"e77fc8f521a2d102717535bf3f7caab22fea047b21cd00c52c95dc933a4487b6"}
Sep 16 16:07:23.758604 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:23.758575    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bb4d7"
Sep 16 16:07:23.758715 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:23.758614    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bb4d7" event={"ID":"3f46d911-5d34-4305-857a-42212946fd62","Type":"ContainerDied","Data":"4b875cf2ad691d1a517db0aeab6df37458f520c5162a96a05f17f2a64ac65c77"}
Sep 16 16:07:23.758715 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:23.758648    2554 scope.go:117] "RemoveContainer" containerID="e77fc8f521a2d102717535bf3f7caab22fea047b21cd00c52c95dc933a4487b6"
Sep 16 16:07:23.760350 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:23.760320    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f46d911-5d34-4305-857a-42212946fd62-kube-api-access-kf6wr" (OuterVolumeSpecName: "kube-api-access-kf6wr") pod "3f46d911-5d34-4305-857a-42212946fd62" (UID: "3f46d911-5d34-4305-857a-42212946fd62"). InnerVolumeSpecName "kube-api-access-kf6wr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:07:23.760350 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:23.760358    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f46d911-5d34-4305-857a-42212946fd62-dshm" (OuterVolumeSpecName: "dshm") pod "3f46d911-5d34-4305-857a-42212946fd62" (UID: "3f46d911-5d34-4305-857a-42212946fd62"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:07:23.760350 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:23.760390    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f46d911-5d34-4305-857a-42212946fd62-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3f46d911-5d34-4305-857a-42212946fd62" (UID: "3f46d911-5d34-4305-857a-42212946fd62"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:07:23.774849 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:23.774826    2554 scope.go:117] "RemoveContainer" containerID="e77fc8f521a2d102717535bf3f7caab22fea047b21cd00c52c95dc933a4487b6"
Sep 16 16:07:23.775204 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:07:23.775185    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e77fc8f521a2d102717535bf3f7caab22fea047b21cd00c52c95dc933a4487b6\": container with ID starting with e77fc8f521a2d102717535bf3f7caab22fea047b21cd00c52c95dc933a4487b6 not found: ID does not exist" containerID="e77fc8f521a2d102717535bf3f7caab22fea047b21cd00c52c95dc933a4487b6"
Sep 16 16:07:23.775260 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:23.775213    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e77fc8f521a2d102717535bf3f7caab22fea047b21cd00c52c95dc933a4487b6"} err="failed to get container status \"e77fc8f521a2d102717535bf3f7caab22fea047b21cd00c52c95dc933a4487b6\": rpc error: code = NotFound desc = could not find container \"e77fc8f521a2d102717535bf3f7caab22fea047b21cd00c52c95dc933a4487b6\": container with ID starting with e77fc8f521a2d102717535bf3f7caab22fea047b21cd00c52c95dc933a4487b6 not found: ID does not exist"
Sep 16 16:07:23.858786 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:23.858703    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf6wr\" (UniqueName: \"kubernetes.io/projected/3f46d911-5d34-4305-857a-42212946fd62-kube-api-access-kf6wr\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:07:23.858786 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:23.858736    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f46d911-5d34-4305-857a-42212946fd62-dshm\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:07:23.858786 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:23.858747    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f46d911-5d34-4305-857a-42212946fd62-tmp-dir\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:07:23.858786 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:23.858756    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f46d911-5d34-4305-857a-42212946fd62-home\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:07:23.858786 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:23.858764    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f46d911-5d34-4305-857a-42212946fd62-tls-certs\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:07:24.080116 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:24.080080    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bb4d7"]
Sep 16 16:07:24.081994 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:24.081960    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bb4d7"]
Sep 16 16:07:26.076889 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:26.076863    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f46d911-5d34-4305-857a-42212946fd62" path="/var/lib/kubelet/pods/3f46d911-5d34-4305-857a-42212946fd62/volumes"
Sep 16 16:07:29.683192 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:29.683150    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="75ecbd33-d755-4900-ac4d-3cf9e7954507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.20:8000/health\": dial tcp 10.132.0.20:8000: connect: connection refused"
Sep 16 16:07:39.683080 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:39.683035    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="75ecbd33-d755-4900-ac4d-3cf9e7954507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.20:8000/health\": dial tcp 10.132.0.20:8000: connect: connection refused"
Sep 16 16:07:41.905131 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:41.905100    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:07:42.002955 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:42.002921    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:07:49.683382 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:49.683339    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="75ecbd33-d755-4900-ac4d-3cf9e7954507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.20:8000/health\": dial tcp 10.132.0.20:8000: connect: connection refused"
Sep 16 16:07:59.683271 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:07:59.683217    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="75ecbd33-d755-4900-ac4d-3cf9e7954507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.20:8000/health\": dial tcp 10.132.0.20:8000: connect: connection refused"
Sep 16 16:08:09.682804 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:08:09.682755    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="75ecbd33-d755-4900-ac4d-3cf9e7954507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.20:8000/health\": dial tcp 10.132.0.20:8000: connect: connection refused"
Sep 16 16:08:13.376311 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:08:13.376273    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:08:16.137269 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:08:16.137228    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:08:19.682612 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:08:19.682563    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="75ecbd33-d755-4900-ac4d-3cf9e7954507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.20:8000/health\": dial tcp 10.132.0.20:8000: connect: connection refused"
Sep 16 16:08:19.919851 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:08:19.919820    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:08:25.896694 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:08:25.896651    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:08:25.997696 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:08:25.997661    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:08:27.613549 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:08:27.613497    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:08:28.829357 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:08:28.829327    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:08:29.683012 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:08:29.682974    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="75ecbd33-d755-4900-ac4d-3cf9e7954507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.20:8000/health\": dial tcp 10.132.0.20:8000: connect: connection refused"
Sep 16 16:08:39.692022 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:08:39.691985    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 16:08:39.702640 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:08:39.702593    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 16:08:39.978216 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:08:39.978127    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:08:48.353001 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:08:48.352963    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:08:48.387577 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:08:48.387544    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 16:08:48.449286 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:08:48.449256    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:08:53.173081 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:08:53.173048    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:08:53.231099 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:08:53.231065    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:09:03.410626 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:03.410572    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="75ecbd33-d755-4900-ac4d-3cf9e7954507" containerName="main" containerID="cri-o://e6c6f445f2003564f2d72d4e66aff15fd9d130b39f0181794ab7c12c8df7c861" gracePeriod=45
Sep 16 16:09:03.692788 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:03.692698    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="75ecbd33-d755-4900-ac4d-3cf9e7954507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.20:8000/health\": dial tcp 10.132.0.20:8000: connect: connection refused"
Sep 16 16:09:04.146985 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:04.146957    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 16:09:04.161656 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:04.161628    2554 generic.go:356] "Generic (PLEG): container finished" podID="75ecbd33-d755-4900-ac4d-3cf9e7954507" containerID="e6c6f445f2003564f2d72d4e66aff15fd9d130b39f0181794ab7c12c8df7c861" exitCode=0
Sep 16 16:09:04.161829 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:04.161699    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 16:09:04.161829 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:04.161716    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"75ecbd33-d755-4900-ac4d-3cf9e7954507","Type":"ContainerDied","Data":"e6c6f445f2003564f2d72d4e66aff15fd9d130b39f0181794ab7c12c8df7c861"}
Sep 16 16:09:04.161829 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:04.161815    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"75ecbd33-d755-4900-ac4d-3cf9e7954507","Type":"ContainerDied","Data":"deca0bf2a6ebac492a2d6cbd14340e7f9bf37ab34538e750d373dd5c15b59c37"}
Sep 16 16:09:04.161968 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:04.161850    2554 scope.go:117] "RemoveContainer" containerID="e6c6f445f2003564f2d72d4e66aff15fd9d130b39f0181794ab7c12c8df7c861"
Sep 16 16:09:04.169831 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:04.169777    2554 scope.go:117] "RemoveContainer" containerID="c25ad27d7921e6ed55fd01c593cf13455247049014e6943cce0580df7d2e0851"
Sep 16 16:09:04.177728 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:04.177706    2554 scope.go:117] "RemoveContainer" containerID="e6c6f445f2003564f2d72d4e66aff15fd9d130b39f0181794ab7c12c8df7c861"
Sep 16 16:09:04.178179 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:09:04.178158    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6c6f445f2003564f2d72d4e66aff15fd9d130b39f0181794ab7c12c8df7c861\": container with ID starting with e6c6f445f2003564f2d72d4e66aff15fd9d130b39f0181794ab7c12c8df7c861 not found: ID does not exist" containerID="e6c6f445f2003564f2d72d4e66aff15fd9d130b39f0181794ab7c12c8df7c861"
Sep 16 16:09:04.178252 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:04.178187    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6c6f445f2003564f2d72d4e66aff15fd9d130b39f0181794ab7c12c8df7c861"} err="failed to get container status \"e6c6f445f2003564f2d72d4e66aff15fd9d130b39f0181794ab7c12c8df7c861\": rpc error: code = NotFound desc = could not find container \"e6c6f445f2003564f2d72d4e66aff15fd9d130b39f0181794ab7c12c8df7c861\": container with ID starting with e6c6f445f2003564f2d72d4e66aff15fd9d130b39f0181794ab7c12c8df7c861 not found: ID does not exist"
Sep 16 16:09:04.178252 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:04.178207    2554 scope.go:117] "RemoveContainer" containerID="c25ad27d7921e6ed55fd01c593cf13455247049014e6943cce0580df7d2e0851"
Sep 16 16:09:04.178517 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:09:04.178490    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c25ad27d7921e6ed55fd01c593cf13455247049014e6943cce0580df7d2e0851\": container with ID starting with c25ad27d7921e6ed55fd01c593cf13455247049014e6943cce0580df7d2e0851 not found: ID does not exist" containerID="c25ad27d7921e6ed55fd01c593cf13455247049014e6943cce0580df7d2e0851"
Sep 16 16:09:04.178589 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:04.178528    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c25ad27d7921e6ed55fd01c593cf13455247049014e6943cce0580df7d2e0851"} err="failed to get container status \"c25ad27d7921e6ed55fd01c593cf13455247049014e6943cce0580df7d2e0851\": rpc error: code = NotFound desc = could not find container \"c25ad27d7921e6ed55fd01c593cf13455247049014e6943cce0580df7d2e0851\": container with ID starting with c25ad27d7921e6ed55fd01c593cf13455247049014e6943cce0580df7d2e0851 not found: ID does not exist"
Sep 16 16:09:04.246943 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:04.246913    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75ecbd33-d755-4900-ac4d-3cf9e7954507-home\") pod \"75ecbd33-d755-4900-ac4d-3cf9e7954507\" (UID: \"75ecbd33-d755-4900-ac4d-3cf9e7954507\") "
Sep 16 16:09:04.246943 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:04.246948    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75ecbd33-d755-4900-ac4d-3cf9e7954507-tls-certs\") pod \"75ecbd33-d755-4900-ac4d-3cf9e7954507\" (UID: \"75ecbd33-d755-4900-ac4d-3cf9e7954507\") "
Sep 16 16:09:04.247154 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:04.246977    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pt8kp\" (UniqueName: \"kubernetes.io/projected/75ecbd33-d755-4900-ac4d-3cf9e7954507-kube-api-access-pt8kp\") pod \"75ecbd33-d755-4900-ac4d-3cf9e7954507\" (UID: \"75ecbd33-d755-4900-ac4d-3cf9e7954507\") "
Sep 16 16:09:04.247154 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:04.247011    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75ecbd33-d755-4900-ac4d-3cf9e7954507-kserve-provision-location\") pod \"75ecbd33-d755-4900-ac4d-3cf9e7954507\" (UID: \"75ecbd33-d755-4900-ac4d-3cf9e7954507\") "
Sep 16 16:09:04.247154 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:04.247065    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75ecbd33-d755-4900-ac4d-3cf9e7954507-dshm\") pod \"75ecbd33-d755-4900-ac4d-3cf9e7954507\" (UID: \"75ecbd33-d755-4900-ac4d-3cf9e7954507\") "
Sep 16 16:09:04.247154 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:04.247095    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75ecbd33-d755-4900-ac4d-3cf9e7954507-tmp-dir\") pod \"75ecbd33-d755-4900-ac4d-3cf9e7954507\" (UID: \"75ecbd33-d755-4900-ac4d-3cf9e7954507\") "
Sep 16 16:09:04.247154 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:04.247153    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75ecbd33-d755-4900-ac4d-3cf9e7954507-cabundle-cert\") pod \"75ecbd33-d755-4900-ac4d-3cf9e7954507\" (UID: \"75ecbd33-d755-4900-ac4d-3cf9e7954507\") "
Sep 16 16:09:04.247414 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:04.247182    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75ecbd33-d755-4900-ac4d-3cf9e7954507-model-cache\") pod \"75ecbd33-d755-4900-ac4d-3cf9e7954507\" (UID: \"75ecbd33-d755-4900-ac4d-3cf9e7954507\") "
Sep 16 16:09:04.247623 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:04.247552    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ecbd33-d755-4900-ac4d-3cf9e7954507-model-cache" (OuterVolumeSpecName: "model-cache") pod "75ecbd33-d755-4900-ac4d-3cf9e7954507" (UID: "75ecbd33-d755-4900-ac4d-3cf9e7954507"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:09:04.247623 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:04.247602    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75ecbd33-d755-4900-ac4d-3cf9e7954507-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "75ecbd33-d755-4900-ac4d-3cf9e7954507" (UID: "75ecbd33-d755-4900-ac4d-3cf9e7954507"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 16:09:04.247771 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:04.247748    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ecbd33-d755-4900-ac4d-3cf9e7954507-home" (OuterVolumeSpecName: "home") pod "75ecbd33-d755-4900-ac4d-3cf9e7954507" (UID: "75ecbd33-d755-4900-ac4d-3cf9e7954507"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:09:04.249408 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:04.249377    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ecbd33-d755-4900-ac4d-3cf9e7954507-dshm" (OuterVolumeSpecName: "dshm") pod "75ecbd33-d755-4900-ac4d-3cf9e7954507" (UID: "75ecbd33-d755-4900-ac4d-3cf9e7954507"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:09:04.249529 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:04.249488    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75ecbd33-d755-4900-ac4d-3cf9e7954507-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "75ecbd33-d755-4900-ac4d-3cf9e7954507" (UID: "75ecbd33-d755-4900-ac4d-3cf9e7954507"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:09:04.249594 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:04.249573    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75ecbd33-d755-4900-ac4d-3cf9e7954507-kube-api-access-pt8kp" (OuterVolumeSpecName: "kube-api-access-pt8kp") pod "75ecbd33-d755-4900-ac4d-3cf9e7954507" (UID: "75ecbd33-d755-4900-ac4d-3cf9e7954507"). InnerVolumeSpecName "kube-api-access-pt8kp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:09:04.265097 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:04.265062    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ecbd33-d755-4900-ac4d-3cf9e7954507-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "75ecbd33-d755-4900-ac4d-3cf9e7954507" (UID: "75ecbd33-d755-4900-ac4d-3cf9e7954507"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:09:04.267943 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:04.267920    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ecbd33-d755-4900-ac4d-3cf9e7954507-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "75ecbd33-d755-4900-ac4d-3cf9e7954507" (UID: "75ecbd33-d755-4900-ac4d-3cf9e7954507"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:09:04.348312 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:04.348274    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75ecbd33-d755-4900-ac4d-3cf9e7954507-dshm\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:09:04.348312 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:04.348306    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75ecbd33-d755-4900-ac4d-3cf9e7954507-tmp-dir\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:09:04.348312 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:04.348317    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75ecbd33-d755-4900-ac4d-3cf9e7954507-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:09:04.348549 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:04.348327    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75ecbd33-d755-4900-ac4d-3cf9e7954507-model-cache\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:09:04.348549 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:04.348335    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75ecbd33-d755-4900-ac4d-3cf9e7954507-home\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:09:04.348549 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:04.348342    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75ecbd33-d755-4900-ac4d-3cf9e7954507-tls-certs\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:09:04.348549 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:04.348350    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt8kp\" (UniqueName: \"kubernetes.io/projected/75ecbd33-d755-4900-ac4d-3cf9e7954507-kube-api-access-pt8kp\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:09:04.348549 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:04.348359    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75ecbd33-d755-4900-ac4d-3cf9e7954507-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:09:04.484417 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:04.484387    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 16:09:04.491095 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:04.491070    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 16:09:06.076611 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:06.076575    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75ecbd33-d755-4900-ac4d-3cf9e7954507" path="/var/lib/kubelet/pods/75ecbd33-d755-4900-ac4d-3cf9e7954507/volumes"
Sep 16 16:09:08.561703 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:08.561670    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:09:08.664722 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:08.664687    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:09:36.884705 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:36.884617    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:09:36.977576 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:36.977542    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:09:43.326400 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:43.326364    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:09:45.073698 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:45.073666    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6g6cz"]
Sep 16 16:09:45.074058 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:45.073937    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75ecbd33-d755-4900-ac4d-3cf9e7954507" containerName="storage-initializer"
Sep 16 16:09:45.074058 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:45.073949    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="75ecbd33-d755-4900-ac4d-3cf9e7954507" containerName="storage-initializer"
Sep 16 16:09:45.074058 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:45.073963    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f46d911-5d34-4305-857a-42212946fd62" containerName="main"
Sep 16 16:09:45.074058 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:45.073969    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f46d911-5d34-4305-857a-42212946fd62" containerName="main"
Sep 16 16:09:45.074058 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:45.073979    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75ecbd33-d755-4900-ac4d-3cf9e7954507" containerName="main"
Sep 16 16:09:45.074058 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:45.073985    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="75ecbd33-d755-4900-ac4d-3cf9e7954507" containerName="main"
Sep 16 16:09:45.074058 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:45.074038    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="75ecbd33-d755-4900-ac4d-3cf9e7954507" containerName="main"
Sep 16 16:09:45.074058 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:45.074048    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f46d911-5d34-4305-857a-42212946fd62" containerName="main"
Sep 16 16:09:45.077096 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:45.077079    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6g6cz"
Sep 16 16:09:45.081416 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:45.081394    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-7lwdm\""
Sep 16 16:09:45.081584 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:45.081467    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 16 16:09:45.081584 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:45.081519    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 16 16:09:45.081584 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:45.081534    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 16 16:09:45.086276 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:45.086252    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6g6cz"]
Sep 16 16:09:45.101604 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:45.101579    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:09:45.258657 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:45.258620    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42619d7f-d360-483b-9f9c-6e08b7d9b610-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-6g6cz\" (UID: \"42619d7f-d360-483b-9f9c-6e08b7d9b610\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6g6cz"
Sep 16 16:09:45.258657 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:45.258653    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42619d7f-d360-483b-9f9c-6e08b7d9b610-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-6g6cz\" (UID: \"42619d7f-d360-483b-9f9c-6e08b7d9b610\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6g6cz"
Sep 16 16:09:45.258898 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:45.258675    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h55gt\" (UniqueName: \"kubernetes.io/projected/42619d7f-d360-483b-9f9c-6e08b7d9b610-kube-api-access-h55gt\") pod \"scheduler-inline-config-test-kserve-8655bc757-6g6cz\" (UID: \"42619d7f-d360-483b-9f9c-6e08b7d9b610\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6g6cz"
Sep 16 16:09:45.258898 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:45.258739    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42619d7f-d360-483b-9f9c-6e08b7d9b610-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-6g6cz\" (UID: \"42619d7f-d360-483b-9f9c-6e08b7d9b610\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6g6cz"
Sep 16 16:09:45.258898 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:45.258781    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42619d7f-d360-483b-9f9c-6e08b7d9b610-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-6g6cz\" (UID: \"42619d7f-d360-483b-9f9c-6e08b7d9b610\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6g6cz"
Sep 16 16:09:45.258898 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:45.258826    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42619d7f-d360-483b-9f9c-6e08b7d9b610-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-6g6cz\" (UID: \"42619d7f-d360-483b-9f9c-6e08b7d9b610\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6g6cz"
Sep 16 16:09:45.312966 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:45.312937    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:09:45.359203 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:45.359127    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42619d7f-d360-483b-9f9c-6e08b7d9b610-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-6g6cz\" (UID: \"42619d7f-d360-483b-9f9c-6e08b7d9b610\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6g6cz"
Sep 16 16:09:45.359203 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:45.359158    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42619d7f-d360-483b-9f9c-6e08b7d9b610-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-6g6cz\" (UID: \"42619d7f-d360-483b-9f9c-6e08b7d9b610\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6g6cz"
Sep 16 16:09:45.359398 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:45.359254    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42619d7f-d360-483b-9f9c-6e08b7d9b610-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-6g6cz\" (UID: \"42619d7f-d360-483b-9f9c-6e08b7d9b610\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6g6cz"
Sep 16 16:09:45.359398 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:45.359304    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42619d7f-d360-483b-9f9c-6e08b7d9b610-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-6g6cz\" (UID: \"42619d7f-d360-483b-9f9c-6e08b7d9b610\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6g6cz"
Sep 16 16:09:45.359398 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:45.359320    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42619d7f-d360-483b-9f9c-6e08b7d9b610-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-6g6cz\" (UID: \"42619d7f-d360-483b-9f9c-6e08b7d9b610\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6g6cz"
Sep 16 16:09:45.359398 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:45.359339    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h55gt\" (UniqueName: \"kubernetes.io/projected/42619d7f-d360-483b-9f9c-6e08b7d9b610-kube-api-access-h55gt\") pod \"scheduler-inline-config-test-kserve-8655bc757-6g6cz\" (UID: \"42619d7f-d360-483b-9f9c-6e08b7d9b610\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6g6cz"
Sep 16 16:09:45.359592 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:45.359541    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42619d7f-d360-483b-9f9c-6e08b7d9b610-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-6g6cz\" (UID: \"42619d7f-d360-483b-9f9c-6e08b7d9b610\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6g6cz"
Sep 16 16:09:45.359653 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:45.359619    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42619d7f-d360-483b-9f9c-6e08b7d9b610-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-6g6cz\" (UID: \"42619d7f-d360-483b-9f9c-6e08b7d9b610\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6g6cz"
Sep 16 16:09:45.359715 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:45.359692    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42619d7f-d360-483b-9f9c-6e08b7d9b610-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-6g6cz\" (UID: \"42619d7f-d360-483b-9f9c-6e08b7d9b610\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6g6cz"
Sep 16 16:09:45.361472 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:45.361449    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42619d7f-d360-483b-9f9c-6e08b7d9b610-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-6g6cz\" (UID: \"42619d7f-d360-483b-9f9c-6e08b7d9b610\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6g6cz"
Sep 16 16:09:45.361874 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:45.361853    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42619d7f-d360-483b-9f9c-6e08b7d9b610-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-6g6cz\" (UID: \"42619d7f-d360-483b-9f9c-6e08b7d9b610\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6g6cz"
Sep 16 16:09:45.367200 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:45.367172    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h55gt\" (UniqueName: \"kubernetes.io/projected/42619d7f-d360-483b-9f9c-6e08b7d9b610-kube-api-access-h55gt\") pod \"scheduler-inline-config-test-kserve-8655bc757-6g6cz\" (UID: \"42619d7f-d360-483b-9f9c-6e08b7d9b610\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6g6cz"
Sep 16 16:09:45.387584 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:45.387547    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6g6cz"
Sep 16 16:09:45.664968 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:45.664893    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6g6cz"]
Sep 16 16:09:45.667959 ip-10-0-136-228 kubenswrapper[2554]: W0916 16:09:45.667931    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42619d7f_d360_483b_9f9c_6e08b7d9b610.slice/crio-19c08e8470704307b5148313ce8d92e90aad729dda7bf57cbd630be51aa372cf WatchSource:0}: Error finding container 19c08e8470704307b5148313ce8d92e90aad729dda7bf57cbd630be51aa372cf: Status 404 returned error can't find the container with id 19c08e8470704307b5148313ce8d92e90aad729dda7bf57cbd630be51aa372cf
Sep 16 16:09:46.315192 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:46.315155    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6g6cz" event={"ID":"42619d7f-d360-483b-9f9c-6e08b7d9b610","Type":"ContainerStarted","Data":"83272bb2a4082e248e9faa6757ea89aba7bd629830af2add1ba78cf45aec60ee"}
Sep 16 16:09:46.315646 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:46.315213    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6g6cz" event={"ID":"42619d7f-d360-483b-9f9c-6e08b7d9b610","Type":"ContainerStarted","Data":"19c08e8470704307b5148313ce8d92e90aad729dda7bf57cbd630be51aa372cf"}
Sep 16 16:09:46.335119 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:46.335066    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6g6cz" podStartSLOduration=1.335049394 podStartE2EDuration="1.335049394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:09:46.333633735 +0000 UTC m=+1186.778325933" watchObservedRunningTime="2026-09-16 16:09:46.335049394 +0000 UTC m=+1186.779741571"
Sep 16 16:09:46.388617 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:46.388587    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6g6cz"
Sep 16 16:09:49.860941 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:49.860909    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:09:49.962220 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:49.962186    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:09:55.388058 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:55.388020    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6g6cz"
Sep 16 16:09:55.400351 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:55.400319    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6g6cz"
Sep 16 16:09:55.758014 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:55.757936    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:09:56.370362 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:56.370333    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6g6cz"
Sep 16 16:09:58.360519 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:58.360488    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:09:58.464021 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:58.463989    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:09:58.560985 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:58.560948    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:09:58.975172 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:58.975095    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:09:59.076044 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:09:59.076014    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:10:01.990947 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:01.990916    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:10:05.068314 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:05.068285    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:10:05.369854 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:05.369771    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:10:07.308489 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:07.308456    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:10:07.407346 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:07.407309    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:10:07.509671 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:07.509641    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:10:07.712701 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:07.712669    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:10:07.966292 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:07.966207    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:10:08.099628 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:08.099591    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:10:10.680319 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:10.680289    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:10:11.064485 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:11.064454    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:10:11.164604 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:11.164575    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:10:11.666745 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:11.666715    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:10:11.767044 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:11.767015    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:10:12.402201 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:12.402167    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:10:17.431990 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:17.431952    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:10:18.998972 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:18.998940    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:10:23.823895 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:23.823862    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:10:24.796090 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:24.796054    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:10:24.893958 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:24.893930    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:10:33.443598 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:33.443564    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:10:35.102071 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:35.102037    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:10:37.324362 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:37.324336    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:10:37.423447 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:37.423413    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:10:39.291896 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:39.291867    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:10:42.402995 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:42.402956    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:10:47.820585 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:47.820552    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:10:49.090038 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:49.090005    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:10:50.781959 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:50.781922    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:10:52.531138 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:52.531046    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:10:52.806404 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:52.806364    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:10:55.406709 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:55.406662    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:10:55.424199 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:55.424168    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6g6cz"]
Sep 16 16:10:55.432815 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:10:55.432766    2554 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 16:10:55.432815 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:10:55.432815 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 16:10:55.432815 ip-10-0-136-228 kubenswrapper[2554]:  > containerID="83272bb2a4082e248e9faa6757ea89aba7bd629830af2add1ba78cf45aec60ee" cmd=["/bin/sleep","15"]
Sep 16 16:10:55.433120 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:55.432833    2554 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6g6cz" message="" err=<
Sep 16 16:10:55.433120 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:10:55.433120 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 16:10:55.433120 ip-10-0-136-228 kubenswrapper[2554]:  >
Sep 16 16:10:55.433120 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:10:55.432846    2554 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 16:10:55.433120 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:10:55.433120 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 16:10:55.433120 ip-10-0-136-228 kubenswrapper[2554]:  > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6g6cz" podUID="42619d7f-d360-483b-9f9c-6e08b7d9b610" containerName="main" containerID="cri-o://83272bb2a4082e248e9faa6757ea89aba7bd629830af2add1ba78cf45aec60ee"
Sep 16 16:10:55.433120 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:55.432910    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6g6cz" podUID="42619d7f-d360-483b-9f9c-6e08b7d9b610" containerName="main" containerID="cri-o://83272bb2a4082e248e9faa6757ea89aba7bd629830af2add1ba78cf45aec60ee" gracePeriod=60
Sep 16 16:10:55.479419 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:55.479390    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:10:55.580744 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:55.580715    2554 generic.go:356] "Generic (PLEG): container finished" podID="42619d7f-d360-483b-9f9c-6e08b7d9b610" containerID="83272bb2a4082e248e9faa6757ea89aba7bd629830af2add1ba78cf45aec60ee" exitCode=0
Sep 16 16:10:55.580889 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:55.580766    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6g6cz" event={"ID":"42619d7f-d360-483b-9f9c-6e08b7d9b610","Type":"ContainerDied","Data":"83272bb2a4082e248e9faa6757ea89aba7bd629830af2add1ba78cf45aec60ee"}
Sep 16 16:10:56.172592 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:56.172569    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6g6cz"
Sep 16 16:10:56.301918 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:56.301830    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42619d7f-d360-483b-9f9c-6e08b7d9b610-home\") pod \"42619d7f-d360-483b-9f9c-6e08b7d9b610\" (UID: \"42619d7f-d360-483b-9f9c-6e08b7d9b610\") "
Sep 16 16:10:56.301918 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:56.301891    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42619d7f-d360-483b-9f9c-6e08b7d9b610-tmp-dir\") pod \"42619d7f-d360-483b-9f9c-6e08b7d9b610\" (UID: \"42619d7f-d360-483b-9f9c-6e08b7d9b610\") "
Sep 16 16:10:56.301918 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:56.301910    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42619d7f-d360-483b-9f9c-6e08b7d9b610-dshm\") pod \"42619d7f-d360-483b-9f9c-6e08b7d9b610\" (UID: \"42619d7f-d360-483b-9f9c-6e08b7d9b610\") "
Sep 16 16:10:56.302203 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:56.301933    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42619d7f-d360-483b-9f9c-6e08b7d9b610-tls-certs\") pod \"42619d7f-d360-483b-9f9c-6e08b7d9b610\" (UID: \"42619d7f-d360-483b-9f9c-6e08b7d9b610\") "
Sep 16 16:10:56.302203 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:56.301978    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h55gt\" (UniqueName: \"kubernetes.io/projected/42619d7f-d360-483b-9f9c-6e08b7d9b610-kube-api-access-h55gt\") pod \"42619d7f-d360-483b-9f9c-6e08b7d9b610\" (UID: \"42619d7f-d360-483b-9f9c-6e08b7d9b610\") "
Sep 16 16:10:56.302203 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:56.301999    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42619d7f-d360-483b-9f9c-6e08b7d9b610-model-cache\") pod \"42619d7f-d360-483b-9f9c-6e08b7d9b610\" (UID: \"42619d7f-d360-483b-9f9c-6e08b7d9b610\") "
Sep 16 16:10:56.302203 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:56.302042    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42619d7f-d360-483b-9f9c-6e08b7d9b610-home" (OuterVolumeSpecName: "home") pod "42619d7f-d360-483b-9f9c-6e08b7d9b610" (UID: "42619d7f-d360-483b-9f9c-6e08b7d9b610"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:10:56.302203 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:56.302170    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42619d7f-d360-483b-9f9c-6e08b7d9b610-home\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:10:56.302407 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:56.302191    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42619d7f-d360-483b-9f9c-6e08b7d9b610-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "42619d7f-d360-483b-9f9c-6e08b7d9b610" (UID: "42619d7f-d360-483b-9f9c-6e08b7d9b610"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:10:56.302407 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:56.302244    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42619d7f-d360-483b-9f9c-6e08b7d9b610-model-cache" (OuterVolumeSpecName: "model-cache") pod "42619d7f-d360-483b-9f9c-6e08b7d9b610" (UID: "42619d7f-d360-483b-9f9c-6e08b7d9b610"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:10:56.304208 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:56.304177    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42619d7f-d360-483b-9f9c-6e08b7d9b610-dshm" (OuterVolumeSpecName: "dshm") pod "42619d7f-d360-483b-9f9c-6e08b7d9b610" (UID: "42619d7f-d360-483b-9f9c-6e08b7d9b610"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:10:56.304208 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:56.304202    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42619d7f-d360-483b-9f9c-6e08b7d9b610-kube-api-access-h55gt" (OuterVolumeSpecName: "kube-api-access-h55gt") pod "42619d7f-d360-483b-9f9c-6e08b7d9b610" (UID: "42619d7f-d360-483b-9f9c-6e08b7d9b610"). InnerVolumeSpecName "kube-api-access-h55gt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:10:56.304358 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:56.304244    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42619d7f-d360-483b-9f9c-6e08b7d9b610-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "42619d7f-d360-483b-9f9c-6e08b7d9b610" (UID: "42619d7f-d360-483b-9f9c-6e08b7d9b610"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:10:56.403042 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:56.402988    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42619d7f-d360-483b-9f9c-6e08b7d9b610-tmp-dir\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:10:56.403042 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:56.403036    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42619d7f-d360-483b-9f9c-6e08b7d9b610-dshm\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:10:56.403042 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:56.403049    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42619d7f-d360-483b-9f9c-6e08b7d9b610-tls-certs\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:10:56.403285 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:56.403061    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h55gt\" (UniqueName: \"kubernetes.io/projected/42619d7f-d360-483b-9f9c-6e08b7d9b610-kube-api-access-h55gt\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:10:56.403285 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:56.403075    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42619d7f-d360-483b-9f9c-6e08b7d9b610-model-cache\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:10:56.586157 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:56.586071    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6g6cz" event={"ID":"42619d7f-d360-483b-9f9c-6e08b7d9b610","Type":"ContainerDied","Data":"19c08e8470704307b5148313ce8d92e90aad729dda7bf57cbd630be51aa372cf"}
Sep 16 16:10:56.586157 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:56.586094    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6g6cz"
Sep 16 16:10:56.586157 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:56.586128    2554 scope.go:117] "RemoveContainer" containerID="83272bb2a4082e248e9faa6757ea89aba7bd629830af2add1ba78cf45aec60ee"
Sep 16 16:10:56.606511 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:56.606486    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6g6cz"]
Sep 16 16:10:56.610192 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:56.610170    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6g6cz"]
Sep 16 16:10:58.076185 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:58.076150    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42619d7f-d360-483b-9f9c-6e08b7d9b610" path="/var/lib/kubelet/pods/42619d7f-d360-483b-9f9c-6e08b7d9b610/volumes"
Sep 16 16:10:59.248137 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:10:59.248101    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:11:00.988279 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:11:00.988243    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:11:04.991365 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:11:04.991326    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:11:05.061621 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:11:05.061588    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:11:44.542479 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:11:44.542443    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:11:44.639502 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:11:44.639464    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:11:51.800840 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:11:51.800812    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:11:57.757124 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:11:57.757089    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:11:57.836010 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:11:57.835958    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:11:59.774610 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:11:59.774568    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q"]
Sep 16 16:11:59.775106 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:11:59.774970    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42619d7f-d360-483b-9f9c-6e08b7d9b610" containerName="main"
Sep 16 16:11:59.775106 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:11:59.774987    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="42619d7f-d360-483b-9f9c-6e08b7d9b610" containerName="main"
Sep 16 16:11:59.775106 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:11:59.775055    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="42619d7f-d360-483b-9f9c-6e08b7d9b610" containerName="main"
Sep 16 16:11:59.777845 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:11:59.777823    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q"
Sep 16 16:11:59.780627 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:11:59.780604    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 16 16:11:59.781960 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:11:59.781943    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 16 16:11:59.781960 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:11:59.781952    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 16 16:11:59.782098 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:11:59.782020    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-xvt2d\""
Sep 16 16:11:59.788941 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:11:59.788919    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q"]
Sep 16 16:11:59.867755 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:11:59.867718    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33bdd9a5-861b-46e2-866f-7374625f911b-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q\" (UID: \"33bdd9a5-861b-46e2-866f-7374625f911b\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q"
Sep 16 16:11:59.867755 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:11:59.867759    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrkl5\" (UniqueName: \"kubernetes.io/projected/33bdd9a5-861b-46e2-866f-7374625f911b-kube-api-access-hrkl5\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q\" (UID: \"33bdd9a5-861b-46e2-866f-7374625f911b\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q"
Sep 16 16:11:59.867974 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:11:59.867841    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33bdd9a5-861b-46e2-866f-7374625f911b-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q\" (UID: \"33bdd9a5-861b-46e2-866f-7374625f911b\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q"
Sep 16 16:11:59.867974 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:11:59.867887    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33bdd9a5-861b-46e2-866f-7374625f911b-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q\" (UID: \"33bdd9a5-861b-46e2-866f-7374625f911b\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q"
Sep 16 16:11:59.867974 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:11:59.867920    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33bdd9a5-861b-46e2-866f-7374625f911b-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q\" (UID: \"33bdd9a5-861b-46e2-866f-7374625f911b\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q"
Sep 16 16:11:59.867974 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:11:59.867937    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33bdd9a5-861b-46e2-866f-7374625f911b-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q\" (UID: \"33bdd9a5-861b-46e2-866f-7374625f911b\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q"
Sep 16 16:11:59.882396 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:11:59.882340    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:11:59.968726 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:11:59.968697    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33bdd9a5-861b-46e2-866f-7374625f911b-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q\" (UID: \"33bdd9a5-861b-46e2-866f-7374625f911b\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q"
Sep 16 16:11:59.968891 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:11:59.968738    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33bdd9a5-861b-46e2-866f-7374625f911b-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q\" (UID: \"33bdd9a5-861b-46e2-866f-7374625f911b\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q"
Sep 16 16:11:59.968891 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:11:59.968763    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33bdd9a5-861b-46e2-866f-7374625f911b-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q\" (UID: \"33bdd9a5-861b-46e2-866f-7374625f911b\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q"
Sep 16 16:11:59.968891 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:11:59.968781    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33bdd9a5-861b-46e2-866f-7374625f911b-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q\" (UID: \"33bdd9a5-861b-46e2-866f-7374625f911b\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q"
Sep 16 16:11:59.969004 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:11:59.968955    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33bdd9a5-861b-46e2-866f-7374625f911b-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q\" (UID: \"33bdd9a5-861b-46e2-866f-7374625f911b\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q"
Sep 16 16:11:59.969114 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:11:59.969002    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrkl5\" (UniqueName: \"kubernetes.io/projected/33bdd9a5-861b-46e2-866f-7374625f911b-kube-api-access-hrkl5\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q\" (UID: \"33bdd9a5-861b-46e2-866f-7374625f911b\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q"
Sep 16 16:11:59.969192 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:11:59.969175    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33bdd9a5-861b-46e2-866f-7374625f911b-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q\" (UID: \"33bdd9a5-861b-46e2-866f-7374625f911b\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q"
Sep 16 16:11:59.969245 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:11:59.969216    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33bdd9a5-861b-46e2-866f-7374625f911b-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q\" (UID: \"33bdd9a5-861b-46e2-866f-7374625f911b\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q"
Sep 16 16:11:59.969302 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:11:59.969241    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33bdd9a5-861b-46e2-866f-7374625f911b-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q\" (UID: \"33bdd9a5-861b-46e2-866f-7374625f911b\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q"
Sep 16 16:11:59.971122 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:11:59.971102    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33bdd9a5-861b-46e2-866f-7374625f911b-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q\" (UID: \"33bdd9a5-861b-46e2-866f-7374625f911b\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q"
Sep 16 16:11:59.971705 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:11:59.971687    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 16 16:11:59.977570 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:11:59.977552    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 16 16:11:59.981626 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:11:59.981606    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33bdd9a5-861b-46e2-866f-7374625f911b-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q\" (UID: \"33bdd9a5-861b-46e2-866f-7374625f911b\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q"
Sep 16 16:11:59.988023 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:11:59.988005    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 16 16:11:59.997618 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:11:59.997599    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrkl5\" (UniqueName: \"kubernetes.io/projected/33bdd9a5-861b-46e2-866f-7374625f911b-kube-api-access-hrkl5\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q\" (UID: \"33bdd9a5-861b-46e2-866f-7374625f911b\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q"
Sep 16 16:12:00.067323 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:00.067289    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:12:00.090322 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:00.090296    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-xvt2d\""
Sep 16 16:12:00.098766 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:00.098738    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q"
Sep 16 16:12:00.369397 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:00.369373    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q"]
Sep 16 16:12:00.371341 ip-10-0-136-228 kubenswrapper[2554]: W0916 16:12:00.371318    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33bdd9a5_861b_46e2_866f_7374625f911b.slice/crio-3b8f85c89268a80b9d07f15e38ed5fd4e2cfcd8292d9c7e6b68cd7d6cdb55c57 WatchSource:0}: Error finding container 3b8f85c89268a80b9d07f15e38ed5fd4e2cfcd8292d9c7e6b68cd7d6cdb55c57: Status 404 returned error can't find the container with id 3b8f85c89268a80b9d07f15e38ed5fd4e2cfcd8292d9c7e6b68cd7d6cdb55c57
Sep 16 16:12:00.822899 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:00.822864    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q" event={"ID":"33bdd9a5-861b-46e2-866f-7374625f911b","Type":"ContainerStarted","Data":"0d5d689ce3fb75cb18d9d06b9eb6a8aa9b8cc4739114df464d13a5b49e243358"}
Sep 16 16:12:00.823283 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:00.822913    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q" event={"ID":"33bdd9a5-861b-46e2-866f-7374625f911b","Type":"ContainerStarted","Data":"3b8f85c89268a80b9d07f15e38ed5fd4e2cfcd8292d9c7e6b68cd7d6cdb55c57"}
Sep 16 16:12:00.840769 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:00.840721    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q" podStartSLOduration=1.840703711 podStartE2EDuration="1.840703711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:11:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:12:00.840497944 +0000 UTC m=+1321.285190151" watchObservedRunningTime="2026-09-16 16:12:00.840703711 +0000 UTC m=+1321.285395890"
Sep 16 16:12:01.099950 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:01.099863    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q"
Sep 16 16:12:09.509904 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:09.509859    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:12:10.099934 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:10.099884    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q"
Sep 16 16:12:10.112468 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:10.112434    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q"
Sep 16 16:12:10.872649 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:10.872616    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q"
Sep 16 16:12:34.131023 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:34.130945    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:12:34.139739 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:34.139711    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q"]
Sep 16 16:12:34.148817 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:12:34.148763    2554 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 16:12:34.148817 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:12:34.148817 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 16:12:34.148817 ip-10-0-136-228 kubenswrapper[2554]:  > containerID="0d5d689ce3fb75cb18d9d06b9eb6a8aa9b8cc4739114df464d13a5b49e243358" cmd=["/bin/sleep","15"]
Sep 16 16:12:34.149121 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:34.148849    2554 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q" message="" err=<
Sep 16 16:12:34.149121 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:12:34.149121 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 16:12:34.149121 ip-10-0-136-228 kubenswrapper[2554]:  >
Sep 16 16:12:34.149121 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:12:34.148869    2554 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 16:12:34.149121 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:12:34.149121 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 16:12:34.149121 ip-10-0-136-228 kubenswrapper[2554]:  > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q" podUID="33bdd9a5-861b-46e2-866f-7374625f911b" containerName="main" containerID="cri-o://0d5d689ce3fb75cb18d9d06b9eb6a8aa9b8cc4739114df464d13a5b49e243358"
Sep 16 16:12:34.149121 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:34.148917    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q" podUID="33bdd9a5-861b-46e2-866f-7374625f911b" containerName="main" containerID="cri-o://0d5d689ce3fb75cb18d9d06b9eb6a8aa9b8cc4739114df464d13a5b49e243358" gracePeriod=60
Sep 16 16:12:34.260040 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:34.260011    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:12:34.487965 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:34.487939    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q"
Sep 16 16:12:34.529019 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:34.528988    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33bdd9a5-861b-46e2-866f-7374625f911b-model-cache\") pod \"33bdd9a5-861b-46e2-866f-7374625f911b\" (UID: \"33bdd9a5-861b-46e2-866f-7374625f911b\") "
Sep 16 16:12:34.529019 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:34.529022    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33bdd9a5-861b-46e2-866f-7374625f911b-tls-certs\") pod \"33bdd9a5-861b-46e2-866f-7374625f911b\" (UID: \"33bdd9a5-861b-46e2-866f-7374625f911b\") "
Sep 16 16:12:34.529233 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:34.529050    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33bdd9a5-861b-46e2-866f-7374625f911b-dshm\") pod \"33bdd9a5-861b-46e2-866f-7374625f911b\" (UID: \"33bdd9a5-861b-46e2-866f-7374625f911b\") "
Sep 16 16:12:34.529233 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:34.529072    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33bdd9a5-861b-46e2-866f-7374625f911b-home\") pod \"33bdd9a5-861b-46e2-866f-7374625f911b\" (UID: \"33bdd9a5-861b-46e2-866f-7374625f911b\") "
Sep 16 16:12:34.529233 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:34.529091    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33bdd9a5-861b-46e2-866f-7374625f911b-tmp-dir\") pod \"33bdd9a5-861b-46e2-866f-7374625f911b\" (UID: \"33bdd9a5-861b-46e2-866f-7374625f911b\") "
Sep 16 16:12:34.529233 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:34.529124    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrkl5\" (UniqueName: \"kubernetes.io/projected/33bdd9a5-861b-46e2-866f-7374625f911b-kube-api-access-hrkl5\") pod \"33bdd9a5-861b-46e2-866f-7374625f911b\" (UID: \"33bdd9a5-861b-46e2-866f-7374625f911b\") "
Sep 16 16:12:34.529441 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:34.529274    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33bdd9a5-861b-46e2-866f-7374625f911b-model-cache" (OuterVolumeSpecName: "model-cache") pod "33bdd9a5-861b-46e2-866f-7374625f911b" (UID: "33bdd9a5-861b-46e2-866f-7374625f911b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:12:34.529441 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:34.529346    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33bdd9a5-861b-46e2-866f-7374625f911b-home" (OuterVolumeSpecName: "home") pod "33bdd9a5-861b-46e2-866f-7374625f911b" (UID: "33bdd9a5-861b-46e2-866f-7374625f911b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:12:34.529441 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:34.529363    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33bdd9a5-861b-46e2-866f-7374625f911b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "33bdd9a5-861b-46e2-866f-7374625f911b" (UID: "33bdd9a5-861b-46e2-866f-7374625f911b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:12:34.531306 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:34.531282    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33bdd9a5-861b-46e2-866f-7374625f911b-dshm" (OuterVolumeSpecName: "dshm") pod "33bdd9a5-861b-46e2-866f-7374625f911b" (UID: "33bdd9a5-861b-46e2-866f-7374625f911b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:12:34.531431 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:34.531325    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33bdd9a5-861b-46e2-866f-7374625f911b-kube-api-access-hrkl5" (OuterVolumeSpecName: "kube-api-access-hrkl5") pod "33bdd9a5-861b-46e2-866f-7374625f911b" (UID: "33bdd9a5-861b-46e2-866f-7374625f911b"). InnerVolumeSpecName "kube-api-access-hrkl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:12:34.531431 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:34.531362    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33bdd9a5-861b-46e2-866f-7374625f911b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "33bdd9a5-861b-46e2-866f-7374625f911b" (UID: "33bdd9a5-861b-46e2-866f-7374625f911b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:12:34.630228 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:34.630188    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33bdd9a5-861b-46e2-866f-7374625f911b-dshm\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:12:34.630228 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:34.630215    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33bdd9a5-861b-46e2-866f-7374625f911b-home\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:12:34.630228 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:34.630224    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33bdd9a5-861b-46e2-866f-7374625f911b-tmp-dir\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:12:34.630228 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:34.630235    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrkl5\" (UniqueName: \"kubernetes.io/projected/33bdd9a5-861b-46e2-866f-7374625f911b-kube-api-access-hrkl5\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:12:34.630493 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:34.630244    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33bdd9a5-861b-46e2-866f-7374625f911b-model-cache\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:12:34.630493 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:34.630253    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33bdd9a5-861b-46e2-866f-7374625f911b-tls-certs\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:12:34.954584 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:34.954549    2554 generic.go:356] "Generic (PLEG): container finished" podID="33bdd9a5-861b-46e2-866f-7374625f911b" containerID="0d5d689ce3fb75cb18d9d06b9eb6a8aa9b8cc4739114df464d13a5b49e243358" exitCode=0
Sep 16 16:12:34.954813 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:34.954603    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q"
Sep 16 16:12:34.954813 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:34.954622    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q" event={"ID":"33bdd9a5-861b-46e2-866f-7374625f911b","Type":"ContainerDied","Data":"0d5d689ce3fb75cb18d9d06b9eb6a8aa9b8cc4739114df464d13a5b49e243358"}
Sep 16 16:12:34.954813 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:34.954665    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q" event={"ID":"33bdd9a5-861b-46e2-866f-7374625f911b","Type":"ContainerDied","Data":"3b8f85c89268a80b9d07f15e38ed5fd4e2cfcd8292d9c7e6b68cd7d6cdb55c57"}
Sep 16 16:12:34.954813 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:34.954687    2554 scope.go:117] "RemoveContainer" containerID="0d5d689ce3fb75cb18d9d06b9eb6a8aa9b8cc4739114df464d13a5b49e243358"
Sep 16 16:12:34.963578 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:34.963559    2554 scope.go:117] "RemoveContainer" containerID="0d5d689ce3fb75cb18d9d06b9eb6a8aa9b8cc4739114df464d13a5b49e243358"
Sep 16 16:12:34.963948 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:12:34.963930    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d5d689ce3fb75cb18d9d06b9eb6a8aa9b8cc4739114df464d13a5b49e243358\": container with ID starting with 0d5d689ce3fb75cb18d9d06b9eb6a8aa9b8cc4739114df464d13a5b49e243358 not found: ID does not exist" containerID="0d5d689ce3fb75cb18d9d06b9eb6a8aa9b8cc4739114df464d13a5b49e243358"
Sep 16 16:12:34.964007 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:34.963957    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d5d689ce3fb75cb18d9d06b9eb6a8aa9b8cc4739114df464d13a5b49e243358"} err="failed to get container status \"0d5d689ce3fb75cb18d9d06b9eb6a8aa9b8cc4739114df464d13a5b49e243358\": rpc error: code = NotFound desc = could not find container \"0d5d689ce3fb75cb18d9d06b9eb6a8aa9b8cc4739114df464d13a5b49e243358\": container with ID starting with 0d5d689ce3fb75cb18d9d06b9eb6a8aa9b8cc4739114df464d13a5b49e243358 not found: ID does not exist"
Sep 16 16:12:34.976469 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:34.976445    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q"]
Sep 16 16:12:34.982229 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:34.982207    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flb76q"]
Sep 16 16:12:36.076275 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:36.076242    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33bdd9a5-861b-46e2-866f-7374625f911b" path="/var/lib/kubelet/pods/33bdd9a5-861b-46e2-866f-7374625f911b/volumes"
Sep 16 16:12:45.805063 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:45.805030    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:12:57.715488 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:57.715454    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:12:57.790298 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:12:57.790263    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:13:23.883914 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:23.883847    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:13:27.052449 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:27.052409    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:13:30.145602 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:30.145571    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:13:36.773755 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:36.773722    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:13:36.841028 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:36.840973    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:13:39.386251 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:39.386219    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9"]
Sep 16 16:13:39.386617 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:39.386492    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33bdd9a5-861b-46e2-866f-7374625f911b" containerName="main"
Sep 16 16:13:39.386617 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:39.386504    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="33bdd9a5-861b-46e2-866f-7374625f911b" containerName="main"
Sep 16 16:13:39.386617 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:39.386556    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="33bdd9a5-861b-46e2-866f-7374625f911b" containerName="main"
Sep 16 16:13:39.389408 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:39.389393    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9"
Sep 16 16:13:39.392168 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:39.392145    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-4bdwq\""
Sep 16 16:13:39.393566 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:39.393547    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 16 16:13:39.393666 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:39.393581    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 16 16:13:39.393666 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:39.393582    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 16 16:13:39.401929 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:39.401906    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9"]
Sep 16 16:13:39.410405 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:39.410384    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:13:39.497003 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:39.496961    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33b900b1-e5b9-463a-b715-3625ccd42a5d-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9\" (UID: \"33b900b1-e5b9-463a-b715-3625ccd42a5d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9"
Sep 16 16:13:39.497003 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:39.497007    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33b900b1-e5b9-463a-b715-3625ccd42a5d-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9\" (UID: \"33b900b1-e5b9-463a-b715-3625ccd42a5d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9"
Sep 16 16:13:39.497207 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:39.497027    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33b900b1-e5b9-463a-b715-3625ccd42a5d-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9\" (UID: \"33b900b1-e5b9-463a-b715-3625ccd42a5d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9"
Sep 16 16:13:39.497207 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:39.497080    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhxwr\" (UniqueName: \"kubernetes.io/projected/33b900b1-e5b9-463a-b715-3625ccd42a5d-kube-api-access-fhxwr\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9\" (UID: \"33b900b1-e5b9-463a-b715-3625ccd42a5d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9"
Sep 16 16:13:39.497207 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:39.497136    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33b900b1-e5b9-463a-b715-3625ccd42a5d-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9\" (UID: \"33b900b1-e5b9-463a-b715-3625ccd42a5d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9"
Sep 16 16:13:39.497207 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:39.497189    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33b900b1-e5b9-463a-b715-3625ccd42a5d-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9\" (UID: \"33b900b1-e5b9-463a-b715-3625ccd42a5d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9"
Sep 16 16:13:39.597721 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:39.597680    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33b900b1-e5b9-463a-b715-3625ccd42a5d-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9\" (UID: \"33b900b1-e5b9-463a-b715-3625ccd42a5d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9"
Sep 16 16:13:39.598014 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:39.597730    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33b900b1-e5b9-463a-b715-3625ccd42a5d-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9\" (UID: \"33b900b1-e5b9-463a-b715-3625ccd42a5d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9"
Sep 16 16:13:39.598014 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:39.597764    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33b900b1-e5b9-463a-b715-3625ccd42a5d-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9\" (UID: \"33b900b1-e5b9-463a-b715-3625ccd42a5d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9"
Sep 16 16:13:39.598014 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:39.597787    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33b900b1-e5b9-463a-b715-3625ccd42a5d-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9\" (UID: \"33b900b1-e5b9-463a-b715-3625ccd42a5d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9"
Sep 16 16:13:39.598014 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:39.597832    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhxwr\" (UniqueName: \"kubernetes.io/projected/33b900b1-e5b9-463a-b715-3625ccd42a5d-kube-api-access-fhxwr\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9\" (UID: \"33b900b1-e5b9-463a-b715-3625ccd42a5d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9"
Sep 16 16:13:39.598014 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:39.597871    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33b900b1-e5b9-463a-b715-3625ccd42a5d-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9\" (UID: \"33b900b1-e5b9-463a-b715-3625ccd42a5d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9"
Sep 16 16:13:39.598237 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:39.598218    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33b900b1-e5b9-463a-b715-3625ccd42a5d-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9\" (UID: \"33b900b1-e5b9-463a-b715-3625ccd42a5d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9"
Sep 16 16:13:39.598311 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:39.598241    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33b900b1-e5b9-463a-b715-3625ccd42a5d-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9\" (UID: \"33b900b1-e5b9-463a-b715-3625ccd42a5d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9"
Sep 16 16:13:39.598311 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:39.598292    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33b900b1-e5b9-463a-b715-3625ccd42a5d-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9\" (UID: \"33b900b1-e5b9-463a-b715-3625ccd42a5d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9"
Sep 16 16:13:39.600080 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:39.600054    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33b900b1-e5b9-463a-b715-3625ccd42a5d-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9\" (UID: \"33b900b1-e5b9-463a-b715-3625ccd42a5d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9"
Sep 16 16:13:39.600444 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:39.600421    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33b900b1-e5b9-463a-b715-3625ccd42a5d-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9\" (UID: \"33b900b1-e5b9-463a-b715-3625ccd42a5d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9"
Sep 16 16:13:39.608336 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:39.608309    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhxwr\" (UniqueName: \"kubernetes.io/projected/33b900b1-e5b9-463a-b715-3625ccd42a5d-kube-api-access-fhxwr\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9\" (UID: \"33b900b1-e5b9-463a-b715-3625ccd42a5d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9"
Sep 16 16:13:39.677419 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:39.677329    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:13:39.699435 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:39.699400    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9"
Sep 16 16:13:39.970951 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:39.970872    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9"]
Sep 16 16:13:39.973553 ip-10-0-136-228 kubenswrapper[2554]: W0916 16:13:39.973530    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b900b1_e5b9_463a_b715_3625ccd42a5d.slice/crio-91ff1b242f6018228eb1d72963245897cd0326fa80b793366c346161ad34cce6 WatchSource:0}: Error finding container 91ff1b242f6018228eb1d72963245897cd0326fa80b793366c346161ad34cce6: Status 404 returned error can't find the container with id 91ff1b242f6018228eb1d72963245897cd0326fa80b793366c346161ad34cce6
Sep 16 16:13:40.197807 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:40.197758    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9" event={"ID":"33b900b1-e5b9-463a-b715-3625ccd42a5d","Type":"ContainerStarted","Data":"09efb5bb2ec4cdc3d2b706cbbfb66b033ee33e9d410facf317122b42e3afb234"}
Sep 16 16:13:40.197973 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:40.197825    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9" event={"ID":"33b900b1-e5b9-463a-b715-3625ccd42a5d","Type":"ContainerStarted","Data":"91ff1b242f6018228eb1d72963245897cd0326fa80b793366c346161ad34cce6"}
Sep 16 16:13:40.218191 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:40.218141    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9" podStartSLOduration=1.218125817 podStartE2EDuration="1.218125817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:13:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:13:40.216458075 +0000 UTC m=+1420.661150267" watchObservedRunningTime="2026-09-16 16:13:40.218125817 +0000 UTC m=+1420.662817994"
Sep 16 16:13:40.700128 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:40.700075    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9"
Sep 16 16:13:43.278121 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:43.278082    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:13:47.794446 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:47.794409    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:13:47.896432 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:47.896395    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:13:49.700234 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:49.700194    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9"
Sep 16 16:13:49.712645 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:49.712615    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9"
Sep 16 16:13:50.246170 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:50.246140    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9"
Sep 16 16:13:54.511616 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:54.511532    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:13:54.813340 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:54.813310    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:13:55.204528 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:13:55.204455    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:14:02.313433 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:14:02.313403    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:14:02.374107 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:14:02.374077    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:14:03.874613 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:14:03.874577    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:14:13.909426 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:14:13.909390    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9"]
Sep 16 16:14:13.920482 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:14:13.920441    2554 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 16:14:13.920482 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:14:13.920482 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 16:14:13.920482 ip-10-0-136-228 kubenswrapper[2554]:  > containerID="09efb5bb2ec4cdc3d2b706cbbfb66b033ee33e9d410facf317122b42e3afb234" cmd=["/bin/sleep","15"]
Sep 16 16:14:13.920819 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:14:13.920509    2554 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9" message="" err=<
Sep 16 16:14:13.920819 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:14:13.920819 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 16:14:13.920819 ip-10-0-136-228 kubenswrapper[2554]:  >
Sep 16 16:14:13.920819 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:14:13.920527    2554 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 16:14:13.920819 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:14:13.920819 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 16:14:13.920819 ip-10-0-136-228 kubenswrapper[2554]:  > pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9" podUID="33b900b1-e5b9-463a-b715-3625ccd42a5d" containerName="main" containerID="cri-o://09efb5bb2ec4cdc3d2b706cbbfb66b033ee33e9d410facf317122b42e3afb234"
Sep 16 16:14:13.920819 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:14:13.920576    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9" podUID="33b900b1-e5b9-463a-b715-3625ccd42a5d" containerName="main" containerID="cri-o://09efb5bb2ec4cdc3d2b706cbbfb66b033ee33e9d410facf317122b42e3afb234" gracePeriod=60
Sep 16 16:14:14.081978 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:14:14.081949    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:14:14.188406 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:14:14.188370    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:14:14.261220 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:14:14.261191    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9"
Sep 16 16:14:14.323686 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:14:14.323658    2554 generic.go:356] "Generic (PLEG): container finished" podID="33b900b1-e5b9-463a-b715-3625ccd42a5d" containerID="09efb5bb2ec4cdc3d2b706cbbfb66b033ee33e9d410facf317122b42e3afb234" exitCode=0
Sep 16 16:14:14.323908 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:14:14.323705    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9"
Sep 16 16:14:14.323908 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:14:14.323750    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9" event={"ID":"33b900b1-e5b9-463a-b715-3625ccd42a5d","Type":"ContainerDied","Data":"09efb5bb2ec4cdc3d2b706cbbfb66b033ee33e9d410facf317122b42e3afb234"}
Sep 16 16:14:14.323908 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:14:14.323861    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9" event={"ID":"33b900b1-e5b9-463a-b715-3625ccd42a5d","Type":"ContainerDied","Data":"91ff1b242f6018228eb1d72963245897cd0326fa80b793366c346161ad34cce6"}
Sep 16 16:14:14.323908 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:14:14.323892    2554 scope.go:117] "RemoveContainer" containerID="09efb5bb2ec4cdc3d2b706cbbfb66b033ee33e9d410facf317122b42e3afb234"
Sep 16 16:14:14.331806 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:14:14.331769    2554 scope.go:117] "RemoveContainer" containerID="09efb5bb2ec4cdc3d2b706cbbfb66b033ee33e9d410facf317122b42e3afb234"
Sep 16 16:14:14.332097 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:14:14.332078    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09efb5bb2ec4cdc3d2b706cbbfb66b033ee33e9d410facf317122b42e3afb234\": container with ID starting with 09efb5bb2ec4cdc3d2b706cbbfb66b033ee33e9d410facf317122b42e3afb234 not found: ID does not exist" containerID="09efb5bb2ec4cdc3d2b706cbbfb66b033ee33e9d410facf317122b42e3afb234"
Sep 16 16:14:14.332146 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:14:14.332106    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09efb5bb2ec4cdc3d2b706cbbfb66b033ee33e9d410facf317122b42e3afb234"} err="failed to get container status \"09efb5bb2ec4cdc3d2b706cbbfb66b033ee33e9d410facf317122b42e3afb234\": rpc error: code = NotFound desc = could not find container \"09efb5bb2ec4cdc3d2b706cbbfb66b033ee33e9d410facf317122b42e3afb234\": container with ID starting with 09efb5bb2ec4cdc3d2b706cbbfb66b033ee33e9d410facf317122b42e3afb234 not found: ID does not exist"
Sep 16 16:14:14.359433 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:14:14.359403    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33b900b1-e5b9-463a-b715-3625ccd42a5d-tls-certs\") pod \"33b900b1-e5b9-463a-b715-3625ccd42a5d\" (UID: \"33b900b1-e5b9-463a-b715-3625ccd42a5d\") "
Sep 16 16:14:14.359527 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:14:14.359459    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33b900b1-e5b9-463a-b715-3625ccd42a5d-model-cache\") pod \"33b900b1-e5b9-463a-b715-3625ccd42a5d\" (UID: \"33b900b1-e5b9-463a-b715-3625ccd42a5d\") "
Sep 16 16:14:14.359527 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:14:14.359485    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33b900b1-e5b9-463a-b715-3625ccd42a5d-dshm\") pod \"33b900b1-e5b9-463a-b715-3625ccd42a5d\" (UID: \"33b900b1-e5b9-463a-b715-3625ccd42a5d\") "
Sep 16 16:14:14.359527 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:14:14.359511    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33b900b1-e5b9-463a-b715-3625ccd42a5d-tmp-dir\") pod \"33b900b1-e5b9-463a-b715-3625ccd42a5d\" (UID: \"33b900b1-e5b9-463a-b715-3625ccd42a5d\") "
Sep 16 16:14:14.359667 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:14:14.359544    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhxwr\" (UniqueName: \"kubernetes.io/projected/33b900b1-e5b9-463a-b715-3625ccd42a5d-kube-api-access-fhxwr\") pod \"33b900b1-e5b9-463a-b715-3625ccd42a5d\" (UID: \"33b900b1-e5b9-463a-b715-3625ccd42a5d\") "
Sep 16 16:14:14.359667 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:14:14.359598    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33b900b1-e5b9-463a-b715-3625ccd42a5d-home\") pod \"33b900b1-e5b9-463a-b715-3625ccd42a5d\" (UID: \"33b900b1-e5b9-463a-b715-3625ccd42a5d\") "
Sep 16 16:14:14.359822 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:14:14.359763    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b900b1-e5b9-463a-b715-3625ccd42a5d-model-cache" (OuterVolumeSpecName: "model-cache") pod "33b900b1-e5b9-463a-b715-3625ccd42a5d" (UID: "33b900b1-e5b9-463a-b715-3625ccd42a5d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:14:14.359893 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:14:14.359823    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b900b1-e5b9-463a-b715-3625ccd42a5d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "33b900b1-e5b9-463a-b715-3625ccd42a5d" (UID: "33b900b1-e5b9-463a-b715-3625ccd42a5d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:14:14.359977 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:14:14.359957    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b900b1-e5b9-463a-b715-3625ccd42a5d-home" (OuterVolumeSpecName: "home") pod "33b900b1-e5b9-463a-b715-3625ccd42a5d" (UID: "33b900b1-e5b9-463a-b715-3625ccd42a5d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:14:14.361771 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:14:14.361751    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33b900b1-e5b9-463a-b715-3625ccd42a5d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "33b900b1-e5b9-463a-b715-3625ccd42a5d" (UID: "33b900b1-e5b9-463a-b715-3625ccd42a5d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:14:14.361843 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:14:14.361781    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b900b1-e5b9-463a-b715-3625ccd42a5d-dshm" (OuterVolumeSpecName: "dshm") pod "33b900b1-e5b9-463a-b715-3625ccd42a5d" (UID: "33b900b1-e5b9-463a-b715-3625ccd42a5d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:14:14.361887 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:14:14.361836    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b900b1-e5b9-463a-b715-3625ccd42a5d-kube-api-access-fhxwr" (OuterVolumeSpecName: "kube-api-access-fhxwr") pod "33b900b1-e5b9-463a-b715-3625ccd42a5d" (UID: "33b900b1-e5b9-463a-b715-3625ccd42a5d"). InnerVolumeSpecName "kube-api-access-fhxwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:14:14.460702 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:14:14.460616    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33b900b1-e5b9-463a-b715-3625ccd42a5d-home\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:14:14.460702 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:14:14.460645    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33b900b1-e5b9-463a-b715-3625ccd42a5d-tls-certs\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:14:14.460702 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:14:14.460661    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33b900b1-e5b9-463a-b715-3625ccd42a5d-model-cache\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:14:14.460702 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:14:14.460671    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33b900b1-e5b9-463a-b715-3625ccd42a5d-dshm\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:14:14.460702 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:14:14.460695    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33b900b1-e5b9-463a-b715-3625ccd42a5d-tmp-dir\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:14:14.460702 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:14:14.460704    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhxwr\" (UniqueName: \"kubernetes.io/projected/33b900b1-e5b9-463a-b715-3625ccd42a5d-kube-api-access-fhxwr\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:14:14.645537 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:14:14.645506    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9"]
Sep 16 16:14:14.650290 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:14:14.650267    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9"]
Sep 16 16:14:15.236311 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:14:15.236267    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-tvfl9" podUID="33b900b1-e5b9-463a-b715-3625ccd42a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.24:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 16 16:14:16.076206 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:14:16.076174    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33b900b1-e5b9-463a-b715-3625ccd42a5d" path="/var/lib/kubelet/pods/33b900b1-e5b9-463a-b715-3625ccd42a5d/volumes"
Sep 16 16:14:37.206386 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:14:37.206350    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:15:03.982670 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:03.982639    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:15:04.359720 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:04.359690    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:15:04.619294 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:04.619207    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:15:07.931363 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:07.931325    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5"]
Sep 16 16:15:07.931848 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:07.931577    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33b900b1-e5b9-463a-b715-3625ccd42a5d" containerName="main"
Sep 16 16:15:07.931848 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:07.931587    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="33b900b1-e5b9-463a-b715-3625ccd42a5d" containerName="main"
Sep 16 16:15:07.931848 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:07.931645    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="33b900b1-e5b9-463a-b715-3625ccd42a5d" containerName="main"
Sep 16 16:15:07.935992 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:07.935973    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5"
Sep 16 16:15:07.939348 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:07.939325    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-9xtwm\""
Sep 16 16:15:07.940738 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:07.940718    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 16 16:15:07.940853 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:07.940757    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 16 16:15:07.942444 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:07.942105    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 16 16:15:07.953868 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:07.953839    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5"]
Sep 16 16:15:07.999644 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:07.999615    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:15:08.056312 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:08.056273    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b36af79d-7289-4677-8476-50f63907ea6e-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5\" (UID: \"b36af79d-7289-4677-8476-50f63907ea6e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5"
Sep 16 16:15:08.056467 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:08.056379    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b36af79d-7289-4677-8476-50f63907ea6e-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5\" (UID: \"b36af79d-7289-4677-8476-50f63907ea6e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5"
Sep 16 16:15:08.056467 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:08.056407    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b36af79d-7289-4677-8476-50f63907ea6e-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5\" (UID: \"b36af79d-7289-4677-8476-50f63907ea6e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5"
Sep 16 16:15:08.056467 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:08.056449    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b36af79d-7289-4677-8476-50f63907ea6e-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5\" (UID: \"b36af79d-7289-4677-8476-50f63907ea6e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5"
Sep 16 16:15:08.056593 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:08.056489    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b36af79d-7289-4677-8476-50f63907ea6e-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5\" (UID: \"b36af79d-7289-4677-8476-50f63907ea6e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5"
Sep 16 16:15:08.056593 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:08.056515    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqk7w\" (UniqueName: \"kubernetes.io/projected/b36af79d-7289-4677-8476-50f63907ea6e-kube-api-access-cqk7w\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5\" (UID: \"b36af79d-7289-4677-8476-50f63907ea6e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5"
Sep 16 16:15:08.157719 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:08.157676    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b36af79d-7289-4677-8476-50f63907ea6e-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5\" (UID: \"b36af79d-7289-4677-8476-50f63907ea6e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5"
Sep 16 16:15:08.157719 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:08.157714    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b36af79d-7289-4677-8476-50f63907ea6e-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5\" (UID: \"b36af79d-7289-4677-8476-50f63907ea6e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5"
Sep 16 16:15:08.157995 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:08.157731    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b36af79d-7289-4677-8476-50f63907ea6e-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5\" (UID: \"b36af79d-7289-4677-8476-50f63907ea6e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5"
Sep 16 16:15:08.157995 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:08.157754    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b36af79d-7289-4677-8476-50f63907ea6e-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5\" (UID: \"b36af79d-7289-4677-8476-50f63907ea6e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5"
Sep 16 16:15:08.157995 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:08.157776    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqk7w\" (UniqueName: \"kubernetes.io/projected/b36af79d-7289-4677-8476-50f63907ea6e-kube-api-access-cqk7w\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5\" (UID: \"b36af79d-7289-4677-8476-50f63907ea6e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5"
Sep 16 16:15:08.157995 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:08.157942    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b36af79d-7289-4677-8476-50f63907ea6e-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5\" (UID: \"b36af79d-7289-4677-8476-50f63907ea6e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5"
Sep 16 16:15:08.158274 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:08.158247    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b36af79d-7289-4677-8476-50f63907ea6e-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5\" (UID: \"b36af79d-7289-4677-8476-50f63907ea6e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5"
Sep 16 16:15:08.158274 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:08.158266    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b36af79d-7289-4677-8476-50f63907ea6e-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5\" (UID: \"b36af79d-7289-4677-8476-50f63907ea6e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5"
Sep 16 16:15:08.158384 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:08.158371    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b36af79d-7289-4677-8476-50f63907ea6e-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5\" (UID: \"b36af79d-7289-4677-8476-50f63907ea6e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5"
Sep 16 16:15:08.160120 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:08.160100    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b36af79d-7289-4677-8476-50f63907ea6e-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5\" (UID: \"b36af79d-7289-4677-8476-50f63907ea6e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5"
Sep 16 16:15:08.160264 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:08.160245    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b36af79d-7289-4677-8476-50f63907ea6e-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5\" (UID: \"b36af79d-7289-4677-8476-50f63907ea6e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5"
Sep 16 16:15:08.171748 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:08.171719    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqk7w\" (UniqueName: \"kubernetes.io/projected/b36af79d-7289-4677-8476-50f63907ea6e-kube-api-access-cqk7w\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5\" (UID: \"b36af79d-7289-4677-8476-50f63907ea6e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5"
Sep 16 16:15:08.246435 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:08.246350    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5"
Sep 16 16:15:08.269369 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:08.269336    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:15:08.518566 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:08.518539    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5"]
Sep 16 16:15:08.520987 ip-10-0-136-228 kubenswrapper[2554]: W0916 16:15:08.520965    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb36af79d_7289_4677_8476_50f63907ea6e.slice/crio-2f64f8b42dcaae2ed59fa6e18b0b05300fe48f9afc599f046711b44c1fd4c143 WatchSource:0}: Error finding container 2f64f8b42dcaae2ed59fa6e18b0b05300fe48f9afc599f046711b44c1fd4c143: Status 404 returned error can't find the container with id 2f64f8b42dcaae2ed59fa6e18b0b05300fe48f9afc599f046711b44c1fd4c143
Sep 16 16:15:08.528781 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:08.528758    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5" event={"ID":"b36af79d-7289-4677-8476-50f63907ea6e","Type":"ContainerStarted","Data":"2f64f8b42dcaae2ed59fa6e18b0b05300fe48f9afc599f046711b44c1fd4c143"}
Sep 16 16:15:09.533625 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:09.533589    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5" event={"ID":"b36af79d-7289-4677-8476-50f63907ea6e","Type":"ContainerStarted","Data":"f6aca9c89cd31d395237e2f1abc983aea642f22646cb029cff48a11cbb8de015"}
Sep 16 16:15:09.552396 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:09.552346    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5" podStartSLOduration=2.552329408 podStartE2EDuration="2.552329408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:15:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:15:09.551267021 +0000 UTC m=+1509.995959199" watchObservedRunningTime="2026-09-16 16:15:09.552329408 +0000 UTC m=+1509.997021585"
Sep 16 16:15:10.247168 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:10.247128    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5"
Sep 16 16:15:11.151015 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:11.150977    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:15:16.836173 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:16.836136    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:15:16.906518 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:16.906491    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:15:18.246546 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:18.246513    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5"
Sep 16 16:15:18.259281 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:18.259246    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5"
Sep 16 16:15:18.579652 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:18.579617    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5"
Sep 16 16:15:29.284429 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:29.284397    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:15:41.450604 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:41.450569    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5"]
Sep 16 16:15:41.464682 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:15:41.464638    2554 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 16:15:41.464682 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:15:41.464682 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 16:15:41.464682 ip-10-0-136-228 kubenswrapper[2554]:  > containerID="f6aca9c89cd31d395237e2f1abc983aea642f22646cb029cff48a11cbb8de015" cmd=["/bin/sleep","15"]
Sep 16 16:15:41.465005 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:41.464698    2554 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5" message="" err=<
Sep 16 16:15:41.465005 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:15:41.465005 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 16:15:41.465005 ip-10-0-136-228 kubenswrapper[2554]:  >
Sep 16 16:15:41.465005 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:15:41.464711    2554 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 16:15:41.465005 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:15:41.465005 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 16:15:41.465005 ip-10-0-136-228 kubenswrapper[2554]:  > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5" podUID="b36af79d-7289-4677-8476-50f63907ea6e" containerName="main" containerID="cri-o://f6aca9c89cd31d395237e2f1abc983aea642f22646cb029cff48a11cbb8de015"
Sep 16 16:15:41.465005 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:41.464777    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5" podUID="b36af79d-7289-4677-8476-50f63907ea6e" containerName="main" containerID="cri-o://f6aca9c89cd31d395237e2f1abc983aea642f22646cb029cff48a11cbb8de015" gracePeriod=60
Sep 16 16:15:41.465474 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:41.465178    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:15:41.554547 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:41.554447    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:15:41.569631 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:41.569604    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5" podUID="b36af79d-7289-4677-8476-50f63907ea6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.25:8000/health\": dial tcp 10.132.0.25:8000: connect: connection refused"
Sep 16 16:15:41.799578 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:41.799556    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5"
Sep 16 16:15:41.905015 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:41.904985    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b36af79d-7289-4677-8476-50f63907ea6e-home\") pod \"b36af79d-7289-4677-8476-50f63907ea6e\" (UID: \"b36af79d-7289-4677-8476-50f63907ea6e\") "
Sep 16 16:15:41.905015 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:41.905016    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b36af79d-7289-4677-8476-50f63907ea6e-dshm\") pod \"b36af79d-7289-4677-8476-50f63907ea6e\" (UID: \"b36af79d-7289-4677-8476-50f63907ea6e\") "
Sep 16 16:15:41.905252 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:41.905051    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqk7w\" (UniqueName: \"kubernetes.io/projected/b36af79d-7289-4677-8476-50f63907ea6e-kube-api-access-cqk7w\") pod \"b36af79d-7289-4677-8476-50f63907ea6e\" (UID: \"b36af79d-7289-4677-8476-50f63907ea6e\") "
Sep 16 16:15:41.905252 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:41.905082    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b36af79d-7289-4677-8476-50f63907ea6e-tls-certs\") pod \"b36af79d-7289-4677-8476-50f63907ea6e\" (UID: \"b36af79d-7289-4677-8476-50f63907ea6e\") "
Sep 16 16:15:41.905252 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:41.905142    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b36af79d-7289-4677-8476-50f63907ea6e-model-cache\") pod \"b36af79d-7289-4677-8476-50f63907ea6e\" (UID: \"b36af79d-7289-4677-8476-50f63907ea6e\") "
Sep 16 16:15:41.905252 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:41.905165    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b36af79d-7289-4677-8476-50f63907ea6e-tmp-dir\") pod \"b36af79d-7289-4677-8476-50f63907ea6e\" (UID: \"b36af79d-7289-4677-8476-50f63907ea6e\") "
Sep 16 16:15:41.905448 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:41.905284    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b36af79d-7289-4677-8476-50f63907ea6e-home" (OuterVolumeSpecName: "home") pod "b36af79d-7289-4677-8476-50f63907ea6e" (UID: "b36af79d-7289-4677-8476-50f63907ea6e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:15:41.905448 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:41.905398    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b36af79d-7289-4677-8476-50f63907ea6e-home\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:15:41.905448 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:41.905399    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b36af79d-7289-4677-8476-50f63907ea6e-model-cache" (OuterVolumeSpecName: "model-cache") pod "b36af79d-7289-4677-8476-50f63907ea6e" (UID: "b36af79d-7289-4677-8476-50f63907ea6e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:15:41.905597 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:41.905459    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b36af79d-7289-4677-8476-50f63907ea6e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b36af79d-7289-4677-8476-50f63907ea6e" (UID: "b36af79d-7289-4677-8476-50f63907ea6e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:15:41.907272 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:41.907249    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b36af79d-7289-4677-8476-50f63907ea6e-dshm" (OuterVolumeSpecName: "dshm") pod "b36af79d-7289-4677-8476-50f63907ea6e" (UID: "b36af79d-7289-4677-8476-50f63907ea6e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:15:41.907272 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:41.907261    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b36af79d-7289-4677-8476-50f63907ea6e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b36af79d-7289-4677-8476-50f63907ea6e" (UID: "b36af79d-7289-4677-8476-50f63907ea6e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:15:41.907272 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:41.907248    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b36af79d-7289-4677-8476-50f63907ea6e-kube-api-access-cqk7w" (OuterVolumeSpecName: "kube-api-access-cqk7w") pod "b36af79d-7289-4677-8476-50f63907ea6e" (UID: "b36af79d-7289-4677-8476-50f63907ea6e"). InnerVolumeSpecName "kube-api-access-cqk7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:15:42.005980 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:42.005905    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b36af79d-7289-4677-8476-50f63907ea6e-tls-certs\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:15:42.005980 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:42.005935    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b36af79d-7289-4677-8476-50f63907ea6e-model-cache\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:15:42.005980 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:42.005944    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b36af79d-7289-4677-8476-50f63907ea6e-tmp-dir\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:15:42.005980 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:42.005953    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b36af79d-7289-4677-8476-50f63907ea6e-dshm\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:15:42.005980 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:42.005962    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqk7w\" (UniqueName: \"kubernetes.io/projected/b36af79d-7289-4677-8476-50f63907ea6e-kube-api-access-cqk7w\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:15:42.661077 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:42.661045    2554 generic.go:356] "Generic (PLEG): container finished" podID="b36af79d-7289-4677-8476-50f63907ea6e" containerID="f6aca9c89cd31d395237e2f1abc983aea642f22646cb029cff48a11cbb8de015" exitCode=0
Sep 16 16:15:42.661471 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:42.661086    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5"
Sep 16 16:15:42.661471 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:42.661120    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5" event={"ID":"b36af79d-7289-4677-8476-50f63907ea6e","Type":"ContainerDied","Data":"f6aca9c89cd31d395237e2f1abc983aea642f22646cb029cff48a11cbb8de015"}
Sep 16 16:15:42.661471 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:42.661161    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5" event={"ID":"b36af79d-7289-4677-8476-50f63907ea6e","Type":"ContainerDied","Data":"2f64f8b42dcaae2ed59fa6e18b0b05300fe48f9afc599f046711b44c1fd4c143"}
Sep 16 16:15:42.661471 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:42.661182    2554 scope.go:117] "RemoveContainer" containerID="f6aca9c89cd31d395237e2f1abc983aea642f22646cb029cff48a11cbb8de015"
Sep 16 16:15:42.669541 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:42.669521    2554 scope.go:117] "RemoveContainer" containerID="f6aca9c89cd31d395237e2f1abc983aea642f22646cb029cff48a11cbb8de015"
Sep 16 16:15:42.669838 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:15:42.669816    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6aca9c89cd31d395237e2f1abc983aea642f22646cb029cff48a11cbb8de015\": container with ID starting with f6aca9c89cd31d395237e2f1abc983aea642f22646cb029cff48a11cbb8de015 not found: ID does not exist" containerID="f6aca9c89cd31d395237e2f1abc983aea642f22646cb029cff48a11cbb8de015"
Sep 16 16:15:42.669894 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:42.669847    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6aca9c89cd31d395237e2f1abc983aea642f22646cb029cff48a11cbb8de015"} err="failed to get container status \"f6aca9c89cd31d395237e2f1abc983aea642f22646cb029cff48a11cbb8de015\": rpc error: code = NotFound desc = could not find container \"f6aca9c89cd31d395237e2f1abc983aea642f22646cb029cff48a11cbb8de015\": container with ID starting with f6aca9c89cd31d395237e2f1abc983aea642f22646cb029cff48a11cbb8de015 not found: ID does not exist"
Sep 16 16:15:42.677495 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:42.677475    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5"]
Sep 16 16:15:42.683243 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:42.683223    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-fjvs5"]
Sep 16 16:15:44.076409 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:44.076378    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b36af79d-7289-4677-8476-50f63907ea6e" path="/var/lib/kubelet/pods/b36af79d-7289-4677-8476-50f63907ea6e/volumes"
Sep 16 16:15:56.529967 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:56.529934    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:15:56.594121 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:15:56.594093    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:16:31.970222 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:31.970188    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:16:32.056150 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:32.056121    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:16:37.604553 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:37.604517    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:16:40.589424 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:40.589386    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:16:40.652025 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:40.651995    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:16:42.775648 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:42.775608    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vnbpn"]
Sep 16 16:16:42.776122 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:42.776048    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b36af79d-7289-4677-8476-50f63907ea6e" containerName="main"
Sep 16 16:16:42.776122 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:42.776068    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b36af79d-7289-4677-8476-50f63907ea6e" containerName="main"
Sep 16 16:16:42.776236 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:42.776149    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="b36af79d-7289-4677-8476-50f63907ea6e" containerName="main"
Sep 16 16:16:42.779152 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:42.779132    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vnbpn"
Sep 16 16:16:42.783565 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:42.783543    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 16 16:16:42.783676 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:42.783546    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-hd65n\""
Sep 16 16:16:42.783676 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:42.783551    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 16 16:16:42.783676 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:42.783631    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 16 16:16:42.787411 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:42.787389    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vnbpn"]
Sep 16 16:16:42.862544 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:42.862512    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:16:42.937065 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:42.937040    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4937e90d-852f-4235-9f98-cb191330dbe8-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vnbpn\" (UID: \"4937e90d-852f-4235-9f98-cb191330dbe8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vnbpn"
Sep 16 16:16:42.937209 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:42.937069    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4937e90d-852f-4235-9f98-cb191330dbe8-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vnbpn\" (UID: \"4937e90d-852f-4235-9f98-cb191330dbe8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vnbpn"
Sep 16 16:16:42.937209 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:42.937099    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7svb5\" (UniqueName: \"kubernetes.io/projected/4937e90d-852f-4235-9f98-cb191330dbe8-kube-api-access-7svb5\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vnbpn\" (UID: \"4937e90d-852f-4235-9f98-cb191330dbe8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vnbpn"
Sep 16 16:16:42.937209 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:42.937175    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4937e90d-852f-4235-9f98-cb191330dbe8-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vnbpn\" (UID: \"4937e90d-852f-4235-9f98-cb191330dbe8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vnbpn"
Sep 16 16:16:42.937209 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:42.937200    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4937e90d-852f-4235-9f98-cb191330dbe8-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vnbpn\" (UID: \"4937e90d-852f-4235-9f98-cb191330dbe8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vnbpn"
Sep 16 16:16:42.937339 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:42.937224    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4937e90d-852f-4235-9f98-cb191330dbe8-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vnbpn\" (UID: \"4937e90d-852f-4235-9f98-cb191330dbe8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vnbpn"
Sep 16 16:16:43.038232 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:43.038201    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4937e90d-852f-4235-9f98-cb191330dbe8-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vnbpn\" (UID: \"4937e90d-852f-4235-9f98-cb191330dbe8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vnbpn"
Sep 16 16:16:43.038372 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:43.038236    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4937e90d-852f-4235-9f98-cb191330dbe8-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vnbpn\" (UID: \"4937e90d-852f-4235-9f98-cb191330dbe8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vnbpn"
Sep 16 16:16:43.038372 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:43.038258    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4937e90d-852f-4235-9f98-cb191330dbe8-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vnbpn\" (UID: \"4937e90d-852f-4235-9f98-cb191330dbe8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vnbpn"
Sep 16 16:16:43.038488 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:43.038418    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4937e90d-852f-4235-9f98-cb191330dbe8-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vnbpn\" (UID: \"4937e90d-852f-4235-9f98-cb191330dbe8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vnbpn"
Sep 16 16:16:43.038488 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:43.038463    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4937e90d-852f-4235-9f98-cb191330dbe8-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vnbpn\" (UID: \"4937e90d-852f-4235-9f98-cb191330dbe8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vnbpn"
Sep 16 16:16:43.038591 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:43.038520    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7svb5\" (UniqueName: \"kubernetes.io/projected/4937e90d-852f-4235-9f98-cb191330dbe8-kube-api-access-7svb5\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vnbpn\" (UID: \"4937e90d-852f-4235-9f98-cb191330dbe8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vnbpn"
Sep 16 16:16:43.038591 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:43.038568    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4937e90d-852f-4235-9f98-cb191330dbe8-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vnbpn\" (UID: \"4937e90d-852f-4235-9f98-cb191330dbe8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vnbpn"
Sep 16 16:16:43.038688 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:43.038621    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4937e90d-852f-4235-9f98-cb191330dbe8-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vnbpn\" (UID: \"4937e90d-852f-4235-9f98-cb191330dbe8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vnbpn"
Sep 16 16:16:43.038738 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:43.038709    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4937e90d-852f-4235-9f98-cb191330dbe8-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vnbpn\" (UID: \"4937e90d-852f-4235-9f98-cb191330dbe8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vnbpn"
Sep 16 16:16:43.040667 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:43.040646    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4937e90d-852f-4235-9f98-cb191330dbe8-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vnbpn\" (UID: \"4937e90d-852f-4235-9f98-cb191330dbe8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vnbpn"
Sep 16 16:16:43.040769 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:43.040736    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4937e90d-852f-4235-9f98-cb191330dbe8-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vnbpn\" (UID: \"4937e90d-852f-4235-9f98-cb191330dbe8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vnbpn"
Sep 16 16:16:43.046951 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:43.046928    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7svb5\" (UniqueName: \"kubernetes.io/projected/4937e90d-852f-4235-9f98-cb191330dbe8-kube-api-access-7svb5\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vnbpn\" (UID: \"4937e90d-852f-4235-9f98-cb191330dbe8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vnbpn"
Sep 16 16:16:43.050572 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:43.050548    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:16:43.089844 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:43.089819    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vnbpn"
Sep 16 16:16:43.358441 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:43.358369    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vnbpn"]
Sep 16 16:16:43.361823 ip-10-0-136-228 kubenswrapper[2554]: W0916 16:16:43.361782    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4937e90d_852f_4235_9f98_cb191330dbe8.slice/crio-4669b5d685dccb90c6d872d880338d78781b4a70c2455a8ffd51a69f3af4e149 WatchSource:0}: Error finding container 4669b5d685dccb90c6d872d880338d78781b4a70c2455a8ffd51a69f3af4e149: Status 404 returned error can't find the container with id 4669b5d685dccb90c6d872d880338d78781b4a70c2455a8ffd51a69f3af4e149
Sep 16 16:16:43.887898 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:43.887860    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vnbpn" event={"ID":"4937e90d-852f-4235-9f98-cb191330dbe8","Type":"ContainerStarted","Data":"674d988bee1c7c39ac016a0de5531d65a30c13b8104de2a234a5df74981041b6"}
Sep 16 16:16:43.888271 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:43.887920    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vnbpn" event={"ID":"4937e90d-852f-4235-9f98-cb191330dbe8","Type":"ContainerStarted","Data":"4669b5d685dccb90c6d872d880338d78781b4a70c2455a8ffd51a69f3af4e149"}
Sep 16 16:16:43.907360 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:43.907316    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vnbpn" podStartSLOduration=1.907301029 podStartE2EDuration="1.907301029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:16:43.906135578 +0000 UTC m=+1604.350827758" watchObservedRunningTime="2026-09-16 16:16:43.907301029 +0000 UTC m=+1604.351993198"
Sep 16 16:16:44.090276 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:44.090246    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vnbpn"
Sep 16 16:16:44.539405 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:44.539370    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:16:44.689376 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:44.689345    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:16:46.615018 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:46.614982    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:16:46.711847 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:46.711811    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:16:53.055079 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:53.054986    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:16:53.090738 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:53.090703    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vnbpn"
Sep 16 16:16:53.102661 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:53.102639    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vnbpn"
Sep 16 16:16:53.934110 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:53.934084    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vnbpn"
Sep 16 16:16:58.930290 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:58.930254    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:16:59.120333 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:16:59.120295    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:17:09.543206 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:17:09.543174    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:17:18.779488 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:17:18.779450    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:17:19.052111 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:17:19.052051    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:17:31.585206 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:17:31.585162    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:17:33.102595 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:17:33.102558    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vnbpn"]
Sep 16 16:17:33.112876 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:17:33.112829    2554 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 16:17:33.112876 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:17:33.112876 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 16:17:33.112876 ip-10-0-136-228 kubenswrapper[2554]:  > containerID="674d988bee1c7c39ac016a0de5531d65a30c13b8104de2a234a5df74981041b6" cmd=["/bin/sleep","15"]
Sep 16 16:17:33.113206 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:17:33.112902    2554 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vnbpn" message="" err=<
Sep 16 16:17:33.113206 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:17:33.113206 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 16:17:33.113206 ip-10-0-136-228 kubenswrapper[2554]:  >
Sep 16 16:17:33.113206 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:17:33.112921    2554 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 16:17:33.113206 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:17:33.113206 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 16:17:33.113206 ip-10-0-136-228 kubenswrapper[2554]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vnbpn" podUID="4937e90d-852f-4235-9f98-cb191330dbe8" containerName="main" containerID="cri-o://674d988bee1c7c39ac016a0de5531d65a30c13b8104de2a234a5df74981041b6"
Sep 16 16:17:33.113206 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:17:33.112970    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vnbpn" podUID="4937e90d-852f-4235-9f98-cb191330dbe8" containerName="main" containerID="cri-o://674d988bee1c7c39ac016a0de5531d65a30c13b8104de2a234a5df74981041b6" gracePeriod=60
Sep 16 16:17:33.163833 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:17:33.163780    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:17:33.230207 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:17:33.230181    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:17:33.850693 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:17:33.850667    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vnbpn"
Sep 16 16:17:33.885379 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:17:33.885354    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7svb5\" (UniqueName: \"kubernetes.io/projected/4937e90d-852f-4235-9f98-cb191330dbe8-kube-api-access-7svb5\") pod \"4937e90d-852f-4235-9f98-cb191330dbe8\" (UID: \"4937e90d-852f-4235-9f98-cb191330dbe8\") "
Sep 16 16:17:33.885539 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:17:33.885388    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4937e90d-852f-4235-9f98-cb191330dbe8-model-cache\") pod \"4937e90d-852f-4235-9f98-cb191330dbe8\" (UID: \"4937e90d-852f-4235-9f98-cb191330dbe8\") "
Sep 16 16:17:33.885539 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:17:33.885425    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4937e90d-852f-4235-9f98-cb191330dbe8-dshm\") pod \"4937e90d-852f-4235-9f98-cb191330dbe8\" (UID: \"4937e90d-852f-4235-9f98-cb191330dbe8\") "
Sep 16 16:17:33.885539 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:17:33.885441    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4937e90d-852f-4235-9f98-cb191330dbe8-tmp-dir\") pod \"4937e90d-852f-4235-9f98-cb191330dbe8\" (UID: \"4937e90d-852f-4235-9f98-cb191330dbe8\") "
Sep 16 16:17:33.885539 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:17:33.885483    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4937e90d-852f-4235-9f98-cb191330dbe8-home\") pod \"4937e90d-852f-4235-9f98-cb191330dbe8\" (UID: \"4937e90d-852f-4235-9f98-cb191330dbe8\") "
Sep 16 16:17:33.885539 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:17:33.885499    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4937e90d-852f-4235-9f98-cb191330dbe8-tls-certs\") pod \"4937e90d-852f-4235-9f98-cb191330dbe8\" (UID: \"4937e90d-852f-4235-9f98-cb191330dbe8\") "
Sep 16 16:17:33.885822 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:17:33.885679    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4937e90d-852f-4235-9f98-cb191330dbe8-model-cache" (OuterVolumeSpecName: "model-cache") pod "4937e90d-852f-4235-9f98-cb191330dbe8" (UID: "4937e90d-852f-4235-9f98-cb191330dbe8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:17:33.885822 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:17:33.885711    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4937e90d-852f-4235-9f98-cb191330dbe8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4937e90d-852f-4235-9f98-cb191330dbe8" (UID: "4937e90d-852f-4235-9f98-cb191330dbe8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:17:33.885822 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:17:33.885756    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4937e90d-852f-4235-9f98-cb191330dbe8-home" (OuterVolumeSpecName: "home") pod "4937e90d-852f-4235-9f98-cb191330dbe8" (UID: "4937e90d-852f-4235-9f98-cb191330dbe8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:17:33.887789 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:17:33.887758    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4937e90d-852f-4235-9f98-cb191330dbe8-kube-api-access-7svb5" (OuterVolumeSpecName: "kube-api-access-7svb5") pod "4937e90d-852f-4235-9f98-cb191330dbe8" (UID: "4937e90d-852f-4235-9f98-cb191330dbe8"). InnerVolumeSpecName "kube-api-access-7svb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:17:33.887789 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:17:33.887758    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4937e90d-852f-4235-9f98-cb191330dbe8-dshm" (OuterVolumeSpecName: "dshm") pod "4937e90d-852f-4235-9f98-cb191330dbe8" (UID: "4937e90d-852f-4235-9f98-cb191330dbe8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:17:33.887789 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:17:33.887769    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4937e90d-852f-4235-9f98-cb191330dbe8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4937e90d-852f-4235-9f98-cb191330dbe8" (UID: "4937e90d-852f-4235-9f98-cb191330dbe8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:17:33.986230 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:17:33.986150    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4937e90d-852f-4235-9f98-cb191330dbe8-home\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:17:33.986230 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:17:33.986180    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4937e90d-852f-4235-9f98-cb191330dbe8-tls-certs\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:17:33.986230 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:17:33.986191    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7svb5\" (UniqueName: \"kubernetes.io/projected/4937e90d-852f-4235-9f98-cb191330dbe8-kube-api-access-7svb5\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:17:33.986230 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:17:33.986199    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4937e90d-852f-4235-9f98-cb191330dbe8-model-cache\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:17:33.986230 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:17:33.986211    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4937e90d-852f-4235-9f98-cb191330dbe8-dshm\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:17:33.986230 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:17:33.986221    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4937e90d-852f-4235-9f98-cb191330dbe8-tmp-dir\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:17:34.071535 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:17:34.071505    2554 generic.go:356] "Generic (PLEG): container finished" podID="4937e90d-852f-4235-9f98-cb191330dbe8" containerID="674d988bee1c7c39ac016a0de5531d65a30c13b8104de2a234a5df74981041b6" exitCode=0
Sep 16 16:17:34.071679 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:17:34.071613    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vnbpn"
Sep 16 16:17:34.075399 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:17:34.075373    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vnbpn" event={"ID":"4937e90d-852f-4235-9f98-cb191330dbe8","Type":"ContainerDied","Data":"674d988bee1c7c39ac016a0de5531d65a30c13b8104de2a234a5df74981041b6"}
Sep 16 16:17:34.075509 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:17:34.075415    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vnbpn" event={"ID":"4937e90d-852f-4235-9f98-cb191330dbe8","Type":"ContainerDied","Data":"4669b5d685dccb90c6d872d880338d78781b4a70c2455a8ffd51a69f3af4e149"}
Sep 16 16:17:34.075509 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:17:34.075447    2554 scope.go:117] "RemoveContainer" containerID="674d988bee1c7c39ac016a0de5531d65a30c13b8104de2a234a5df74981041b6"
Sep 16 16:17:34.084718 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:17:34.084700    2554 scope.go:117] "RemoveContainer" containerID="674d988bee1c7c39ac016a0de5531d65a30c13b8104de2a234a5df74981041b6"
Sep 16 16:17:34.085015 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:17:34.084994    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"674d988bee1c7c39ac016a0de5531d65a30c13b8104de2a234a5df74981041b6\": container with ID starting with 674d988bee1c7c39ac016a0de5531d65a30c13b8104de2a234a5df74981041b6 not found: ID does not exist" containerID="674d988bee1c7c39ac016a0de5531d65a30c13b8104de2a234a5df74981041b6"
Sep 16 16:17:34.085074 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:17:34.085022    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"674d988bee1c7c39ac016a0de5531d65a30c13b8104de2a234a5df74981041b6"} err="failed to get container status \"674d988bee1c7c39ac016a0de5531d65a30c13b8104de2a234a5df74981041b6\": rpc error: code = NotFound desc = could not find container \"674d988bee1c7c39ac016a0de5531d65a30c13b8104de2a234a5df74981041b6\": container with ID starting with 674d988bee1c7c39ac016a0de5531d65a30c13b8104de2a234a5df74981041b6 not found: ID does not exist"
Sep 16 16:17:34.095935 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:17:34.095912    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vnbpn"]
Sep 16 16:17:34.099254 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:17:34.099228    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vnbpn"]
Sep 16 16:17:36.076122 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:17:36.076092    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4937e90d-852f-4235-9f98-cb191330dbe8" path="/var/lib/kubelet/pods/4937e90d-852f-4235-9f98-cb191330dbe8/volumes"
Sep 16 16:18:22.995961 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:22.995877    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:18:23.091904 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:23.091860    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:18:25.744031 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:25.743999    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx"]
Sep 16 16:18:25.744489 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:25.744353    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4937e90d-852f-4235-9f98-cb191330dbe8" containerName="main"
Sep 16 16:18:25.744489 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:25.744371    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4937e90d-852f-4235-9f98-cb191330dbe8" containerName="main"
Sep 16 16:18:25.744489 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:25.744454    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4937e90d-852f-4235-9f98-cb191330dbe8" containerName="main"
Sep 16 16:18:25.747222 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:25.747200    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx"
Sep 16 16:18:25.751061 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:25.751037    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 16 16:18:25.751061 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:25.751052    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-z7hqw\""
Sep 16 16:18:25.752085 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:25.752066    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 16 16:18:25.752177 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:25.752126    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-t7cvg\""
Sep 16 16:18:25.752466 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:25.752446    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 16 16:18:25.773321 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:25.773268    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx"]
Sep 16 16:18:25.842326 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:25.842291    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:18:25.868104 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:25.868064    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsgxk\" (UniqueName: \"kubernetes.io/projected/9cdf926a-9d37-4bf6-a613-ce5cd6541cd9-kube-api-access-tsgxk\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx\" (UID: \"9cdf926a-9d37-4bf6-a613-ce5cd6541cd9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx"
Sep 16 16:18:25.868285 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:25.868113    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cdf926a-9d37-4bf6-a613-ce5cd6541cd9-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx\" (UID: \"9cdf926a-9d37-4bf6-a613-ce5cd6541cd9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx"
Sep 16 16:18:25.868285 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:25.868144    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cdf926a-9d37-4bf6-a613-ce5cd6541cd9-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx\" (UID: \"9cdf926a-9d37-4bf6-a613-ce5cd6541cd9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx"
Sep 16 16:18:25.868285 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:25.868169    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cdf926a-9d37-4bf6-a613-ce5cd6541cd9-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx\" (UID: \"9cdf926a-9d37-4bf6-a613-ce5cd6541cd9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx"
Sep 16 16:18:25.868285 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:25.868211    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cdf926a-9d37-4bf6-a613-ce5cd6541cd9-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx\" (UID: \"9cdf926a-9d37-4bf6-a613-ce5cd6541cd9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx"
Sep 16 16:18:25.868285 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:25.868250    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cdf926a-9d37-4bf6-a613-ce5cd6541cd9-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx\" (UID: \"9cdf926a-9d37-4bf6-a613-ce5cd6541cd9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx"
Sep 16 16:18:25.969128 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:25.969092    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cdf926a-9d37-4bf6-a613-ce5cd6541cd9-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx\" (UID: \"9cdf926a-9d37-4bf6-a613-ce5cd6541cd9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx"
Sep 16 16:18:25.969322 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:25.969140    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cdf926a-9d37-4bf6-a613-ce5cd6541cd9-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx\" (UID: \"9cdf926a-9d37-4bf6-a613-ce5cd6541cd9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx"
Sep 16 16:18:25.969322 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:25.969161    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cdf926a-9d37-4bf6-a613-ce5cd6541cd9-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx\" (UID: \"9cdf926a-9d37-4bf6-a613-ce5cd6541cd9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx"
Sep 16 16:18:25.969322 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:25.969192    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsgxk\" (UniqueName: \"kubernetes.io/projected/9cdf926a-9d37-4bf6-a613-ce5cd6541cd9-kube-api-access-tsgxk\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx\" (UID: \"9cdf926a-9d37-4bf6-a613-ce5cd6541cd9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx"
Sep 16 16:18:25.969322 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:25.969212    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cdf926a-9d37-4bf6-a613-ce5cd6541cd9-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx\" (UID: \"9cdf926a-9d37-4bf6-a613-ce5cd6541cd9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx"
Sep 16 16:18:25.969322 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:25.969231    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cdf926a-9d37-4bf6-a613-ce5cd6541cd9-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx\" (UID: \"9cdf926a-9d37-4bf6-a613-ce5cd6541cd9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx"
Sep 16 16:18:25.969686 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:25.969656    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cdf926a-9d37-4bf6-a613-ce5cd6541cd9-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx\" (UID: \"9cdf926a-9d37-4bf6-a613-ce5cd6541cd9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx"
Sep 16 16:18:25.969686 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:25.969679    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cdf926a-9d37-4bf6-a613-ce5cd6541cd9-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx\" (UID: \"9cdf926a-9d37-4bf6-a613-ce5cd6541cd9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx"
Sep 16 16:18:25.969900 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:25.969715    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cdf926a-9d37-4bf6-a613-ce5cd6541cd9-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx\" (UID: \"9cdf926a-9d37-4bf6-a613-ce5cd6541cd9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx"
Sep 16 16:18:25.971502 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:25.971477    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cdf926a-9d37-4bf6-a613-ce5cd6541cd9-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx\" (UID: \"9cdf926a-9d37-4bf6-a613-ce5cd6541cd9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx"
Sep 16 16:18:25.971782 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:25.971763    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cdf926a-9d37-4bf6-a613-ce5cd6541cd9-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx\" (UID: \"9cdf926a-9d37-4bf6-a613-ce5cd6541cd9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx"
Sep 16 16:18:25.979401 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:25.979378    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsgxk\" (UniqueName: \"kubernetes.io/projected/9cdf926a-9d37-4bf6-a613-ce5cd6541cd9-kube-api-access-tsgxk\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx\" (UID: \"9cdf926a-9d37-4bf6-a613-ce5cd6541cd9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx"
Sep 16 16:18:26.058021 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:26.057983    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx"
Sep 16 16:18:26.101252 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:26.101221    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:18:26.332566 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:26.332541    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx"]
Sep 16 16:18:26.334901 ip-10-0-136-228 kubenswrapper[2554]: W0916 16:18:26.334861    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cdf926a_9d37_4bf6_a613_ce5cd6541cd9.slice/crio-239db03417907e30d36e60287c30f4e9a6c74aaede1b6c9a2d901086aed7e6c7 WatchSource:0}: Error finding container 239db03417907e30d36e60287c30f4e9a6c74aaede1b6c9a2d901086aed7e6c7: Status 404 returned error can't find the container with id 239db03417907e30d36e60287c30f4e9a6c74aaede1b6c9a2d901086aed7e6c7
Sep 16 16:18:26.336888 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:26.336869    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 16:18:27.276437 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:27.276392    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx" event={"ID":"9cdf926a-9d37-4bf6-a613-ce5cd6541cd9","Type":"ContainerStarted","Data":"239db03417907e30d36e60287c30f4e9a6c74aaede1b6c9a2d901086aed7e6c7"}
Sep 16 16:18:29.286857 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:29.286826    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx" event={"ID":"9cdf926a-9d37-4bf6-a613-ce5cd6541cd9","Type":"ContainerStarted","Data":"1a472689211b6e986fc13d96fa0201f6b695a0762541a99b264614e387b57fd0"}
Sep 16 16:18:29.287223 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:29.286922    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx"
Sep 16 16:18:29.785116 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:29.785067    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:18:30.292285 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:30.292254    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx" event={"ID":"9cdf926a-9d37-4bf6-a613-ce5cd6541cd9","Type":"ContainerStarted","Data":"c0ac147db4a52de2fe45f91ab651d42ecf398a8ad96916b27645905bab5ffaca"}
Sep 16 16:18:30.312118 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:30.312070    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx" podStartSLOduration=2.883871401 podStartE2EDuration="5.312053881s" totalImagesPullingTime="2.42818248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:18:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:18:30.311181684 +0000 UTC m=+1710.755873906" watchObservedRunningTime="2026-09-16 16:18:30.312053881 +0000 UTC m=+1710.756746132"
Sep 16 16:18:31.058107 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:31.058056    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx"
Sep 16 16:18:31.549419 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:31.549385    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:18:35.509435 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:35.509402    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:18:35.609958 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:35.609927    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:18:36.058471 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:36.058429    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx"
Sep 16 16:18:36.070540 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:36.070515    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx"
Sep 16 16:18:36.324463 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:36.324375    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx"
Sep 16 16:18:41.469606 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:41.469567    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:18:47.329500 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:18:47.329466    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx"
Sep 16 16:19:23.330868 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:19:23.330820    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:19:23.389701 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:19:23.389671    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:20:04.593834 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:04.593736    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:20:06.155983 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:06.155949    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:20:06.165771 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:06.165741    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx"]
Sep 16 16:20:06.174670 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:20:06.174635    2554 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 16:20:06.174670 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:20:06.174670 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 16:20:06.174670 ip-10-0-136-228 kubenswrapper[2554]:  > containerID="c0ac147db4a52de2fe45f91ab651d42ecf398a8ad96916b27645905bab5ffaca" cmd=["/bin/sleep","15"]
Sep 16 16:20:06.174928 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:06.174687    2554 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx" message="" err=<
Sep 16 16:20:06.174928 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:20:06.174928 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 16:20:06.174928 ip-10-0-136-228 kubenswrapper[2554]:  >
Sep 16 16:20:06.174928 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:20:06.174699    2554 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 16:20:06.174928 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:20:06.174928 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 16:20:06.174928 ip-10-0-136-228 kubenswrapper[2554]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx" podUID="9cdf926a-9d37-4bf6-a613-ce5cd6541cd9" containerName="main" containerID="cri-o://c0ac147db4a52de2fe45f91ab651d42ecf398a8ad96916b27645905bab5ffaca"
Sep 16 16:20:06.174928 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:06.174744    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx" podUID="9cdf926a-9d37-4bf6-a613-ce5cd6541cd9" containerName="main" containerID="cri-o://c0ac147db4a52de2fe45f91ab651d42ecf398a8ad96916b27645905bab5ffaca" gracePeriod=60
Sep 16 16:20:06.313914 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:06.313873    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx" podUID="9cdf926a-9d37-4bf6-a613-ce5cd6541cd9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.27:8001/health\": dial tcp 10.132.0.27:8001: connect: connection refused"
Sep 16 16:20:06.390835 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:06.390784    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx" podUID="9cdf926a-9d37-4bf6-a613-ce5cd6541cd9" containerName="llm-d-routing-sidecar" containerID="cri-o://1a472689211b6e986fc13d96fa0201f6b695a0762541a99b264614e387b57fd0" gracePeriod=60
Sep 16 16:20:06.495889 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:06.495781    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:20:06.634593 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:06.634569    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx"
Sep 16 16:20:06.656815 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:06.656751    2554 generic.go:356] "Generic (PLEG): container finished" podID="9cdf926a-9d37-4bf6-a613-ce5cd6541cd9" containerID="c0ac147db4a52de2fe45f91ab651d42ecf398a8ad96916b27645905bab5ffaca" exitCode=0
Sep 16 16:20:06.656815 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:06.656776    2554 generic.go:356] "Generic (PLEG): container finished" podID="9cdf926a-9d37-4bf6-a613-ce5cd6541cd9" containerID="1a472689211b6e986fc13d96fa0201f6b695a0762541a99b264614e387b57fd0" exitCode=0
Sep 16 16:20:06.656815 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:06.656816    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx"
Sep 16 16:20:06.657100 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:06.656832    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx" event={"ID":"9cdf926a-9d37-4bf6-a613-ce5cd6541cd9","Type":"ContainerDied","Data":"c0ac147db4a52de2fe45f91ab651d42ecf398a8ad96916b27645905bab5ffaca"}
Sep 16 16:20:06.657100 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:06.656897    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx" event={"ID":"9cdf926a-9d37-4bf6-a613-ce5cd6541cd9","Type":"ContainerDied","Data":"1a472689211b6e986fc13d96fa0201f6b695a0762541a99b264614e387b57fd0"}
Sep 16 16:20:06.657100 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:06.656924    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx" event={"ID":"9cdf926a-9d37-4bf6-a613-ce5cd6541cd9","Type":"ContainerDied","Data":"239db03417907e30d36e60287c30f4e9a6c74aaede1b6c9a2d901086aed7e6c7"}
Sep 16 16:20:06.657100 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:06.656952    2554 scope.go:117] "RemoveContainer" containerID="c0ac147db4a52de2fe45f91ab651d42ecf398a8ad96916b27645905bab5ffaca"
Sep 16 16:20:06.666804 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:06.666772    2554 scope.go:117] "RemoveContainer" containerID="1a472689211b6e986fc13d96fa0201f6b695a0762541a99b264614e387b57fd0"
Sep 16 16:20:06.674277 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:06.674251    2554 scope.go:117] "RemoveContainer" containerID="c0ac147db4a52de2fe45f91ab651d42ecf398a8ad96916b27645905bab5ffaca"
Sep 16 16:20:06.674601 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:20:06.674581    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0ac147db4a52de2fe45f91ab651d42ecf398a8ad96916b27645905bab5ffaca\": container with ID starting with c0ac147db4a52de2fe45f91ab651d42ecf398a8ad96916b27645905bab5ffaca not found: ID does not exist" containerID="c0ac147db4a52de2fe45f91ab651d42ecf398a8ad96916b27645905bab5ffaca"
Sep 16 16:20:06.674670 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:06.674610    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0ac147db4a52de2fe45f91ab651d42ecf398a8ad96916b27645905bab5ffaca"} err="failed to get container status \"c0ac147db4a52de2fe45f91ab651d42ecf398a8ad96916b27645905bab5ffaca\": rpc error: code = NotFound desc = could not find container \"c0ac147db4a52de2fe45f91ab651d42ecf398a8ad96916b27645905bab5ffaca\": container with ID starting with c0ac147db4a52de2fe45f91ab651d42ecf398a8ad96916b27645905bab5ffaca not found: ID does not exist"
Sep 16 16:20:06.674670 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:06.674628    2554 scope.go:117] "RemoveContainer" containerID="1a472689211b6e986fc13d96fa0201f6b695a0762541a99b264614e387b57fd0"
Sep 16 16:20:06.674912 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:20:06.674891    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a472689211b6e986fc13d96fa0201f6b695a0762541a99b264614e387b57fd0\": container with ID starting with 1a472689211b6e986fc13d96fa0201f6b695a0762541a99b264614e387b57fd0 not found: ID does not exist" containerID="1a472689211b6e986fc13d96fa0201f6b695a0762541a99b264614e387b57fd0"
Sep 16 16:20:06.674993 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:06.674921    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a472689211b6e986fc13d96fa0201f6b695a0762541a99b264614e387b57fd0"} err="failed to get container status \"1a472689211b6e986fc13d96fa0201f6b695a0762541a99b264614e387b57fd0\": rpc error: code = NotFound desc = could not find container \"1a472689211b6e986fc13d96fa0201f6b695a0762541a99b264614e387b57fd0\": container with ID starting with 1a472689211b6e986fc13d96fa0201f6b695a0762541a99b264614e387b57fd0 not found: ID does not exist"
Sep 16 16:20:06.674993 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:06.674944    2554 scope.go:117] "RemoveContainer" containerID="c0ac147db4a52de2fe45f91ab651d42ecf398a8ad96916b27645905bab5ffaca"
Sep 16 16:20:06.675203 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:06.675182    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0ac147db4a52de2fe45f91ab651d42ecf398a8ad96916b27645905bab5ffaca"} err="failed to get container status \"c0ac147db4a52de2fe45f91ab651d42ecf398a8ad96916b27645905bab5ffaca\": rpc error: code = NotFound desc = could not find container \"c0ac147db4a52de2fe45f91ab651d42ecf398a8ad96916b27645905bab5ffaca\": container with ID starting with c0ac147db4a52de2fe45f91ab651d42ecf398a8ad96916b27645905bab5ffaca not found: ID does not exist"
Sep 16 16:20:06.675253 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:06.675205    2554 scope.go:117] "RemoveContainer" containerID="1a472689211b6e986fc13d96fa0201f6b695a0762541a99b264614e387b57fd0"
Sep 16 16:20:06.675443 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:06.675422    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a472689211b6e986fc13d96fa0201f6b695a0762541a99b264614e387b57fd0"} err="failed to get container status \"1a472689211b6e986fc13d96fa0201f6b695a0762541a99b264614e387b57fd0\": rpc error: code = NotFound desc = could not find container \"1a472689211b6e986fc13d96fa0201f6b695a0762541a99b264614e387b57fd0\": container with ID starting with 1a472689211b6e986fc13d96fa0201f6b695a0762541a99b264614e387b57fd0 not found: ID does not exist"
Sep 16 16:20:06.735482 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:06.735451    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cdf926a-9d37-4bf6-a613-ce5cd6541cd9-home\") pod \"9cdf926a-9d37-4bf6-a613-ce5cd6541cd9\" (UID: \"9cdf926a-9d37-4bf6-a613-ce5cd6541cd9\") "
Sep 16 16:20:06.735638 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:06.735510    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cdf926a-9d37-4bf6-a613-ce5cd6541cd9-tls-certs\") pod \"9cdf926a-9d37-4bf6-a613-ce5cd6541cd9\" (UID: \"9cdf926a-9d37-4bf6-a613-ce5cd6541cd9\") "
Sep 16 16:20:06.735638 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:06.735550    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsgxk\" (UniqueName: \"kubernetes.io/projected/9cdf926a-9d37-4bf6-a613-ce5cd6541cd9-kube-api-access-tsgxk\") pod \"9cdf926a-9d37-4bf6-a613-ce5cd6541cd9\" (UID: \"9cdf926a-9d37-4bf6-a613-ce5cd6541cd9\") "
Sep 16 16:20:06.735638 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:06.735568    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cdf926a-9d37-4bf6-a613-ce5cd6541cd9-dshm\") pod \"9cdf926a-9d37-4bf6-a613-ce5cd6541cd9\" (UID: \"9cdf926a-9d37-4bf6-a613-ce5cd6541cd9\") "
Sep 16 16:20:06.735638 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:06.735587    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cdf926a-9d37-4bf6-a613-ce5cd6541cd9-tmp-dir\") pod \"9cdf926a-9d37-4bf6-a613-ce5cd6541cd9\" (UID: \"9cdf926a-9d37-4bf6-a613-ce5cd6541cd9\") "
Sep 16 16:20:06.735839 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:06.735659    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cdf926a-9d37-4bf6-a613-ce5cd6541cd9-model-cache\") pod \"9cdf926a-9d37-4bf6-a613-ce5cd6541cd9\" (UID: \"9cdf926a-9d37-4bf6-a613-ce5cd6541cd9\") "
Sep 16 16:20:06.735839 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:06.735765    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cdf926a-9d37-4bf6-a613-ce5cd6541cd9-home" (OuterVolumeSpecName: "home") pod "9cdf926a-9d37-4bf6-a613-ce5cd6541cd9" (UID: "9cdf926a-9d37-4bf6-a613-ce5cd6541cd9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:20:06.735949 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:06.735892    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cdf926a-9d37-4bf6-a613-ce5cd6541cd9-home\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:20:06.736005 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:06.735939    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cdf926a-9d37-4bf6-a613-ce5cd6541cd9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9cdf926a-9d37-4bf6-a613-ce5cd6541cd9" (UID: "9cdf926a-9d37-4bf6-a613-ce5cd6541cd9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:20:06.736058 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:06.735990    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cdf926a-9d37-4bf6-a613-ce5cd6541cd9-model-cache" (OuterVolumeSpecName: "model-cache") pod "9cdf926a-9d37-4bf6-a613-ce5cd6541cd9" (UID: "9cdf926a-9d37-4bf6-a613-ce5cd6541cd9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:20:06.737866 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:06.737836    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cdf926a-9d37-4bf6-a613-ce5cd6541cd9-dshm" (OuterVolumeSpecName: "dshm") pod "9cdf926a-9d37-4bf6-a613-ce5cd6541cd9" (UID: "9cdf926a-9d37-4bf6-a613-ce5cd6541cd9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:20:06.737968 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:06.737889    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cdf926a-9d37-4bf6-a613-ce5cd6541cd9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9cdf926a-9d37-4bf6-a613-ce5cd6541cd9" (UID: "9cdf926a-9d37-4bf6-a613-ce5cd6541cd9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:20:06.737968 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:06.737935    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cdf926a-9d37-4bf6-a613-ce5cd6541cd9-kube-api-access-tsgxk" (OuterVolumeSpecName: "kube-api-access-tsgxk") pod "9cdf926a-9d37-4bf6-a613-ce5cd6541cd9" (UID: "9cdf926a-9d37-4bf6-a613-ce5cd6541cd9"). InnerVolumeSpecName "kube-api-access-tsgxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:20:06.837097 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:06.837063    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cdf926a-9d37-4bf6-a613-ce5cd6541cd9-model-cache\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:20:06.837097 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:06.837092    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cdf926a-9d37-4bf6-a613-ce5cd6541cd9-tls-certs\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:20:06.837097 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:06.837101    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsgxk\" (UniqueName: \"kubernetes.io/projected/9cdf926a-9d37-4bf6-a613-ce5cd6541cd9-kube-api-access-tsgxk\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:20:06.837331 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:06.837111    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cdf926a-9d37-4bf6-a613-ce5cd6541cd9-dshm\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:20:06.837331 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:06.837120    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cdf926a-9d37-4bf6-a613-ce5cd6541cd9-tmp-dir\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:20:06.978575 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:06.978546    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx"]
Sep 16 16:20:06.983040 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:06.983017    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-zzxdx"]
Sep 16 16:20:08.076573 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:08.076541    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cdf926a-9d37-4bf6-a613-ce5cd6541cd9" path="/var/lib/kubelet/pods/9cdf926a-9d37-4bf6-a613-ce5cd6541cd9/volumes"
Sep 16 16:20:13.443882 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:13.443848    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:20:13.543572 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:13.543535    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:20:13.646410 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:13.646374    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:20:19.796697 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:19.796663    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:20:25.546040 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:25.545976    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:20:25.623305 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:25.623262    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:20:43.493472 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:43.493442    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:20:56.252525 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:56.252491    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:20:56.351216 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:20:56.351183    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:21:03.020578 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:03.020546    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:21:09.219395 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:09.219362    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:21:09.298251 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:09.298223    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:21:18.407153 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:18.407107    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:21:27.119728 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:27.119645    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:21:27.219746 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:27.219714    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:21:46.593598 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:46.593564    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:21:46.692456 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:46.692421    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:21:46.804414 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:46.804371    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49"]
Sep 16 16:21:46.804760 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:46.804742    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cdf926a-9d37-4bf6-a613-ce5cd6541cd9" containerName="llm-d-routing-sidecar"
Sep 16 16:21:46.805011 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:46.804763    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cdf926a-9d37-4bf6-a613-ce5cd6541cd9" containerName="llm-d-routing-sidecar"
Sep 16 16:21:46.805011 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:46.804786    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cdf926a-9d37-4bf6-a613-ce5cd6541cd9" containerName="main"
Sep 16 16:21:46.805011 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:46.804808    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cdf926a-9d37-4bf6-a613-ce5cd6541cd9" containerName="main"
Sep 16 16:21:46.805011 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:46.804862    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cdf926a-9d37-4bf6-a613-ce5cd6541cd9" containerName="llm-d-routing-sidecar"
Sep 16 16:21:46.805011 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:46.804872    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cdf926a-9d37-4bf6-a613-ce5cd6541cd9" containerName="main"
Sep 16 16:21:46.807884 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:46.807864    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49"
Sep 16 16:21:46.810824 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:46.810770    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-6qvmv\""
Sep 16 16:21:46.810956 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:46.810833    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 16 16:21:46.810956 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:46.810886    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 16 16:21:46.812231 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:46.812212    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 16 16:21:46.812355 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:46.812232    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-nqrmj\""
Sep 16 16:21:46.820005 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:46.819984    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49"]
Sep 16 16:21:46.846194 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:46.846121    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:21:46.864383 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:46.864354    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea31b0b0-4857-4f59-9433-0098c7deea7f-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49\" (UID: \"ea31b0b0-4857-4f59-9433-0098c7deea7f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49"
Sep 16 16:21:46.864511 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:46.864386    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea31b0b0-4857-4f59-9433-0098c7deea7f-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49\" (UID: \"ea31b0b0-4857-4f59-9433-0098c7deea7f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49"
Sep 16 16:21:46.864511 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:46.864405    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea31b0b0-4857-4f59-9433-0098c7deea7f-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49\" (UID: \"ea31b0b0-4857-4f59-9433-0098c7deea7f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49"
Sep 16 16:21:46.864511 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:46.864466    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea31b0b0-4857-4f59-9433-0098c7deea7f-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49\" (UID: \"ea31b0b0-4857-4f59-9433-0098c7deea7f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49"
Sep 16 16:21:46.864624 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:46.864532    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea31b0b0-4857-4f59-9433-0098c7deea7f-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49\" (UID: \"ea31b0b0-4857-4f59-9433-0098c7deea7f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49"
Sep 16 16:21:46.864624 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:46.864559    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9h8p\" (UniqueName: \"kubernetes.io/projected/ea31b0b0-4857-4f59-9433-0098c7deea7f-kube-api-access-t9h8p\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49\" (UID: \"ea31b0b0-4857-4f59-9433-0098c7deea7f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49"
Sep 16 16:21:46.965505 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:46.965473    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea31b0b0-4857-4f59-9433-0098c7deea7f-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49\" (UID: \"ea31b0b0-4857-4f59-9433-0098c7deea7f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49"
Sep 16 16:21:46.965505 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:46.965505    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea31b0b0-4857-4f59-9433-0098c7deea7f-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49\" (UID: \"ea31b0b0-4857-4f59-9433-0098c7deea7f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49"
Sep 16 16:21:46.965749 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:46.965526    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea31b0b0-4857-4f59-9433-0098c7deea7f-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49\" (UID: \"ea31b0b0-4857-4f59-9433-0098c7deea7f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49"
Sep 16 16:21:46.965749 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:46.965579    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea31b0b0-4857-4f59-9433-0098c7deea7f-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49\" (UID: \"ea31b0b0-4857-4f59-9433-0098c7deea7f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49"
Sep 16 16:21:46.965749 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:46.965600    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9h8p\" (UniqueName: \"kubernetes.io/projected/ea31b0b0-4857-4f59-9433-0098c7deea7f-kube-api-access-t9h8p\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49\" (UID: \"ea31b0b0-4857-4f59-9433-0098c7deea7f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49"
Sep 16 16:21:46.965749 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:46.965631    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea31b0b0-4857-4f59-9433-0098c7deea7f-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49\" (UID: \"ea31b0b0-4857-4f59-9433-0098c7deea7f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49"
Sep 16 16:21:46.965957 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:46.965935    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea31b0b0-4857-4f59-9433-0098c7deea7f-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49\" (UID: \"ea31b0b0-4857-4f59-9433-0098c7deea7f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49"
Sep 16 16:21:46.966013 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:46.965986    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea31b0b0-4857-4f59-9433-0098c7deea7f-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49\" (UID: \"ea31b0b0-4857-4f59-9433-0098c7deea7f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49"
Sep 16 16:21:46.966051 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:46.966025    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea31b0b0-4857-4f59-9433-0098c7deea7f-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49\" (UID: \"ea31b0b0-4857-4f59-9433-0098c7deea7f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49"
Sep 16 16:21:46.967855 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:46.967825    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea31b0b0-4857-4f59-9433-0098c7deea7f-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49\" (UID: \"ea31b0b0-4857-4f59-9433-0098c7deea7f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49"
Sep 16 16:21:46.968207 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:46.968187    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea31b0b0-4857-4f59-9433-0098c7deea7f-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49\" (UID: \"ea31b0b0-4857-4f59-9433-0098c7deea7f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49"
Sep 16 16:21:46.976459 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:46.976435    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9h8p\" (UniqueName: \"kubernetes.io/projected/ea31b0b0-4857-4f59-9433-0098c7deea7f-kube-api-access-t9h8p\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49\" (UID: \"ea31b0b0-4857-4f59-9433-0098c7deea7f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49"
Sep 16 16:21:47.069369 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:47.069334    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:21:47.117391 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:47.117320    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49"
Sep 16 16:21:47.390208 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:47.390125    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49"]
Sep 16 16:21:47.392855 ip-10-0-136-228 kubenswrapper[2554]: W0916 16:21:47.392827    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea31b0b0_4857_4f59_9433_0098c7deea7f.slice/crio-fc26841e8f961901b25144054acb4a632464061365a09da6c37fb27b4b5558bb WatchSource:0}: Error finding container fc26841e8f961901b25144054acb4a632464061365a09da6c37fb27b4b5558bb: Status 404 returned error can't find the container with id fc26841e8f961901b25144054acb4a632464061365a09da6c37fb27b4b5558bb
Sep 16 16:21:48.038120 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:48.038078    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49" event={"ID":"ea31b0b0-4857-4f59-9433-0098c7deea7f","Type":"ContainerStarted","Data":"b4423566c2f040b6e81e62fa907da9b33b7c5c361acccd26f9f6f593ecb8acf6"}
Sep 16 16:21:48.038493 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:48.038138    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49" event={"ID":"ea31b0b0-4857-4f59-9433-0098c7deea7f","Type":"ContainerStarted","Data":"fc26841e8f961901b25144054acb4a632464061365a09da6c37fb27b4b5558bb"}
Sep 16 16:21:48.038493 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:48.038172    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49"
Sep 16 16:21:49.044674 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:49.044637    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49" event={"ID":"ea31b0b0-4857-4f59-9433-0098c7deea7f","Type":"ContainerStarted","Data":"40851b24e83b0367a121eac62a48c58199d7a5983da6c2d6116cf89571931183"}
Sep 16 16:21:49.068343 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:49.068298    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49" podStartSLOduration=3.068281321 podStartE2EDuration="3.068281321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:21:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:21:49.067300084 +0000 UTC m=+1909.511992285" watchObservedRunningTime="2026-09-16 16:21:49.068281321 +0000 UTC m=+1909.512973526"
Sep 16 16:21:49.117861 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:49.117834    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49"
Sep 16 16:21:57.117934 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:57.117898    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49"
Sep 16 16:21:57.135782 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:57.135750    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49"
Sep 16 16:21:57.136785 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:57.136764    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49"
Sep 16 16:21:58.089845 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:21:58.089817    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49"
Sep 16 16:22:11.314266 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:11.314230    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:22:20.632137 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:20.632105    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:22:20.687482 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:20.687445    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49"]
Sep 16 16:22:20.695889 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:22:20.695852    2554 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 16:22:20.695889 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:22:20.695889 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 16:22:20.695889 ip-10-0-136-228 kubenswrapper[2554]:  > containerID="40851b24e83b0367a121eac62a48c58199d7a5983da6c2d6116cf89571931183" cmd=["/bin/sleep","15"]
Sep 16 16:22:20.696109 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:20.695902    2554 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49" message="" err=<
Sep 16 16:22:20.696109 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:22:20.696109 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 16:22:20.696109 ip-10-0-136-228 kubenswrapper[2554]:  >
Sep 16 16:22:20.696109 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:22:20.695914    2554 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 16:22:20.696109 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:22:20.696109 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 16:22:20.696109 ip-10-0-136-228 kubenswrapper[2554]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49" podUID="ea31b0b0-4857-4f59-9433-0098c7deea7f" containerName="main" containerID="cri-o://40851b24e83b0367a121eac62a48c58199d7a5983da6c2d6116cf89571931183"
Sep 16 16:22:20.696109 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:20.695947    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49" podUID="ea31b0b0-4857-4f59-9433-0098c7deea7f" containerName="main" containerID="cri-o://40851b24e83b0367a121eac62a48c58199d7a5983da6c2d6116cf89571931183" gracePeriod=60
Sep 16 16:22:20.788881 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:20.788847    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:22:20.911274 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:20.911241    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49" podUID="ea31b0b0-4857-4f59-9433-0098c7deea7f" containerName="llm-d-routing-sidecar" containerID="cri-o://b4423566c2f040b6e81e62fa907da9b33b7c5c361acccd26f9f6f593ecb8acf6" gracePeriod=60
Sep 16 16:22:21.149206 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:21.149181    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49"
Sep 16 16:22:21.163474 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:21.163421    2554 generic.go:356] "Generic (PLEG): container finished" podID="ea31b0b0-4857-4f59-9433-0098c7deea7f" containerID="40851b24e83b0367a121eac62a48c58199d7a5983da6c2d6116cf89571931183" exitCode=0
Sep 16 16:22:21.163474 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:21.163441    2554 generic.go:356] "Generic (PLEG): container finished" podID="ea31b0b0-4857-4f59-9433-0098c7deea7f" containerID="b4423566c2f040b6e81e62fa907da9b33b7c5c361acccd26f9f6f593ecb8acf6" exitCode=0
Sep 16 16:22:21.163474 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:21.163462    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49"
Sep 16 16:22:21.163644 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:21.163503    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49" event={"ID":"ea31b0b0-4857-4f59-9433-0098c7deea7f","Type":"ContainerDied","Data":"40851b24e83b0367a121eac62a48c58199d7a5983da6c2d6116cf89571931183"}
Sep 16 16:22:21.163644 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:21.163555    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49" event={"ID":"ea31b0b0-4857-4f59-9433-0098c7deea7f","Type":"ContainerDied","Data":"b4423566c2f040b6e81e62fa907da9b33b7c5c361acccd26f9f6f593ecb8acf6"}
Sep 16 16:22:21.163644 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:21.163574    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49" event={"ID":"ea31b0b0-4857-4f59-9433-0098c7deea7f","Type":"ContainerDied","Data":"fc26841e8f961901b25144054acb4a632464061365a09da6c37fb27b4b5558bb"}
Sep 16 16:22:21.163644 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:21.163595    2554 scope.go:117] "RemoveContainer" containerID="40851b24e83b0367a121eac62a48c58199d7a5983da6c2d6116cf89571931183"
Sep 16 16:22:21.171734 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:21.171720    2554 scope.go:117] "RemoveContainer" containerID="b4423566c2f040b6e81e62fa907da9b33b7c5c361acccd26f9f6f593ecb8acf6"
Sep 16 16:22:21.180282 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:21.180266    2554 scope.go:117] "RemoveContainer" containerID="40851b24e83b0367a121eac62a48c58199d7a5983da6c2d6116cf89571931183"
Sep 16 16:22:21.180571 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:22:21.180555    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40851b24e83b0367a121eac62a48c58199d7a5983da6c2d6116cf89571931183\": container with ID starting with 40851b24e83b0367a121eac62a48c58199d7a5983da6c2d6116cf89571931183 not found: ID does not exist" containerID="40851b24e83b0367a121eac62a48c58199d7a5983da6c2d6116cf89571931183"
Sep 16 16:22:21.180645 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:21.180582    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40851b24e83b0367a121eac62a48c58199d7a5983da6c2d6116cf89571931183"} err="failed to get container status \"40851b24e83b0367a121eac62a48c58199d7a5983da6c2d6116cf89571931183\": rpc error: code = NotFound desc = could not find container \"40851b24e83b0367a121eac62a48c58199d7a5983da6c2d6116cf89571931183\": container with ID starting with 40851b24e83b0367a121eac62a48c58199d7a5983da6c2d6116cf89571931183 not found: ID does not exist"
Sep 16 16:22:21.180645 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:21.180604    2554 scope.go:117] "RemoveContainer" containerID="b4423566c2f040b6e81e62fa907da9b33b7c5c361acccd26f9f6f593ecb8acf6"
Sep 16 16:22:21.180914 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:22:21.180892    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4423566c2f040b6e81e62fa907da9b33b7c5c361acccd26f9f6f593ecb8acf6\": container with ID starting with b4423566c2f040b6e81e62fa907da9b33b7c5c361acccd26f9f6f593ecb8acf6 not found: ID does not exist" containerID="b4423566c2f040b6e81e62fa907da9b33b7c5c361acccd26f9f6f593ecb8acf6"
Sep 16 16:22:21.181027 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:21.180917    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4423566c2f040b6e81e62fa907da9b33b7c5c361acccd26f9f6f593ecb8acf6"} err="failed to get container status \"b4423566c2f040b6e81e62fa907da9b33b7c5c361acccd26f9f6f593ecb8acf6\": rpc error: code = NotFound desc = could not find container \"b4423566c2f040b6e81e62fa907da9b33b7c5c361acccd26f9f6f593ecb8acf6\": container with ID starting with b4423566c2f040b6e81e62fa907da9b33b7c5c361acccd26f9f6f593ecb8acf6 not found: ID does not exist"
Sep 16 16:22:21.181027 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:21.180931    2554 scope.go:117] "RemoveContainer" containerID="40851b24e83b0367a121eac62a48c58199d7a5983da6c2d6116cf89571931183"
Sep 16 16:22:21.181202 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:21.181184    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40851b24e83b0367a121eac62a48c58199d7a5983da6c2d6116cf89571931183"} err="failed to get container status \"40851b24e83b0367a121eac62a48c58199d7a5983da6c2d6116cf89571931183\": rpc error: code = NotFound desc = could not find container \"40851b24e83b0367a121eac62a48c58199d7a5983da6c2d6116cf89571931183\": container with ID starting with 40851b24e83b0367a121eac62a48c58199d7a5983da6c2d6116cf89571931183 not found: ID does not exist"
Sep 16 16:22:21.181249 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:21.181203    2554 scope.go:117] "RemoveContainer" containerID="b4423566c2f040b6e81e62fa907da9b33b7c5c361acccd26f9f6f593ecb8acf6"
Sep 16 16:22:21.181426 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:21.181409    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4423566c2f040b6e81e62fa907da9b33b7c5c361acccd26f9f6f593ecb8acf6"} err="failed to get container status \"b4423566c2f040b6e81e62fa907da9b33b7c5c361acccd26f9f6f593ecb8acf6\": rpc error: code = NotFound desc = could not find container \"b4423566c2f040b6e81e62fa907da9b33b7c5c361acccd26f9f6f593ecb8acf6\": container with ID starting with b4423566c2f040b6e81e62fa907da9b33b7c5c361acccd26f9f6f593ecb8acf6 not found: ID does not exist"
Sep 16 16:22:21.207690 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:21.207656    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea31b0b0-4857-4f59-9433-0098c7deea7f-tls-certs\") pod \"ea31b0b0-4857-4f59-9433-0098c7deea7f\" (UID: \"ea31b0b0-4857-4f59-9433-0098c7deea7f\") "
Sep 16 16:22:21.207690 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:21.207698    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea31b0b0-4857-4f59-9433-0098c7deea7f-tmp-dir\") pod \"ea31b0b0-4857-4f59-9433-0098c7deea7f\" (UID: \"ea31b0b0-4857-4f59-9433-0098c7deea7f\") "
Sep 16 16:22:21.207853 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:21.207734    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9h8p\" (UniqueName: \"kubernetes.io/projected/ea31b0b0-4857-4f59-9433-0098c7deea7f-kube-api-access-t9h8p\") pod \"ea31b0b0-4857-4f59-9433-0098c7deea7f\" (UID: \"ea31b0b0-4857-4f59-9433-0098c7deea7f\") "
Sep 16 16:22:21.207938 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:21.207919    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea31b0b0-4857-4f59-9433-0098c7deea7f-home\") pod \"ea31b0b0-4857-4f59-9433-0098c7deea7f\" (UID: \"ea31b0b0-4857-4f59-9433-0098c7deea7f\") "
Sep 16 16:22:21.207997 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:21.207981    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea31b0b0-4857-4f59-9433-0098c7deea7f-dshm\") pod \"ea31b0b0-4857-4f59-9433-0098c7deea7f\" (UID: \"ea31b0b0-4857-4f59-9433-0098c7deea7f\") "
Sep 16 16:22:21.208059 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:21.208031    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea31b0b0-4857-4f59-9433-0098c7deea7f-model-cache\") pod \"ea31b0b0-4857-4f59-9433-0098c7deea7f\" (UID: \"ea31b0b0-4857-4f59-9433-0098c7deea7f\") "
Sep 16 16:22:21.208264 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:21.207979    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea31b0b0-4857-4f59-9433-0098c7deea7f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ea31b0b0-4857-4f59-9433-0098c7deea7f" (UID: "ea31b0b0-4857-4f59-9433-0098c7deea7f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:22:21.208264 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:21.208204    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea31b0b0-4857-4f59-9433-0098c7deea7f-home" (OuterVolumeSpecName: "home") pod "ea31b0b0-4857-4f59-9433-0098c7deea7f" (UID: "ea31b0b0-4857-4f59-9433-0098c7deea7f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:22:21.208435 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:21.208231    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea31b0b0-4857-4f59-9433-0098c7deea7f-model-cache" (OuterVolumeSpecName: "model-cache") pod "ea31b0b0-4857-4f59-9433-0098c7deea7f" (UID: "ea31b0b0-4857-4f59-9433-0098c7deea7f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:22:21.210044 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:21.210012    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea31b0b0-4857-4f59-9433-0098c7deea7f-kube-api-access-t9h8p" (OuterVolumeSpecName: "kube-api-access-t9h8p") pod "ea31b0b0-4857-4f59-9433-0098c7deea7f" (UID: "ea31b0b0-4857-4f59-9433-0098c7deea7f"). InnerVolumeSpecName "kube-api-access-t9h8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:22:21.210140 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:21.210047    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea31b0b0-4857-4f59-9433-0098c7deea7f-dshm" (OuterVolumeSpecName: "dshm") pod "ea31b0b0-4857-4f59-9433-0098c7deea7f" (UID: "ea31b0b0-4857-4f59-9433-0098c7deea7f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:22:21.210140 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:21.210052    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea31b0b0-4857-4f59-9433-0098c7deea7f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ea31b0b0-4857-4f59-9433-0098c7deea7f" (UID: "ea31b0b0-4857-4f59-9433-0098c7deea7f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:22:21.308778 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:21.308749    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9h8p\" (UniqueName: \"kubernetes.io/projected/ea31b0b0-4857-4f59-9433-0098c7deea7f-kube-api-access-t9h8p\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:22:21.308778 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:21.308774    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea31b0b0-4857-4f59-9433-0098c7deea7f-home\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:22:21.308778 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:21.308785    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea31b0b0-4857-4f59-9433-0098c7deea7f-dshm\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:22:21.309008 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:21.308816    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea31b0b0-4857-4f59-9433-0098c7deea7f-model-cache\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:22:21.309008 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:21.308830    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea31b0b0-4857-4f59-9433-0098c7deea7f-tls-certs\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:22:21.309008 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:21.308841    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea31b0b0-4857-4f59-9433-0098c7deea7f-tmp-dir\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:22:21.486621 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:21.486592    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49"]
Sep 16 16:22:21.490200 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:21.490166    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49"]
Sep 16 16:22:22.077545 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:22.077512    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea31b0b0-4857-4f59-9433-0098c7deea7f" path="/var/lib/kubelet/pods/ea31b0b0-4857-4f59-9433-0098c7deea7f/volumes"
Sep 16 16:22:22.079151 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:22:22.079125    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-mwj49" podUID="ea31b0b0-4857-4f59-9433-0098c7deea7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8001/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 16 16:23:10.681250 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:10.681214    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:23:10.781668 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:10.781632    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:23:17.225412 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:17.225374    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:23:22.258362 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:22.258327    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz"]
Sep 16 16:23:22.258744 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:22.258579    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea31b0b0-4857-4f59-9433-0098c7deea7f" containerName="main"
Sep 16 16:23:22.258744 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:22.258590    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea31b0b0-4857-4f59-9433-0098c7deea7f" containerName="main"
Sep 16 16:23:22.258744 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:22.258598    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea31b0b0-4857-4f59-9433-0098c7deea7f" containerName="llm-d-routing-sidecar"
Sep 16 16:23:22.258744 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:22.258605    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea31b0b0-4857-4f59-9433-0098c7deea7f" containerName="llm-d-routing-sidecar"
Sep 16 16:23:22.258744 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:22.258654    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea31b0b0-4857-4f59-9433-0098c7deea7f" containerName="llm-d-routing-sidecar"
Sep 16 16:23:22.258744 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:22.258661    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea31b0b0-4857-4f59-9433-0098c7deea7f" containerName="main"
Sep 16 16:23:22.261512 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:22.261494    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz"
Sep 16 16:23:22.264321 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:22.264301    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 16 16:23:22.264593 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:22.264571    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 16:23:22.264681 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:22.264595    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 16 16:23:22.265691 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:22.265675    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 16 16:23:22.265967 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:22.265941    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-qvcct\""
Sep 16 16:23:22.266071 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:22.265968    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 16 16:23:22.273996 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:22.273975    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz"]
Sep 16 16:23:22.297422 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:22.297396    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:23:22.456422 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:22.456386    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa3b0ce7-31d8-4621-a768-2227418a9b78-dshm\") pod \"tokenizer-clean-path-test-kserve-84c665f65-7w9dz\" (UID: \"fa3b0ce7-31d8-4621-a768-2227418a9b78\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz"
Sep 16 16:23:22.456422 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:22.456429    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa3b0ce7-31d8-4621-a768-2227418a9b78-home\") pod \"tokenizer-clean-path-test-kserve-84c665f65-7w9dz\" (UID: \"fa3b0ce7-31d8-4621-a768-2227418a9b78\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz"
Sep 16 16:23:22.456643 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:22.456453    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa3b0ce7-31d8-4621-a768-2227418a9b78-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-84c665f65-7w9dz\" (UID: \"fa3b0ce7-31d8-4621-a768-2227418a9b78\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz"
Sep 16 16:23:22.456643 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:22.456491    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kvww\" (UniqueName: \"kubernetes.io/projected/fa3b0ce7-31d8-4621-a768-2227418a9b78-kube-api-access-9kvww\") pod \"tokenizer-clean-path-test-kserve-84c665f65-7w9dz\" (UID: \"fa3b0ce7-31d8-4621-a768-2227418a9b78\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz"
Sep 16 16:23:22.456643 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:22.456577    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa3b0ce7-31d8-4621-a768-2227418a9b78-model-cache\") pod \"tokenizer-clean-path-test-kserve-84c665f65-7w9dz\" (UID: \"fa3b0ce7-31d8-4621-a768-2227418a9b78\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz"
Sep 16 16:23:22.456643 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:22.456607    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa3b0ce7-31d8-4621-a768-2227418a9b78-tls-certs\") pod \"tokenizer-clean-path-test-kserve-84c665f65-7w9dz\" (UID: \"fa3b0ce7-31d8-4621-a768-2227418a9b78\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz"
Sep 16 16:23:22.456775 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:22.456665    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa3b0ce7-31d8-4621-a768-2227418a9b78-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-84c665f65-7w9dz\" (UID: \"fa3b0ce7-31d8-4621-a768-2227418a9b78\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz"
Sep 16 16:23:22.456775 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:22.456690    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa3b0ce7-31d8-4621-a768-2227418a9b78-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-84c665f65-7w9dz\" (UID: \"fa3b0ce7-31d8-4621-a768-2227418a9b78\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz"
Sep 16 16:23:22.557419 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:22.557385    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa3b0ce7-31d8-4621-a768-2227418a9b78-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-84c665f65-7w9dz\" (UID: \"fa3b0ce7-31d8-4621-a768-2227418a9b78\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz"
Sep 16 16:23:22.557606 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:22.557432    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa3b0ce7-31d8-4621-a768-2227418a9b78-dshm\") pod \"tokenizer-clean-path-test-kserve-84c665f65-7w9dz\" (UID: \"fa3b0ce7-31d8-4621-a768-2227418a9b78\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz"
Sep 16 16:23:22.557606 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:22.557573    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa3b0ce7-31d8-4621-a768-2227418a9b78-home\") pod \"tokenizer-clean-path-test-kserve-84c665f65-7w9dz\" (UID: \"fa3b0ce7-31d8-4621-a768-2227418a9b78\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz"
Sep 16 16:23:22.557710 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:22.557643    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa3b0ce7-31d8-4621-a768-2227418a9b78-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-84c665f65-7w9dz\" (UID: \"fa3b0ce7-31d8-4621-a768-2227418a9b78\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz"
Sep 16 16:23:22.557710 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:22.557676    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kvww\" (UniqueName: \"kubernetes.io/projected/fa3b0ce7-31d8-4621-a768-2227418a9b78-kube-api-access-9kvww\") pod \"tokenizer-clean-path-test-kserve-84c665f65-7w9dz\" (UID: \"fa3b0ce7-31d8-4621-a768-2227418a9b78\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz"
Sep 16 16:23:22.557845 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:22.557710    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa3b0ce7-31d8-4621-a768-2227418a9b78-model-cache\") pod \"tokenizer-clean-path-test-kserve-84c665f65-7w9dz\" (UID: \"fa3b0ce7-31d8-4621-a768-2227418a9b78\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz"
Sep 16 16:23:22.557845 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:22.557746    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa3b0ce7-31d8-4621-a768-2227418a9b78-tls-certs\") pod \"tokenizer-clean-path-test-kserve-84c665f65-7w9dz\" (UID: \"fa3b0ce7-31d8-4621-a768-2227418a9b78\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz"
Sep 16 16:23:22.557845 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:22.557808    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa3b0ce7-31d8-4621-a768-2227418a9b78-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-84c665f65-7w9dz\" (UID: \"fa3b0ce7-31d8-4621-a768-2227418a9b78\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz"
Sep 16 16:23:22.557845 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:22.557824    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa3b0ce7-31d8-4621-a768-2227418a9b78-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-84c665f65-7w9dz\" (UID: \"fa3b0ce7-31d8-4621-a768-2227418a9b78\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz"
Sep 16 16:23:22.558054 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:22.557956    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa3b0ce7-31d8-4621-a768-2227418a9b78-home\") pod \"tokenizer-clean-path-test-kserve-84c665f65-7w9dz\" (UID: \"fa3b0ce7-31d8-4621-a768-2227418a9b78\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz"
Sep 16 16:23:22.558117 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:22.558055    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa3b0ce7-31d8-4621-a768-2227418a9b78-model-cache\") pod \"tokenizer-clean-path-test-kserve-84c665f65-7w9dz\" (UID: \"fa3b0ce7-31d8-4621-a768-2227418a9b78\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz"
Sep 16 16:23:22.558285 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:22.558264    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa3b0ce7-31d8-4621-a768-2227418a9b78-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-84c665f65-7w9dz\" (UID: \"fa3b0ce7-31d8-4621-a768-2227418a9b78\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz"
Sep 16 16:23:22.558514 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:22.558487    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa3b0ce7-31d8-4621-a768-2227418a9b78-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-84c665f65-7w9dz\" (UID: \"fa3b0ce7-31d8-4621-a768-2227418a9b78\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz"
Sep 16 16:23:22.559769 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:22.559744    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa3b0ce7-31d8-4621-a768-2227418a9b78-dshm\") pod \"tokenizer-clean-path-test-kserve-84c665f65-7w9dz\" (UID: \"fa3b0ce7-31d8-4621-a768-2227418a9b78\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz"
Sep 16 16:23:22.560409 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:22.560388    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa3b0ce7-31d8-4621-a768-2227418a9b78-tls-certs\") pod \"tokenizer-clean-path-test-kserve-84c665f65-7w9dz\" (UID: \"fa3b0ce7-31d8-4621-a768-2227418a9b78\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz"
Sep 16 16:23:22.566004 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:22.565980    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kvww\" (UniqueName: \"kubernetes.io/projected/fa3b0ce7-31d8-4621-a768-2227418a9b78-kube-api-access-9kvww\") pod \"tokenizer-clean-path-test-kserve-84c665f65-7w9dz\" (UID: \"fa3b0ce7-31d8-4621-a768-2227418a9b78\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz"
Sep 16 16:23:22.571664 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:22.571636    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz"
Sep 16 16:23:22.573734 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:22.573706    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:23:22.640211 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:22.640161    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:23:22.845136 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:22.845059    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz"]
Sep 16 16:23:22.848211 ip-10-0-136-228 kubenswrapper[2554]: W0916 16:23:22.848184    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa3b0ce7_31d8_4621_a768_2227418a9b78.slice/crio-474411ad132f678a421df785837625082dc42f00c314762eefd2a43aa71c6bc6 WatchSource:0}: Error finding container 474411ad132f678a421df785837625082dc42f00c314762eefd2a43aa71c6bc6: Status 404 returned error can't find the container with id 474411ad132f678a421df785837625082dc42f00c314762eefd2a43aa71c6bc6
Sep 16 16:23:23.394316 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:23.394281    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz" event={"ID":"fa3b0ce7-31d8-4621-a768-2227418a9b78","Type":"ContainerStarted","Data":"e107d7802fa1028c97935cce4596dd6568bf5fb851bb11988aba62721ca67f14"}
Sep 16 16:23:23.394690 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:23.394331    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz" event={"ID":"fa3b0ce7-31d8-4621-a768-2227418a9b78","Type":"ContainerStarted","Data":"474411ad132f678a421df785837625082dc42f00c314762eefd2a43aa71c6bc6"}
Sep 16 16:23:24.342920 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:24.342885    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:23:24.442933 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:24.442900    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:23:25.403152 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:25.403063    2554 generic.go:356] "Generic (PLEG): container finished" podID="fa3b0ce7-31d8-4621-a768-2227418a9b78" containerID="e107d7802fa1028c97935cce4596dd6568bf5fb851bb11988aba62721ca67f14" exitCode=0
Sep 16 16:23:25.403341 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:25.403144    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz" event={"ID":"fa3b0ce7-31d8-4621-a768-2227418a9b78","Type":"ContainerDied","Data":"e107d7802fa1028c97935cce4596dd6568bf5fb851bb11988aba62721ca67f14"}
Sep 16 16:23:26.411287 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:26.411247    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz" event={"ID":"fa3b0ce7-31d8-4621-a768-2227418a9b78","Type":"ContainerStarted","Data":"5548d2945453c8f87f662ef42924dec26349b92f9a393679d268e58fcf82d0cb"}
Sep 16 16:23:26.431723 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:26.431677    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz" podStartSLOduration=2.284007289 podStartE2EDuration="4.431662791s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.147655502s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:23:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:23:26.430709326 +0000 UTC m=+2006.875401529" watchObservedRunningTime="2026-09-16 16:23:26.431662791 +0000 UTC m=+2006.876354970"
Sep 16 16:23:26.572422 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:26.572373    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz"
Sep 16 16:23:29.842228 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:29.842188    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:23:32.572058 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:32.572014    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz"
Sep 16 16:23:32.584422 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:32.584394    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz"
Sep 16 16:23:33.452453 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:33.452420    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz"
Sep 16 16:23:51.197043 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:51.197008    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:23:53.076749 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:53.076704    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:23:56.279621 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:23:56.279589    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:24:23.353555 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:23.353479    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-7hwbc"]
Sep 16 16:24:23.355435 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:23.355420    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-7hwbc"
Sep 16 16:24:23.358377 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:23.358353    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"openshift-service-ca.crt\""
Sep 16 16:24:23.359738 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:23.359717    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"kube-root-ca.crt\""
Sep 16 16:24:23.359837 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:23.359749    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"test-div-v1-kserve-self-signed-certs\""
Sep 16 16:24:23.359837 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:23.359719    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"default-dockercfg-67wfp\""
Sep 16 16:24:23.365142 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:23.365117    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-7hwbc"]
Sep 16 16:24:23.393529 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:23.393488    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:24:23.440672 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:23.440637    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8872b63b-1948-496d-bccf-0743b46a3b6a-dshm\") pod \"test-div-v1-kserve-fb566d667-7hwbc\" (UID: \"8872b63b-1948-496d-bccf-0743b46a3b6a\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-7hwbc"
Sep 16 16:24:23.440857 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:23.440676    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8872b63b-1948-496d-bccf-0743b46a3b6a-tls-certs\") pod \"test-div-v1-kserve-fb566d667-7hwbc\" (UID: \"8872b63b-1948-496d-bccf-0743b46a3b6a\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-7hwbc"
Sep 16 16:24:23.440857 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:23.440714    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wz8p\" (UniqueName: \"kubernetes.io/projected/8872b63b-1948-496d-bccf-0743b46a3b6a-kube-api-access-2wz8p\") pod \"test-div-v1-kserve-fb566d667-7hwbc\" (UID: \"8872b63b-1948-496d-bccf-0743b46a3b6a\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-7hwbc"
Sep 16 16:24:23.440857 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:23.440742    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8872b63b-1948-496d-bccf-0743b46a3b6a-home\") pod \"test-div-v1-kserve-fb566d667-7hwbc\" (UID: \"8872b63b-1948-496d-bccf-0743b46a3b6a\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-7hwbc"
Sep 16 16:24:23.440857 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:23.440778    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8872b63b-1948-496d-bccf-0743b46a3b6a-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-7hwbc\" (UID: \"8872b63b-1948-496d-bccf-0743b46a3b6a\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-7hwbc"
Sep 16 16:24:23.440857 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:23.440830    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8872b63b-1948-496d-bccf-0743b46a3b6a-model-cache\") pod \"test-div-v1-kserve-fb566d667-7hwbc\" (UID: \"8872b63b-1948-496d-bccf-0743b46a3b6a\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-7hwbc"
Sep 16 16:24:23.542105 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:23.542068    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wz8p\" (UniqueName: \"kubernetes.io/projected/8872b63b-1948-496d-bccf-0743b46a3b6a-kube-api-access-2wz8p\") pod \"test-div-v1-kserve-fb566d667-7hwbc\" (UID: \"8872b63b-1948-496d-bccf-0743b46a3b6a\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-7hwbc"
Sep 16 16:24:23.542105 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:23.542102    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8872b63b-1948-496d-bccf-0743b46a3b6a-home\") pod \"test-div-v1-kserve-fb566d667-7hwbc\" (UID: \"8872b63b-1948-496d-bccf-0743b46a3b6a\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-7hwbc"
Sep 16 16:24:23.542351 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:23.542130    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8872b63b-1948-496d-bccf-0743b46a3b6a-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-7hwbc\" (UID: \"8872b63b-1948-496d-bccf-0743b46a3b6a\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-7hwbc"
Sep 16 16:24:23.542351 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:23.542153    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8872b63b-1948-496d-bccf-0743b46a3b6a-model-cache\") pod \"test-div-v1-kserve-fb566d667-7hwbc\" (UID: \"8872b63b-1948-496d-bccf-0743b46a3b6a\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-7hwbc"
Sep 16 16:24:23.542351 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:23.542239    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8872b63b-1948-496d-bccf-0743b46a3b6a-dshm\") pod \"test-div-v1-kserve-fb566d667-7hwbc\" (UID: \"8872b63b-1948-496d-bccf-0743b46a3b6a\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-7hwbc"
Sep 16 16:24:23.542351 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:23.542275    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8872b63b-1948-496d-bccf-0743b46a3b6a-tls-certs\") pod \"test-div-v1-kserve-fb566d667-7hwbc\" (UID: \"8872b63b-1948-496d-bccf-0743b46a3b6a\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-7hwbc"
Sep 16 16:24:23.542621 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:23.542596    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8872b63b-1948-496d-bccf-0743b46a3b6a-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-7hwbc\" (UID: \"8872b63b-1948-496d-bccf-0743b46a3b6a\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-7hwbc"
Sep 16 16:24:23.542701 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:23.542623    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8872b63b-1948-496d-bccf-0743b46a3b6a-home\") pod \"test-div-v1-kserve-fb566d667-7hwbc\" (UID: \"8872b63b-1948-496d-bccf-0743b46a3b6a\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-7hwbc"
Sep 16 16:24:23.542701 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:23.542646    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8872b63b-1948-496d-bccf-0743b46a3b6a-model-cache\") pod \"test-div-v1-kserve-fb566d667-7hwbc\" (UID: \"8872b63b-1948-496d-bccf-0743b46a3b6a\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-7hwbc"
Sep 16 16:24:23.544485 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:23.544460    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8872b63b-1948-496d-bccf-0743b46a3b6a-dshm\") pod \"test-div-v1-kserve-fb566d667-7hwbc\" (UID: \"8872b63b-1948-496d-bccf-0743b46a3b6a\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-7hwbc"
Sep 16 16:24:23.544851 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:23.544835    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8872b63b-1948-496d-bccf-0743b46a3b6a-tls-certs\") pod \"test-div-v1-kserve-fb566d667-7hwbc\" (UID: \"8872b63b-1948-496d-bccf-0743b46a3b6a\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-7hwbc"
Sep 16 16:24:23.551457 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:23.551434    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wz8p\" (UniqueName: \"kubernetes.io/projected/8872b63b-1948-496d-bccf-0743b46a3b6a-kube-api-access-2wz8p\") pod \"test-div-v1-kserve-fb566d667-7hwbc\" (UID: \"8872b63b-1948-496d-bccf-0743b46a3b6a\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-7hwbc"
Sep 16 16:24:23.665813 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:23.665706    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-7hwbc"
Sep 16 16:24:23.936277 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:23.936251    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-7hwbc"]
Sep 16 16:24:23.938863 ip-10-0-136-228 kubenswrapper[2554]: W0916 16:24:23.938834    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8872b63b_1948_496d_bccf_0743b46a3b6a.slice/crio-9e203e9fb9a45551039ec874bb3cef60aa422ff4e7ba5de8ff261d1721f4e2fe WatchSource:0}: Error finding container 9e203e9fb9a45551039ec874bb3cef60aa422ff4e7ba5de8ff261d1721f4e2fe: Status 404 returned error can't find the container with id 9e203e9fb9a45551039ec874bb3cef60aa422ff4e7ba5de8ff261d1721f4e2fe
Sep 16 16:24:23.940749 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:23.940731    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 16:24:24.632428 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:24.632388    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-7hwbc" event={"ID":"8872b63b-1948-496d-bccf-0743b46a3b6a","Type":"ContainerStarted","Data":"9e203e9fb9a45551039ec874bb3cef60aa422ff4e7ba5de8ff261d1721f4e2fe"}
Sep 16 16:24:25.637145 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:25.637109    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-7hwbc" event={"ID":"8872b63b-1948-496d-bccf-0743b46a3b6a","Type":"ContainerStarted","Data":"dce4fac7b992d15a5e774992e3733733f4aa0a410b43c13fee90428cccd728b6"}
Sep 16 16:24:25.654099 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:25.654048    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-7hwbc" podStartSLOduration=1.661860002 podStartE2EDuration="2.654025298s" totalImagesPullingTime="992.165296ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:24:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:24:25.653704725 +0000 UTC m=+2066.098396922" watchObservedRunningTime="2026-09-16 16:24:25.654025298 +0000 UTC m=+2066.098717478"
Sep 16 16:24:25.665935 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:25.665906    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-7hwbc"
Sep 16 16:24:26.810110 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:26.810077    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:24:33.665903 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:33.665866    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-7hwbc"
Sep 16 16:24:33.666471 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:33.666418    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:24:33.678302 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:33.678280    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-7hwbc"
Sep 16 16:24:34.680222 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:34.680186    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-7hwbc"
Sep 16 16:24:39.422184 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:39.422149    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:24:39.493303 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:39.493272    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:24:51.644206 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:51.644170    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz"]
Sep 16 16:24:51.644748 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:51.644384    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:24:51.653432 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:24:51.653388    2554 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 16:24:51.653432 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:24:51.653432 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 16:24:51.653432 ip-10-0-136-228 kubenswrapper[2554]:  > containerID="5548d2945453c8f87f662ef42924dec26349b92f9a393679d268e58fcf82d0cb" cmd=["/bin/sleep","15"]
Sep 16 16:24:51.653661 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:51.653452    2554 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz" message="" err=<
Sep 16 16:24:51.653661 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:24:51.653661 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 16:24:51.653661 ip-10-0-136-228 kubenswrapper[2554]:  >
Sep 16 16:24:51.653661 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:24:51.653470    2554 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 16:24:51.653661 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:24:51.653661 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 16:24:51.653661 ip-10-0-136-228 kubenswrapper[2554]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz" podUID="fa3b0ce7-31d8-4621-a768-2227418a9b78" containerName="main" containerID="cri-o://5548d2945453c8f87f662ef42924dec26349b92f9a393679d268e58fcf82d0cb"
Sep 16 16:24:51.653661 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:51.653527    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz" podUID="fa3b0ce7-31d8-4621-a768-2227418a9b78" containerName="main" containerID="cri-o://5548d2945453c8f87f662ef42924dec26349b92f9a393679d268e58fcf82d0cb" gracePeriod=60
Sep 16 16:24:51.713521 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:51.713488    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:24:51.994053 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:51.994030    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz"
Sep 16 16:24:52.060013 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:52.059979    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa3b0ce7-31d8-4621-a768-2227418a9b78-home\") pod \"fa3b0ce7-31d8-4621-a768-2227418a9b78\" (UID: \"fa3b0ce7-31d8-4621-a768-2227418a9b78\") "
Sep 16 16:24:52.060201 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:52.060035    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa3b0ce7-31d8-4621-a768-2227418a9b78-tls-certs\") pod \"fa3b0ce7-31d8-4621-a768-2227418a9b78\" (UID: \"fa3b0ce7-31d8-4621-a768-2227418a9b78\") "
Sep 16 16:24:52.060201 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:52.060067    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kvww\" (UniqueName: \"kubernetes.io/projected/fa3b0ce7-31d8-4621-a768-2227418a9b78-kube-api-access-9kvww\") pod \"fa3b0ce7-31d8-4621-a768-2227418a9b78\" (UID: \"fa3b0ce7-31d8-4621-a768-2227418a9b78\") "
Sep 16 16:24:52.060201 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:52.060096    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa3b0ce7-31d8-4621-a768-2227418a9b78-tmp-dir\") pod \"fa3b0ce7-31d8-4621-a768-2227418a9b78\" (UID: \"fa3b0ce7-31d8-4621-a768-2227418a9b78\") "
Sep 16 16:24:52.060201 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:52.060131    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa3b0ce7-31d8-4621-a768-2227418a9b78-dshm\") pod \"fa3b0ce7-31d8-4621-a768-2227418a9b78\" (UID: \"fa3b0ce7-31d8-4621-a768-2227418a9b78\") "
Sep 16 16:24:52.060201 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:52.060168    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa3b0ce7-31d8-4621-a768-2227418a9b78-cabundle-cert\") pod \"fa3b0ce7-31d8-4621-a768-2227418a9b78\" (UID: \"fa3b0ce7-31d8-4621-a768-2227418a9b78\") "
Sep 16 16:24:52.060201 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:52.060195    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa3b0ce7-31d8-4621-a768-2227418a9b78-kserve-provision-location\") pod \"fa3b0ce7-31d8-4621-a768-2227418a9b78\" (UID: \"fa3b0ce7-31d8-4621-a768-2227418a9b78\") "
Sep 16 16:24:52.060505 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:52.060218    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa3b0ce7-31d8-4621-a768-2227418a9b78-model-cache\") pod \"fa3b0ce7-31d8-4621-a768-2227418a9b78\" (UID: \"fa3b0ce7-31d8-4621-a768-2227418a9b78\") "
Sep 16 16:24:52.060505 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:52.060242    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa3b0ce7-31d8-4621-a768-2227418a9b78-home" (OuterVolumeSpecName: "home") pod "fa3b0ce7-31d8-4621-a768-2227418a9b78" (UID: "fa3b0ce7-31d8-4621-a768-2227418a9b78"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:24:52.060505 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:52.060390    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa3b0ce7-31d8-4621-a768-2227418a9b78-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fa3b0ce7-31d8-4621-a768-2227418a9b78" (UID: "fa3b0ce7-31d8-4621-a768-2227418a9b78"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:24:52.060505 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:52.060419    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa3b0ce7-31d8-4621-a768-2227418a9b78-home\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:24:52.060700 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:52.060591    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa3b0ce7-31d8-4621-a768-2227418a9b78-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fa3b0ce7-31d8-4621-a768-2227418a9b78" (UID: "fa3b0ce7-31d8-4621-a768-2227418a9b78"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 16:24:52.060700 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:52.060645    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa3b0ce7-31d8-4621-a768-2227418a9b78-model-cache" (OuterVolumeSpecName: "model-cache") pod "fa3b0ce7-31d8-4621-a768-2227418a9b78" (UID: "fa3b0ce7-31d8-4621-a768-2227418a9b78"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:24:52.062947 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:52.062926    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa3b0ce7-31d8-4621-a768-2227418a9b78-dshm" (OuterVolumeSpecName: "dshm") pod "fa3b0ce7-31d8-4621-a768-2227418a9b78" (UID: "fa3b0ce7-31d8-4621-a768-2227418a9b78"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:24:52.063041 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:52.062968    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa3b0ce7-31d8-4621-a768-2227418a9b78-kube-api-access-9kvww" (OuterVolumeSpecName: "kube-api-access-9kvww") pod "fa3b0ce7-31d8-4621-a768-2227418a9b78" (UID: "fa3b0ce7-31d8-4621-a768-2227418a9b78"). InnerVolumeSpecName "kube-api-access-9kvww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:24:52.063041 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:52.062969    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa3b0ce7-31d8-4621-a768-2227418a9b78-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fa3b0ce7-31d8-4621-a768-2227418a9b78" (UID: "fa3b0ce7-31d8-4621-a768-2227418a9b78"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:24:52.082260 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:52.082227    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa3b0ce7-31d8-4621-a768-2227418a9b78-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fa3b0ce7-31d8-4621-a768-2227418a9b78" (UID: "fa3b0ce7-31d8-4621-a768-2227418a9b78"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:24:52.160817 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:52.160764    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa3b0ce7-31d8-4621-a768-2227418a9b78-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:24:52.160817 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:52.160806    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa3b0ce7-31d8-4621-a768-2227418a9b78-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:24:52.160817 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:52.160818    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa3b0ce7-31d8-4621-a768-2227418a9b78-model-cache\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:24:52.160817 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:52.160827    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa3b0ce7-31d8-4621-a768-2227418a9b78-tls-certs\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:24:52.161147 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:52.160834    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kvww\" (UniqueName: \"kubernetes.io/projected/fa3b0ce7-31d8-4621-a768-2227418a9b78-kube-api-access-9kvww\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:24:52.161147 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:52.160842    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa3b0ce7-31d8-4621-a768-2227418a9b78-tmp-dir\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:24:52.161147 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:52.160851    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa3b0ce7-31d8-4621-a768-2227418a9b78-dshm\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:24:52.750638 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:52.750602    2554 generic.go:356] "Generic (PLEG): container finished" podID="fa3b0ce7-31d8-4621-a768-2227418a9b78" containerID="5548d2945453c8f87f662ef42924dec26349b92f9a393679d268e58fcf82d0cb" exitCode=0
Sep 16 16:24:52.751089 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:52.750662    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz"
Sep 16 16:24:52.751089 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:52.750691    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz" event={"ID":"fa3b0ce7-31d8-4621-a768-2227418a9b78","Type":"ContainerDied","Data":"5548d2945453c8f87f662ef42924dec26349b92f9a393679d268e58fcf82d0cb"}
Sep 16 16:24:52.751089 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:52.750752    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz" event={"ID":"fa3b0ce7-31d8-4621-a768-2227418a9b78","Type":"ContainerDied","Data":"474411ad132f678a421df785837625082dc42f00c314762eefd2a43aa71c6bc6"}
Sep 16 16:24:52.751089 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:52.750774    2554 scope.go:117] "RemoveContainer" containerID="5548d2945453c8f87f662ef42924dec26349b92f9a393679d268e58fcf82d0cb"
Sep 16 16:24:52.761980 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:52.761955    2554 scope.go:117] "RemoveContainer" containerID="e107d7802fa1028c97935cce4596dd6568bf5fb851bb11988aba62721ca67f14"
Sep 16 16:24:52.769243 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:52.769224    2554 scope.go:117] "RemoveContainer" containerID="5548d2945453c8f87f662ef42924dec26349b92f9a393679d268e58fcf82d0cb"
Sep 16 16:24:52.769514 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:24:52.769491    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5548d2945453c8f87f662ef42924dec26349b92f9a393679d268e58fcf82d0cb\": container with ID starting with 5548d2945453c8f87f662ef42924dec26349b92f9a393679d268e58fcf82d0cb not found: ID does not exist" containerID="5548d2945453c8f87f662ef42924dec26349b92f9a393679d268e58fcf82d0cb"
Sep 16 16:24:52.769559 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:52.769524    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5548d2945453c8f87f662ef42924dec26349b92f9a393679d268e58fcf82d0cb"} err="failed to get container status \"5548d2945453c8f87f662ef42924dec26349b92f9a393679d268e58fcf82d0cb\": rpc error: code = NotFound desc = could not find container \"5548d2945453c8f87f662ef42924dec26349b92f9a393679d268e58fcf82d0cb\": container with ID starting with 5548d2945453c8f87f662ef42924dec26349b92f9a393679d268e58fcf82d0cb not found: ID does not exist"
Sep 16 16:24:52.769559 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:52.769541    2554 scope.go:117] "RemoveContainer" containerID="e107d7802fa1028c97935cce4596dd6568bf5fb851bb11988aba62721ca67f14"
Sep 16 16:24:52.769827 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:24:52.769787    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e107d7802fa1028c97935cce4596dd6568bf5fb851bb11988aba62721ca67f14\": container with ID starting with e107d7802fa1028c97935cce4596dd6568bf5fb851bb11988aba62721ca67f14 not found: ID does not exist" containerID="e107d7802fa1028c97935cce4596dd6568bf5fb851bb11988aba62721ca67f14"
Sep 16 16:24:52.769890 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:52.769834    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e107d7802fa1028c97935cce4596dd6568bf5fb851bb11988aba62721ca67f14"} err="failed to get container status \"e107d7802fa1028c97935cce4596dd6568bf5fb851bb11988aba62721ca67f14\": rpc error: code = NotFound desc = could not find container \"e107d7802fa1028c97935cce4596dd6568bf5fb851bb11988aba62721ca67f14\": container with ID starting with e107d7802fa1028c97935cce4596dd6568bf5fb851bb11988aba62721ca67f14 not found: ID does not exist"
Sep 16 16:24:52.773409 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:52.773389    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz"]
Sep 16 16:24:52.777847 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:52.777827    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-7w9dz"]
Sep 16 16:24:52.797866 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:52.797844    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:24:52.892277 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:52.892249    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:24:52.988991 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:52.988963    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:24:53.085136 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:53.085108    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-7hwbc"]
Sep 16 16:24:53.120183 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:53.120154    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:24:53.677450 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:53.677419    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:24:54.076776 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:54.076745    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa3b0ce7-31d8-4621-a768-2227418a9b78" path="/var/lib/kubelet/pods/fa3b0ce7-31d8-4621-a768-2227418a9b78/volumes"
Sep 16 16:24:57.882330 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:57.882299    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:24:58.008708 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:24:58.008678    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:25:07.809335 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:07.809296    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:25:08.103371 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:08.103284    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-7hwbc" podUID="8872b63b-1948-496d-bccf-0743b46a3b6a" containerName="main" containerID="cri-o://dce4fac7b992d15a5e774992e3733733f4aa0a410b43c13fee90428cccd728b6" gracePeriod=45
Sep 16 16:25:08.345877 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:08.345847    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-7hwbc"
Sep 16 16:25:08.378022 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:08.377947    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8872b63b-1948-496d-bccf-0743b46a3b6a-tls-certs\") pod \"8872b63b-1948-496d-bccf-0743b46a3b6a\" (UID: \"8872b63b-1948-496d-bccf-0743b46a3b6a\") "
Sep 16 16:25:08.378022 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:08.377990    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8872b63b-1948-496d-bccf-0743b46a3b6a-dshm\") pod \"8872b63b-1948-496d-bccf-0743b46a3b6a\" (UID: \"8872b63b-1948-496d-bccf-0743b46a3b6a\") "
Sep 16 16:25:08.378244 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:08.378029    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8872b63b-1948-496d-bccf-0743b46a3b6a-model-cache\") pod \"8872b63b-1948-496d-bccf-0743b46a3b6a\" (UID: \"8872b63b-1948-496d-bccf-0743b46a3b6a\") "
Sep 16 16:25:08.378244 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:08.378058    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wz8p\" (UniqueName: \"kubernetes.io/projected/8872b63b-1948-496d-bccf-0743b46a3b6a-kube-api-access-2wz8p\") pod \"8872b63b-1948-496d-bccf-0743b46a3b6a\" (UID: \"8872b63b-1948-496d-bccf-0743b46a3b6a\") "
Sep 16 16:25:08.378244 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:08.378077    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8872b63b-1948-496d-bccf-0743b46a3b6a-home\") pod \"8872b63b-1948-496d-bccf-0743b46a3b6a\" (UID: \"8872b63b-1948-496d-bccf-0743b46a3b6a\") "
Sep 16 16:25:08.378244 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:08.378111    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8872b63b-1948-496d-bccf-0743b46a3b6a-tmp-dir\") pod \"8872b63b-1948-496d-bccf-0743b46a3b6a\" (UID: \"8872b63b-1948-496d-bccf-0743b46a3b6a\") "
Sep 16 16:25:08.378463 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:08.378364    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8872b63b-1948-496d-bccf-0743b46a3b6a-model-cache" (OuterVolumeSpecName: "model-cache") pod "8872b63b-1948-496d-bccf-0743b46a3b6a" (UID: "8872b63b-1948-496d-bccf-0743b46a3b6a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:25:08.378525 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:08.378463    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8872b63b-1948-496d-bccf-0743b46a3b6a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8872b63b-1948-496d-bccf-0743b46a3b6a" (UID: "8872b63b-1948-496d-bccf-0743b46a3b6a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:25:08.378525 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:08.378478    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8872b63b-1948-496d-bccf-0743b46a3b6a-home" (OuterVolumeSpecName: "home") pod "8872b63b-1948-496d-bccf-0743b46a3b6a" (UID: "8872b63b-1948-496d-bccf-0743b46a3b6a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:25:08.380374 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:08.380343    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8872b63b-1948-496d-bccf-0743b46a3b6a-kube-api-access-2wz8p" (OuterVolumeSpecName: "kube-api-access-2wz8p") pod "8872b63b-1948-496d-bccf-0743b46a3b6a" (UID: "8872b63b-1948-496d-bccf-0743b46a3b6a"). InnerVolumeSpecName "kube-api-access-2wz8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:25:08.380530 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:08.380510    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8872b63b-1948-496d-bccf-0743b46a3b6a-dshm" (OuterVolumeSpecName: "dshm") pod "8872b63b-1948-496d-bccf-0743b46a3b6a" (UID: "8872b63b-1948-496d-bccf-0743b46a3b6a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:25:08.380601 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:08.380560    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8872b63b-1948-496d-bccf-0743b46a3b6a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8872b63b-1948-496d-bccf-0743b46a3b6a" (UID: "8872b63b-1948-496d-bccf-0743b46a3b6a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:25:08.479227 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:08.479190    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8872b63b-1948-496d-bccf-0743b46a3b6a-model-cache\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:25:08.479227 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:08.479219    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wz8p\" (UniqueName: \"kubernetes.io/projected/8872b63b-1948-496d-bccf-0743b46a3b6a-kube-api-access-2wz8p\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:25:08.479227 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:08.479229    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8872b63b-1948-496d-bccf-0743b46a3b6a-home\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:25:08.479227 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:08.479238    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8872b63b-1948-496d-bccf-0743b46a3b6a-tmp-dir\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:25:08.479494 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:08.479247    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8872b63b-1948-496d-bccf-0743b46a3b6a-tls-certs\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:25:08.479494 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:08.479255    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8872b63b-1948-496d-bccf-0743b46a3b6a-dshm\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:25:08.816152 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:08.816120    2554 generic.go:356] "Generic (PLEG): container finished" podID="8872b63b-1948-496d-bccf-0743b46a3b6a" containerID="dce4fac7b992d15a5e774992e3733733f4aa0a410b43c13fee90428cccd728b6" exitCode=0
Sep 16 16:25:08.816594 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:08.816165    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-7hwbc"
Sep 16 16:25:08.816594 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:08.816166    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-7hwbc" event={"ID":"8872b63b-1948-496d-bccf-0743b46a3b6a","Type":"ContainerDied","Data":"dce4fac7b992d15a5e774992e3733733f4aa0a410b43c13fee90428cccd728b6"}
Sep 16 16:25:08.816594 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:08.816211    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-7hwbc" event={"ID":"8872b63b-1948-496d-bccf-0743b46a3b6a","Type":"ContainerDied","Data":"9e203e9fb9a45551039ec874bb3cef60aa422ff4e7ba5de8ff261d1721f4e2fe"}
Sep 16 16:25:08.816594 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:08.816234    2554 scope.go:117] "RemoveContainer" containerID="dce4fac7b992d15a5e774992e3733733f4aa0a410b43c13fee90428cccd728b6"
Sep 16 16:25:08.824700 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:08.824681    2554 scope.go:117] "RemoveContainer" containerID="dce4fac7b992d15a5e774992e3733733f4aa0a410b43c13fee90428cccd728b6"
Sep 16 16:25:08.824994 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:25:08.824972    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dce4fac7b992d15a5e774992e3733733f4aa0a410b43c13fee90428cccd728b6\": container with ID starting with dce4fac7b992d15a5e774992e3733733f4aa0a410b43c13fee90428cccd728b6 not found: ID does not exist" containerID="dce4fac7b992d15a5e774992e3733733f4aa0a410b43c13fee90428cccd728b6"
Sep 16 16:25:08.825069 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:08.825001    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dce4fac7b992d15a5e774992e3733733f4aa0a410b43c13fee90428cccd728b6"} err="failed to get container status \"dce4fac7b992d15a5e774992e3733733f4aa0a410b43c13fee90428cccd728b6\": rpc error: code = NotFound desc = could not find container \"dce4fac7b992d15a5e774992e3733733f4aa0a410b43c13fee90428cccd728b6\": container with ID starting with dce4fac7b992d15a5e774992e3733733f4aa0a410b43c13fee90428cccd728b6 not found: ID does not exist"
Sep 16 16:25:08.847124 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:08.847093    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-7hwbc"]
Sep 16 16:25:08.854511 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:08.854489    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-7hwbc"]
Sep 16 16:25:10.076714 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:10.076681    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8872b63b-1948-496d-bccf-0743b46a3b6a" path="/var/lib/kubelet/pods/8872b63b-1948-496d-bccf-0743b46a3b6a/volumes"
Sep 16 16:25:35.746163 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:35.746129    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:25:37.916601 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:37.916562    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:25:38.046322 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:38.046288    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:25:38.146351 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:38.146302    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:25:38.246415 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:38.246336    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:25:38.493531 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:38.493495    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:25:38.594363 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:38.594326    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:25:38.698367 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:38.698331    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:25:38.798319 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:38.798289    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:25:41.686700 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:41.686668    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:25:41.766015 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:41.765984    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:25:41.868275 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:41.868237    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:25:42.991752 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:42.991717    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:25:45.038171 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:45.038134    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:25:47.690983 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:47.690949    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:25:48.563444 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:48.563416    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:25:50.732952 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:50.732919    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:25:50.799542 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:50.799510    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:25:52.603106 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:52.603031    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:25:54.978132 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:54.978102    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:25:55.079557 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:25:55.079521    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:26:14.291217 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.291183    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv"]
Sep 16 16:26:14.291767 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.291584    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa3b0ce7-31d8-4621-a768-2227418a9b78" containerName="main"
Sep 16 16:26:14.291767 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.291602    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa3b0ce7-31d8-4621-a768-2227418a9b78" containerName="main"
Sep 16 16:26:14.291767 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.291618    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa3b0ce7-31d8-4621-a768-2227418a9b78" containerName="storage-initializer"
Sep 16 16:26:14.291767 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.291627    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa3b0ce7-31d8-4621-a768-2227418a9b78" containerName="storage-initializer"
Sep 16 16:26:14.291767 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.291644    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8872b63b-1948-496d-bccf-0743b46a3b6a" containerName="main"
Sep 16 16:26:14.291767 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.291652    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="8872b63b-1948-496d-bccf-0743b46a3b6a" containerName="main"
Sep 16 16:26:14.291767 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.291719    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa3b0ce7-31d8-4621-a768-2227418a9b78" containerName="main"
Sep 16 16:26:14.291767 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.291729    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="8872b63b-1948-496d-bccf-0743b46a3b6a" containerName="main"
Sep 16 16:26:14.294749 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.294730    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv"
Sep 16 16:26:14.297727 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.297703    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 16 16:26:14.297845 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.297703    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 16 16:26:14.299118 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.299093    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-swnq8\""
Sep 16 16:26:14.299118 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.299112    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 16 16:26:14.299282 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.299162    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 16 16:26:14.299479 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.299464    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 16:26:14.308480 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.308455    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv"]
Sep 16 16:26:14.329143 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.329116    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:26:14.357339 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.357299    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv\" (UID: \"ecb0c455-2324-40af-b6a4-86a2d4cb2f05\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv"
Sep 16 16:26:14.357449 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.357344    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-model-cache\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv\" (UID: \"ecb0c455-2324-40af-b6a4-86a2d4cb2f05\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv"
Sep 16 16:26:14.357449 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.357364    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv\" (UID: \"ecb0c455-2324-40af-b6a4-86a2d4cb2f05\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv"
Sep 16 16:26:14.357449 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.357389    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbbpl\" (UniqueName: \"kubernetes.io/projected/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-kube-api-access-nbbpl\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv\" (UID: \"ecb0c455-2324-40af-b6a4-86a2d4cb2f05\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv"
Sep 16 16:26:14.357449 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.357410    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-tls-certs\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv\" (UID: \"ecb0c455-2324-40af-b6a4-86a2d4cb2f05\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv"
Sep 16 16:26:14.357449 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.357431    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv\" (UID: \"ecb0c455-2324-40af-b6a4-86a2d4cb2f05\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv"
Sep 16 16:26:14.357449 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.357447    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-dshm\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv\" (UID: \"ecb0c455-2324-40af-b6a4-86a2d4cb2f05\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv"
Sep 16 16:26:14.357645 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.357478    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-home\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv\" (UID: \"ecb0c455-2324-40af-b6a4-86a2d4cb2f05\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv"
Sep 16 16:26:14.457853 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.457787    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-home\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv\" (UID: \"ecb0c455-2324-40af-b6a4-86a2d4cb2f05\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv"
Sep 16 16:26:14.458008 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.457891    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv\" (UID: \"ecb0c455-2324-40af-b6a4-86a2d4cb2f05\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv"
Sep 16 16:26:14.458008 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.457914    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-model-cache\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv\" (UID: \"ecb0c455-2324-40af-b6a4-86a2d4cb2f05\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv"
Sep 16 16:26:14.458008 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.457932    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv\" (UID: \"ecb0c455-2324-40af-b6a4-86a2d4cb2f05\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv"
Sep 16 16:26:14.458008 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.457955    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbbpl\" (UniqueName: \"kubernetes.io/projected/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-kube-api-access-nbbpl\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv\" (UID: \"ecb0c455-2324-40af-b6a4-86a2d4cb2f05\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv"
Sep 16 16:26:14.458199 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.458010    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-tls-certs\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv\" (UID: \"ecb0c455-2324-40af-b6a4-86a2d4cb2f05\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv"
Sep 16 16:26:14.458199 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.458067    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv\" (UID: \"ecb0c455-2324-40af-b6a4-86a2d4cb2f05\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv"
Sep 16 16:26:14.458199 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.458094    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-dshm\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv\" (UID: \"ecb0c455-2324-40af-b6a4-86a2d4cb2f05\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv"
Sep 16 16:26:14.458351 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.458219    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-home\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv\" (UID: \"ecb0c455-2324-40af-b6a4-86a2d4cb2f05\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv"
Sep 16 16:26:14.458351 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.458295    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv\" (UID: \"ecb0c455-2324-40af-b6a4-86a2d4cb2f05\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv"
Sep 16 16:26:14.458514 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.458364    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv\" (UID: \"ecb0c455-2324-40af-b6a4-86a2d4cb2f05\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv"
Sep 16 16:26:14.458514 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.458426    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-model-cache\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv\" (UID: \"ecb0c455-2324-40af-b6a4-86a2d4cb2f05\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv"
Sep 16 16:26:14.458600 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.458581    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv\" (UID: \"ecb0c455-2324-40af-b6a4-86a2d4cb2f05\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv"
Sep 16 16:26:14.460406 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.460382    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-dshm\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv\" (UID: \"ecb0c455-2324-40af-b6a4-86a2d4cb2f05\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv"
Sep 16 16:26:14.460619 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.460601    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-tls-certs\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv\" (UID: \"ecb0c455-2324-40af-b6a4-86a2d4cb2f05\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv"
Sep 16 16:26:14.465923 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.465894    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbbpl\" (UniqueName: \"kubernetes.io/projected/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-kube-api-access-nbbpl\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv\" (UID: \"ecb0c455-2324-40af-b6a4-86a2d4cb2f05\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv"
Sep 16 16:26:14.548429 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.548356    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:26:14.602854 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.602818    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:26:14.604805 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.604775    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv"
Sep 16 16:26:14.878205 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:14.878152    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv"]
Sep 16 16:26:14.880391 ip-10-0-136-228 kubenswrapper[2554]: W0916 16:26:14.880362    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecb0c455_2324_40af_b6a4_86a2d4cb2f05.slice/crio-d2263bc35f6cd7b0545ce7c82edfafe0e12e6ffa33425efa4e0a83e7ef5d49f9 WatchSource:0}: Error finding container d2263bc35f6cd7b0545ce7c82edfafe0e12e6ffa33425efa4e0a83e7ef5d49f9: Status 404 returned error can't find the container with id d2263bc35f6cd7b0545ce7c82edfafe0e12e6ffa33425efa4e0a83e7ef5d49f9
Sep 16 16:26:15.058616 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:15.058576    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv" event={"ID":"ecb0c455-2324-40af-b6a4-86a2d4cb2f05","Type":"ContainerStarted","Data":"35ad6993f3608563d53c4ad25ad7f11472bd241ca6698600e137a949cfbae319"}
Sep 16 16:26:15.058811 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:15.058644    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv" event={"ID":"ecb0c455-2324-40af-b6a4-86a2d4cb2f05","Type":"ContainerStarted","Data":"d2263bc35f6cd7b0545ce7c82edfafe0e12e6ffa33425efa4e0a83e7ef5d49f9"}
Sep 16 16:26:18.073439 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:18.073402    2554 generic.go:356] "Generic (PLEG): container finished" podID="ecb0c455-2324-40af-b6a4-86a2d4cb2f05" containerID="35ad6993f3608563d53c4ad25ad7f11472bd241ca6698600e137a949cfbae319" exitCode=0
Sep 16 16:26:18.075876 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:18.075850    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv" event={"ID":"ecb0c455-2324-40af-b6a4-86a2d4cb2f05","Type":"ContainerDied","Data":"35ad6993f3608563d53c4ad25ad7f11472bd241ca6698600e137a949cfbae319"}
Sep 16 16:26:19.079705 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:19.079673    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv" event={"ID":"ecb0c455-2324-40af-b6a4-86a2d4cb2f05","Type":"ContainerStarted","Data":"fed41f352edd9272eac779111868608412bceb11da694903a77526614d1aa9b4"}
Sep 16 16:26:19.099727 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:19.099679    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv" podStartSLOduration=2.93596055 podStartE2EDuration="5.099662631s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.163702081s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:26:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:26:19.098909107 +0000 UTC m=+2179.543601305" watchObservedRunningTime="2026-09-16 16:26:19.099662631 +0000 UTC m=+2179.544354808"
Sep 16 16:26:19.605098 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:19.605053    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv"
Sep 16 16:26:20.340656 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:20.340611    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:26:24.605632 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:24.605582    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv"
Sep 16 16:26:24.618261 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:24.618231    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv"
Sep 16 16:26:25.111230 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:25.111201    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv"
Sep 16 16:26:26.061144 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:26.061114    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:26:26.118471 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:26.118438    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:26:40.289084 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:40.289048    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:26:51.118511 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:51.118480    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:26:51.292410 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:51.292364    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:26:51.300584 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:51.300554    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv"]
Sep 16 16:26:51.309748 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:26:51.309707    2554 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 16:26:51.309748 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:26:51.309748 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 16:26:51.309748 ip-10-0-136-228 kubenswrapper[2554]:  > containerID="fed41f352edd9272eac779111868608412bceb11da694903a77526614d1aa9b4" cmd=["/bin/sleep","15"]
Sep 16 16:26:51.310022 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:51.309771    2554 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv" message="" err=<
Sep 16 16:26:51.310022 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:26:51.310022 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 16:26:51.310022 ip-10-0-136-228 kubenswrapper[2554]:  >
Sep 16 16:26:51.310022 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:26:51.309788    2554 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 16:26:51.310022 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:26:51.310022 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 16:26:51.310022 ip-10-0-136-228 kubenswrapper[2554]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv" podUID="ecb0c455-2324-40af-b6a4-86a2d4cb2f05" containerName="main" containerID="cri-o://fed41f352edd9272eac779111868608412bceb11da694903a77526614d1aa9b4"
Sep 16 16:26:51.310022 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:51.309870    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv" podUID="ecb0c455-2324-40af-b6a4-86a2d4cb2f05" containerName="main" containerID="cri-o://fed41f352edd9272eac779111868608412bceb11da694903a77526614d1aa9b4" gracePeriod=60
Sep 16 16:26:51.365185 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:51.365149    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:26:51.546202 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:51.546178    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv"
Sep 16 16:26:51.645354 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:51.645322    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbbpl\" (UniqueName: \"kubernetes.io/projected/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-kube-api-access-nbbpl\") pod \"ecb0c455-2324-40af-b6a4-86a2d4cb2f05\" (UID: \"ecb0c455-2324-40af-b6a4-86a2d4cb2f05\") "
Sep 16 16:26:51.645544 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:51.645364    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-home\") pod \"ecb0c455-2324-40af-b6a4-86a2d4cb2f05\" (UID: \"ecb0c455-2324-40af-b6a4-86a2d4cb2f05\") "
Sep 16 16:26:51.645544 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:51.645403    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-kserve-provision-location\") pod \"ecb0c455-2324-40af-b6a4-86a2d4cb2f05\" (UID: \"ecb0c455-2324-40af-b6a4-86a2d4cb2f05\") "
Sep 16 16:26:51.645544 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:51.645429    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-tls-certs\") pod \"ecb0c455-2324-40af-b6a4-86a2d4cb2f05\" (UID: \"ecb0c455-2324-40af-b6a4-86a2d4cb2f05\") "
Sep 16 16:26:51.645544 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:51.645465    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-cabundle-cert\") pod \"ecb0c455-2324-40af-b6a4-86a2d4cb2f05\" (UID: \"ecb0c455-2324-40af-b6a4-86a2d4cb2f05\") "
Sep 16 16:26:51.645544 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:51.645498    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-model-cache\") pod \"ecb0c455-2324-40af-b6a4-86a2d4cb2f05\" (UID: \"ecb0c455-2324-40af-b6a4-86a2d4cb2f05\") "
Sep 16 16:26:51.645835 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:51.645545    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-dshm\") pod \"ecb0c455-2324-40af-b6a4-86a2d4cb2f05\" (UID: \"ecb0c455-2324-40af-b6a4-86a2d4cb2f05\") "
Sep 16 16:26:51.645835 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:51.645578    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-tmp-dir\") pod \"ecb0c455-2324-40af-b6a4-86a2d4cb2f05\" (UID: \"ecb0c455-2324-40af-b6a4-86a2d4cb2f05\") "
Sep 16 16:26:51.645835 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:51.645696    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-home" (OuterVolumeSpecName: "home") pod "ecb0c455-2324-40af-b6a4-86a2d4cb2f05" (UID: "ecb0c455-2324-40af-b6a4-86a2d4cb2f05"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:26:51.645835 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:51.645784    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-model-cache" (OuterVolumeSpecName: "model-cache") pod "ecb0c455-2324-40af-b6a4-86a2d4cb2f05" (UID: "ecb0c455-2324-40af-b6a4-86a2d4cb2f05"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:26:51.646051 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:51.645973    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ecb0c455-2324-40af-b6a4-86a2d4cb2f05" (UID: "ecb0c455-2324-40af-b6a4-86a2d4cb2f05"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:26:51.646051 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:51.645996    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ecb0c455-2324-40af-b6a4-86a2d4cb2f05" (UID: "ecb0c455-2324-40af-b6a4-86a2d4cb2f05"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 16:26:51.647742 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:51.647707    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ecb0c455-2324-40af-b6a4-86a2d4cb2f05" (UID: "ecb0c455-2324-40af-b6a4-86a2d4cb2f05"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:26:51.648197 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:51.648174    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-kube-api-access-nbbpl" (OuterVolumeSpecName: "kube-api-access-nbbpl") pod "ecb0c455-2324-40af-b6a4-86a2d4cb2f05" (UID: "ecb0c455-2324-40af-b6a4-86a2d4cb2f05"). InnerVolumeSpecName "kube-api-access-nbbpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:26:51.648197 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:51.648185    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-dshm" (OuterVolumeSpecName: "dshm") pod "ecb0c455-2324-40af-b6a4-86a2d4cb2f05" (UID: "ecb0c455-2324-40af-b6a4-86a2d4cb2f05"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:26:51.665513 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:51.665484    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ecb0c455-2324-40af-b6a4-86a2d4cb2f05" (UID: "ecb0c455-2324-40af-b6a4-86a2d4cb2f05"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:26:51.746620 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:51.746537    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbbpl\" (UniqueName: \"kubernetes.io/projected/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-kube-api-access-nbbpl\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:26:51.746620 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:51.746566    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-home\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:26:51.746620 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:51.746577    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:26:51.746620 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:51.746586    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-tls-certs\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:26:51.746620 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:51.746595    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:26:51.746620 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:51.746603    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-model-cache\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:26:51.746620 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:51.746610    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-dshm\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:26:51.746620 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:51.746617    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecb0c455-2324-40af-b6a4-86a2d4cb2f05-tmp-dir\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:26:52.200599 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:52.200570    2554 generic.go:356] "Generic (PLEG): container finished" podID="ecb0c455-2324-40af-b6a4-86a2d4cb2f05" containerID="fed41f352edd9272eac779111868608412bceb11da694903a77526614d1aa9b4" exitCode=0
Sep 16 16:26:52.201090 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:52.200623    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv" event={"ID":"ecb0c455-2324-40af-b6a4-86a2d4cb2f05","Type":"ContainerDied","Data":"fed41f352edd9272eac779111868608412bceb11da694903a77526614d1aa9b4"}
Sep 16 16:26:52.201090 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:52.200633    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv"
Sep 16 16:26:52.201090 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:52.200669    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv" event={"ID":"ecb0c455-2324-40af-b6a4-86a2d4cb2f05","Type":"ContainerDied","Data":"d2263bc35f6cd7b0545ce7c82edfafe0e12e6ffa33425efa4e0a83e7ef5d49f9"}
Sep 16 16:26:52.201090 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:52.200698    2554 scope.go:117] "RemoveContainer" containerID="fed41f352edd9272eac779111868608412bceb11da694903a77526614d1aa9b4"
Sep 16 16:26:52.209098 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:52.209075    2554 scope.go:117] "RemoveContainer" containerID="35ad6993f3608563d53c4ad25ad7f11472bd241ca6698600e137a949cfbae319"
Sep 16 16:26:52.216537 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:52.216516    2554 scope.go:117] "RemoveContainer" containerID="fed41f352edd9272eac779111868608412bceb11da694903a77526614d1aa9b4"
Sep 16 16:26:52.216886 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:26:52.216855    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fed41f352edd9272eac779111868608412bceb11da694903a77526614d1aa9b4\": container with ID starting with fed41f352edd9272eac779111868608412bceb11da694903a77526614d1aa9b4 not found: ID does not exist" containerID="fed41f352edd9272eac779111868608412bceb11da694903a77526614d1aa9b4"
Sep 16 16:26:52.217026 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:52.216907    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fed41f352edd9272eac779111868608412bceb11da694903a77526614d1aa9b4"} err="failed to get container status \"fed41f352edd9272eac779111868608412bceb11da694903a77526614d1aa9b4\": rpc error: code = NotFound desc = could not find container \"fed41f352edd9272eac779111868608412bceb11da694903a77526614d1aa9b4\": container with ID starting with fed41f352edd9272eac779111868608412bceb11da694903a77526614d1aa9b4 not found: ID does not exist"
Sep 16 16:26:52.217026 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:52.216934    2554 scope.go:117] "RemoveContainer" containerID="35ad6993f3608563d53c4ad25ad7f11472bd241ca6698600e137a949cfbae319"
Sep 16 16:26:52.217275 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:26:52.217247    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35ad6993f3608563d53c4ad25ad7f11472bd241ca6698600e137a949cfbae319\": container with ID starting with 35ad6993f3608563d53c4ad25ad7f11472bd241ca6698600e137a949cfbae319 not found: ID does not exist" containerID="35ad6993f3608563d53c4ad25ad7f11472bd241ca6698600e137a949cfbae319"
Sep 16 16:26:52.217391 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:52.217281    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35ad6993f3608563d53c4ad25ad7f11472bd241ca6698600e137a949cfbae319"} err="failed to get container status \"35ad6993f3608563d53c4ad25ad7f11472bd241ca6698600e137a949cfbae319\": rpc error: code = NotFound desc = could not find container \"35ad6993f3608563d53c4ad25ad7f11472bd241ca6698600e137a949cfbae319\": container with ID starting with 35ad6993f3608563d53c4ad25ad7f11472bd241ca6698600e137a949cfbae319 not found: ID does not exist"
Sep 16 16:26:52.219468 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:52.219445    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv"]
Sep 16 16:26:52.227167 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:52.227147    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-5zjpv"]
Sep 16 16:26:54.076535 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:54.076503    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecb0c455-2324-40af-b6a4-86a2d4cb2f05" path="/var/lib/kubelet/pods/ecb0c455-2324-40af-b6a4-86a2d4cb2f05/volumes"
Sep 16 16:26:55.105947 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:26:55.105914    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:27:09.848974 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:27:09.848939    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:27:16.640531 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:27:16.640493    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:27:16.745184 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:27:16.745146    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:27:16.847251 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:27:16.847219    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:27:21.719835 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:27:21.719784    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:27:28.577050 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:27:28.576966    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:27:41.236118 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:27:41.236081    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:27:41.334469 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:27:41.334437    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:27:41.751951 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:27:41.751921    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:27:41.853446 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:27:41.853397    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:27:42.995809 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:27:42.995759    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:27:44.770815 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:27:44.770768    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:27:48.110318 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:27:48.110283    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:27:48.699032 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:27:48.698998    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:27:48.799614 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:27:48.799581    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:27:54.114716 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:27:54.114683    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:27:54.215683 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:27:54.215653    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:28:06.684680 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:06.684647    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7"]
Sep 16 16:28:06.685053 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:06.684913    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecb0c455-2324-40af-b6a4-86a2d4cb2f05" containerName="main"
Sep 16 16:28:06.685053 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:06.684925    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecb0c455-2324-40af-b6a4-86a2d4cb2f05" containerName="main"
Sep 16 16:28:06.685053 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:06.684936    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecb0c455-2324-40af-b6a4-86a2d4cb2f05" containerName="storage-initializer"
Sep 16 16:28:06.685053 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:06.684941    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecb0c455-2324-40af-b6a4-86a2d4cb2f05" containerName="storage-initializer"
Sep 16 16:28:06.685053 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:06.684995    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecb0c455-2324-40af-b6a4-86a2d4cb2f05" containerName="main"
Sep 16 16:28:06.686616 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:06.686599    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7"
Sep 16 16:28:06.690968 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:06.690945    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 16 16:28:06.691116 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:06.691060    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 16:28:06.692370 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:06.692352    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 16 16:28:06.692470 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:06.692354    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-qqx6s\""
Sep 16 16:28:06.692470 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:06.692383    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 16 16:28:06.692470 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:06.692410    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 16 16:28:06.704355 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:06.704335    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7"]
Sep 16 16:28:06.717451 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:06.717432    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:28:06.788468 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:06.788445    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-k24b7\" (UID: \"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7"
Sep 16 16:28:06.788588 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:06.788479    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-k24b7\" (UID: \"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7"
Sep 16 16:28:06.788588 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:06.788497    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-home\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-k24b7\" (UID: \"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7"
Sep 16 16:28:06.788588 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:06.788536    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-dshm\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-k24b7\" (UID: \"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7"
Sep 16 16:28:06.788588 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:06.788550    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-model-cache\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-k24b7\" (UID: \"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7"
Sep 16 16:28:06.788588 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:06.788571    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf2w9\" (UniqueName: \"kubernetes.io/projected/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-kube-api-access-vf2w9\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-k24b7\" (UID: \"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7"
Sep 16 16:28:06.788780 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:06.788593    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-tls-certs\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-k24b7\" (UID: \"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7"
Sep 16 16:28:06.788780 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:06.788609    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-tmp-dir\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-k24b7\" (UID: \"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7"
Sep 16 16:28:06.889531 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:06.889502    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-dshm\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-k24b7\" (UID: \"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7"
Sep 16 16:28:06.889531 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:06.889530    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-model-cache\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-k24b7\" (UID: \"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7"
Sep 16 16:28:06.889693 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:06.889555    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf2w9\" (UniqueName: \"kubernetes.io/projected/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-kube-api-access-vf2w9\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-k24b7\" (UID: \"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7"
Sep 16 16:28:06.889693 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:06.889580    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-tls-certs\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-k24b7\" (UID: \"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7"
Sep 16 16:28:06.889693 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:06.889608    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-tmp-dir\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-k24b7\" (UID: \"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7"
Sep 16 16:28:06.889693 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:06.889639    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-k24b7\" (UID: \"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7"
Sep 16 16:28:06.889693 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:06.889674    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-k24b7\" (UID: \"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7"
Sep 16 16:28:06.889963 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:06.889820    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-home\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-k24b7\" (UID: \"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7"
Sep 16 16:28:06.890073 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:06.890046    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-model-cache\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-k24b7\" (UID: \"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7"
Sep 16 16:28:06.890133 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:06.890102    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-k24b7\" (UID: \"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7"
Sep 16 16:28:06.890193 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:06.890127    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-tmp-dir\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-k24b7\" (UID: \"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7"
Sep 16 16:28:06.890236 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:06.890204    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-home\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-k24b7\" (UID: \"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7"
Sep 16 16:28:06.890458 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:06.890441    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-k24b7\" (UID: \"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7"
Sep 16 16:28:06.891909 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:06.891891    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-dshm\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-k24b7\" (UID: \"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7"
Sep 16 16:28:06.892055 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:06.892038    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-tls-certs\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-k24b7\" (UID: \"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7"
Sep 16 16:28:06.900044 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:06.900021    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf2w9\" (UniqueName: \"kubernetes.io/projected/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-kube-api-access-vf2w9\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-k24b7\" (UID: \"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7"
Sep 16 16:28:06.971015 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:06.970935    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:28:06.997081 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:06.997052    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7"
Sep 16 16:28:07.266709 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:07.266684    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7"]
Sep 16 16:28:07.269235 ip-10-0-136-228 kubenswrapper[2554]: W0916 16:28:07.269208    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87cf9a2c_f9f9_4df8_b749_aa3ba5d9a4e8.slice/crio-5117d06ba2db25030ddadb609bfc300de90cc2c08784d2f875899fd1bbf471cf WatchSource:0}: Error finding container 5117d06ba2db25030ddadb609bfc300de90cc2c08784d2f875899fd1bbf471cf: Status 404 returned error can't find the container with id 5117d06ba2db25030ddadb609bfc300de90cc2c08784d2f875899fd1bbf471cf
Sep 16 16:28:07.333632 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:07.333607    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:28:07.473085 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:07.473049    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7" event={"ID":"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8","Type":"ContainerStarted","Data":"1b9f6758bfce09b0dc6bf010db4b73ebce4bbf6152eb2fa63e0f9067e976de6c"}
Sep 16 16:28:07.473227 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:07.473112    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7" event={"ID":"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8","Type":"ContainerStarted","Data":"5117d06ba2db25030ddadb609bfc300de90cc2c08784d2f875899fd1bbf471cf"}
Sep 16 16:28:09.482320 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:09.482291    2554 generic.go:356] "Generic (PLEG): container finished" podID="87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8" containerID="1b9f6758bfce09b0dc6bf010db4b73ebce4bbf6152eb2fa63e0f9067e976de6c" exitCode=0
Sep 16 16:28:09.482639 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:09.482368    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7" event={"ID":"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8","Type":"ContainerDied","Data":"1b9f6758bfce09b0dc6bf010db4b73ebce4bbf6152eb2fa63e0f9067e976de6c"}
Sep 16 16:28:10.488169 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:10.488134    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7" event={"ID":"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8","Type":"ContainerStarted","Data":"eed175e8b1e34836b15503276a82fbdd1acbe931471b6c4fd093768943cb0ef1"}
Sep 16 16:28:10.509827 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:10.509763    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7" podStartSLOduration=2.415731506 podStartE2EDuration="4.509748777s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.094017271s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:28:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:28:10.508343709 +0000 UTC m=+2290.953035887" watchObservedRunningTime="2026-09-16 16:28:10.509748777 +0000 UTC m=+2290.954440950"
Sep 16 16:28:10.997832 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:10.997779    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7"
Sep 16 16:28:16.997261 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:16.997216    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7"
Sep 16 16:28:17.009725 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:17.009700    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7"
Sep 16 16:28:17.525214 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:17.525178    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7"
Sep 16 16:28:20.918847 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:20.918815    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:28:34.826672 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:34.826637    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:28:44.685985 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:44.685949    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:28:44.701821 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:44.701778    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7"]
Sep 16 16:28:44.710922 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:28:44.710888    2554 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 16:28:44.710922 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:28:44.710922 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 16:28:44.710922 ip-10-0-136-228 kubenswrapper[2554]:  > containerID="eed175e8b1e34836b15503276a82fbdd1acbe931471b6c4fd093768943cb0ef1" cmd=["/bin/sleep","15"]
Sep 16 16:28:44.711237 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:44.710938    2554 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7" message="" err=<
Sep 16 16:28:44.711237 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:28:44.711237 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 16:28:44.711237 ip-10-0-136-228 kubenswrapper[2554]:  >
Sep 16 16:28:44.711237 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:28:44.710950    2554 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 16:28:44.711237 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:28:44.711237 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 16:28:44.711237 ip-10-0-136-228 kubenswrapper[2554]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7" podUID="87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8" containerName="main" containerID="cri-o://eed175e8b1e34836b15503276a82fbdd1acbe931471b6c4fd093768943cb0ef1"
Sep 16 16:28:44.711237 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:44.710991    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7" podUID="87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8" containerName="main" containerID="cri-o://eed175e8b1e34836b15503276a82fbdd1acbe931471b6c4fd093768943cb0ef1" gracePeriod=60
Sep 16 16:28:44.752452 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:44.752425    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:28:45.453644 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:45.453619    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7"
Sep 16 16:28:45.581082 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:45.580999    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-home\") pod \"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8\" (UID: \"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8\") "
Sep 16 16:28:45.581082 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:45.581041    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-model-cache\") pod \"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8\" (UID: \"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8\") "
Sep 16 16:28:45.581082 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:45.581065    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vf2w9\" (UniqueName: \"kubernetes.io/projected/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-kube-api-access-vf2w9\") pod \"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8\" (UID: \"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8\") "
Sep 16 16:28:45.581353 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:45.581093    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-dshm\") pod \"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8\" (UID: \"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8\") "
Sep 16 16:28:45.581353 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:45.581115    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-kserve-provision-location\") pod \"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8\" (UID: \"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8\") "
Sep 16 16:28:45.581353 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:45.581133    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-tls-certs\") pod \"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8\" (UID: \"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8\") "
Sep 16 16:28:45.581353 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:45.581153    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-cabundle-cert\") pod \"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8\" (UID: \"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8\") "
Sep 16 16:28:45.581353 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:45.581182    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-tmp-dir\") pod \"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8\" (UID: \"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8\") "
Sep 16 16:28:45.581353 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:45.581333    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-home" (OuterVolumeSpecName: "home") pod "87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8" (UID: "87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:28:45.581630 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:45.581328    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-model-cache" (OuterVolumeSpecName: "model-cache") pod "87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8" (UID: "87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:28:45.581630 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:45.581511    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8" (UID: "87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:28:45.581726 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:45.581619    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8" (UID: "87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 16:28:45.583386 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:45.583358    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-dshm" (OuterVolumeSpecName: "dshm") pod "87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8" (UID: "87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:28:45.583494 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:45.583433    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-kube-api-access-vf2w9" (OuterVolumeSpecName: "kube-api-access-vf2w9") pod "87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8" (UID: "87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8"). InnerVolumeSpecName "kube-api-access-vf2w9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:28:45.583494 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:45.583431    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8" (UID: "87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:28:45.607110 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:45.607084    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8" (UID: "87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:28:45.616280 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:45.616257    2554 generic.go:356] "Generic (PLEG): container finished" podID="87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8" containerID="eed175e8b1e34836b15503276a82fbdd1acbe931471b6c4fd093768943cb0ef1" exitCode=0
Sep 16 16:28:45.616383 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:45.616312    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7"
Sep 16 16:28:45.616383 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:45.616317    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7" event={"ID":"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8","Type":"ContainerDied","Data":"eed175e8b1e34836b15503276a82fbdd1acbe931471b6c4fd093768943cb0ef1"}
Sep 16 16:28:45.616465 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:45.616404    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7" event={"ID":"87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8","Type":"ContainerDied","Data":"5117d06ba2db25030ddadb609bfc300de90cc2c08784d2f875899fd1bbf471cf"}
Sep 16 16:28:45.616465 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:45.616435    2554 scope.go:117] "RemoveContainer" containerID="eed175e8b1e34836b15503276a82fbdd1acbe931471b6c4fd093768943cb0ef1"
Sep 16 16:28:45.627129 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:45.627103    2554 scope.go:117] "RemoveContainer" containerID="1b9f6758bfce09b0dc6bf010db4b73ebce4bbf6152eb2fa63e0f9067e976de6c"
Sep 16 16:28:45.635163 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:45.635144    2554 scope.go:117] "RemoveContainer" containerID="eed175e8b1e34836b15503276a82fbdd1acbe931471b6c4fd093768943cb0ef1"
Sep 16 16:28:45.635443 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:28:45.635426    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eed175e8b1e34836b15503276a82fbdd1acbe931471b6c4fd093768943cb0ef1\": container with ID starting with eed175e8b1e34836b15503276a82fbdd1acbe931471b6c4fd093768943cb0ef1 not found: ID does not exist" containerID="eed175e8b1e34836b15503276a82fbdd1acbe931471b6c4fd093768943cb0ef1"
Sep 16 16:28:45.635489 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:45.635449    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eed175e8b1e34836b15503276a82fbdd1acbe931471b6c4fd093768943cb0ef1"} err="failed to get container status \"eed175e8b1e34836b15503276a82fbdd1acbe931471b6c4fd093768943cb0ef1\": rpc error: code = NotFound desc = could not find container \"eed175e8b1e34836b15503276a82fbdd1acbe931471b6c4fd093768943cb0ef1\": container with ID starting with eed175e8b1e34836b15503276a82fbdd1acbe931471b6c4fd093768943cb0ef1 not found: ID does not exist"
Sep 16 16:28:45.635489 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:45.635464    2554 scope.go:117] "RemoveContainer" containerID="1b9f6758bfce09b0dc6bf010db4b73ebce4bbf6152eb2fa63e0f9067e976de6c"
Sep 16 16:28:45.635701 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:28:45.635681    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b9f6758bfce09b0dc6bf010db4b73ebce4bbf6152eb2fa63e0f9067e976de6c\": container with ID starting with 1b9f6758bfce09b0dc6bf010db4b73ebce4bbf6152eb2fa63e0f9067e976de6c not found: ID does not exist" containerID="1b9f6758bfce09b0dc6bf010db4b73ebce4bbf6152eb2fa63e0f9067e976de6c"
Sep 16 16:28:45.635764 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:45.635711    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b9f6758bfce09b0dc6bf010db4b73ebce4bbf6152eb2fa63e0f9067e976de6c"} err="failed to get container status \"1b9f6758bfce09b0dc6bf010db4b73ebce4bbf6152eb2fa63e0f9067e976de6c\": rpc error: code = NotFound desc = could not find container \"1b9f6758bfce09b0dc6bf010db4b73ebce4bbf6152eb2fa63e0f9067e976de6c\": container with ID starting with 1b9f6758bfce09b0dc6bf010db4b73ebce4bbf6152eb2fa63e0f9067e976de6c not found: ID does not exist"
Sep 16 16:28:45.644778 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:45.644757    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7"]
Sep 16 16:28:45.652623 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:45.652603    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-k24b7"]
Sep 16 16:28:45.682571 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:45.682548    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-model-cache\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:28:45.682571 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:45.682567    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf2w9\" (UniqueName: \"kubernetes.io/projected/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-kube-api-access-vf2w9\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:28:45.682688 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:45.682578    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-dshm\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:28:45.682688 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:45.682588    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:28:45.682688 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:45.682597    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-tls-certs\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:28:45.682688 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:45.682605    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:28:45.682688 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:45.682629    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-tmp-dir\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:28:45.682688 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:45.682636    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8-home\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:28:46.076889 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:46.076858    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8" path="/var/lib/kubelet/pods/87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8/volumes"
Sep 16 16:28:50.400472 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:50.400444    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:28:50.531408 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:50.531376    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:28:50.613853 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:50.613819    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:28:55.484368 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:28:55.484276    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:29:00.031610 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:29:00.031553    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:29:05.110463 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:29:05.110428    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:29:32.636856 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:29:32.636811    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:29:34.809060 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:29:34.809018    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:29:34.905725 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:29:34.905688    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:29:38.665392 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:29:38.665358    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:29:38.768469 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:29:38.768430    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:29:40.907472 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:29:40.907429    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:29:40.995143 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:29:40.995096    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:29:47.139951 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:29:47.139916    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:29:47.203904 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:29:47.203871    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:30:26.578137 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:26.578042    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv"]
Sep 16 16:30:26.578657 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:26.578327    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8" containerName="storage-initializer"
Sep 16 16:30:26.578657 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:26.578339    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8" containerName="storage-initializer"
Sep 16 16:30:26.578657 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:26.578347    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8" containerName="main"
Sep 16 16:30:26.578657 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:26.578353    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8" containerName="main"
Sep 16 16:30:26.578657 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:26.578402    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="87cf9a2c-f9f9-4df8-b749-aa3ba5d9a4e8" containerName="main"
Sep 16 16:30:26.581193 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:26.581174    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv"
Sep 16 16:30:26.583903 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:26.583871    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 16 16:30:26.585253 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:26.585232    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 16 16:30:26.585361 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:26.585241    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 16 16:30:26.585361 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:26.585283    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-29xkz\""
Sep 16 16:30:26.590865 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:26.590844    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv"]
Sep 16 16:30:26.620617 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:26.620583    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:30:26.720265 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:26.720233    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f2bbe5b-56de-4fc6-8174-fe8d61242d1c-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv\" (UID: \"9f2bbe5b-56de-4fc6-8174-fe8d61242d1c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv"
Sep 16 16:30:26.720265 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:26.720274    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f2bbe5b-56de-4fc6-8174-fe8d61242d1c-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv\" (UID: \"9f2bbe5b-56de-4fc6-8174-fe8d61242d1c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv"
Sep 16 16:30:26.720519 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:26.720296    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78hrg\" (UniqueName: \"kubernetes.io/projected/9f2bbe5b-56de-4fc6-8174-fe8d61242d1c-kube-api-access-78hrg\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv\" (UID: \"9f2bbe5b-56de-4fc6-8174-fe8d61242d1c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv"
Sep 16 16:30:26.720519 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:26.720389    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f2bbe5b-56de-4fc6-8174-fe8d61242d1c-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv\" (UID: \"9f2bbe5b-56de-4fc6-8174-fe8d61242d1c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv"
Sep 16 16:30:26.720519 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:26.720439    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f2bbe5b-56de-4fc6-8174-fe8d61242d1c-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv\" (UID: \"9f2bbe5b-56de-4fc6-8174-fe8d61242d1c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv"
Sep 16 16:30:26.720519 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:26.720478    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f2bbe5b-56de-4fc6-8174-fe8d61242d1c-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv\" (UID: \"9f2bbe5b-56de-4fc6-8174-fe8d61242d1c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv"
Sep 16 16:30:26.821033 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:26.820996    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f2bbe5b-56de-4fc6-8174-fe8d61242d1c-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv\" (UID: \"9f2bbe5b-56de-4fc6-8174-fe8d61242d1c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv"
Sep 16 16:30:26.821033 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:26.821032    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f2bbe5b-56de-4fc6-8174-fe8d61242d1c-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv\" (UID: \"9f2bbe5b-56de-4fc6-8174-fe8d61242d1c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv"
Sep 16 16:30:26.821261 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:26.821057    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f2bbe5b-56de-4fc6-8174-fe8d61242d1c-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv\" (UID: \"9f2bbe5b-56de-4fc6-8174-fe8d61242d1c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv"
Sep 16 16:30:26.821261 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:26.821077    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f2bbe5b-56de-4fc6-8174-fe8d61242d1c-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv\" (UID: \"9f2bbe5b-56de-4fc6-8174-fe8d61242d1c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv"
Sep 16 16:30:26.821261 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:26.821099    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78hrg\" (UniqueName: \"kubernetes.io/projected/9f2bbe5b-56de-4fc6-8174-fe8d61242d1c-kube-api-access-78hrg\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv\" (UID: \"9f2bbe5b-56de-4fc6-8174-fe8d61242d1c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv"
Sep 16 16:30:26.821261 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:26.821142    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f2bbe5b-56de-4fc6-8174-fe8d61242d1c-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv\" (UID: \"9f2bbe5b-56de-4fc6-8174-fe8d61242d1c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv"
Sep 16 16:30:26.821513 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:26.821493    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f2bbe5b-56de-4fc6-8174-fe8d61242d1c-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv\" (UID: \"9f2bbe5b-56de-4fc6-8174-fe8d61242d1c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv"
Sep 16 16:30:26.821581 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:26.821515    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f2bbe5b-56de-4fc6-8174-fe8d61242d1c-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv\" (UID: \"9f2bbe5b-56de-4fc6-8174-fe8d61242d1c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv"
Sep 16 16:30:26.821581 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:26.821559    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f2bbe5b-56de-4fc6-8174-fe8d61242d1c-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv\" (UID: \"9f2bbe5b-56de-4fc6-8174-fe8d61242d1c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv"
Sep 16 16:30:26.823392 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:26.823371    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f2bbe5b-56de-4fc6-8174-fe8d61242d1c-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv\" (UID: \"9f2bbe5b-56de-4fc6-8174-fe8d61242d1c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv"
Sep 16 16:30:26.823531 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:26.823515    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f2bbe5b-56de-4fc6-8174-fe8d61242d1c-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv\" (UID: \"9f2bbe5b-56de-4fc6-8174-fe8d61242d1c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv"
Sep 16 16:30:26.829122 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:26.829061    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78hrg\" (UniqueName: \"kubernetes.io/projected/9f2bbe5b-56de-4fc6-8174-fe8d61242d1c-kube-api-access-78hrg\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv\" (UID: \"9f2bbe5b-56de-4fc6-8174-fe8d61242d1c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv"
Sep 16 16:30:26.893679 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:26.893644    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv"
Sep 16 16:30:26.923545 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:26.923495    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:30:27.163805 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:27.163708    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv"]
Sep 16 16:30:27.166636 ip-10-0-136-228 kubenswrapper[2554]: W0916 16:30:27.166613    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f2bbe5b_56de_4fc6_8174_fe8d61242d1c.slice/crio-9715b40b7d369ae14fd8ffe01cf140baa570f0aa343031abfd0831626002c164 WatchSource:0}: Error finding container 9715b40b7d369ae14fd8ffe01cf140baa570f0aa343031abfd0831626002c164: Status 404 returned error can't find the container with id 9715b40b7d369ae14fd8ffe01cf140baa570f0aa343031abfd0831626002c164
Sep 16 16:30:27.988819 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:27.988764    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv" event={"ID":"9f2bbe5b-56de-4fc6-8174-fe8d61242d1c","Type":"ContainerStarted","Data":"9fe772a814c247059081d4bd0bec09c74fbdd661b4ae3d3a595ce3ef660b5b5f"}
Sep 16 16:30:27.989239 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:27.988835    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv" event={"ID":"9f2bbe5b-56de-4fc6-8174-fe8d61242d1c","Type":"ContainerStarted","Data":"9715b40b7d369ae14fd8ffe01cf140baa570f0aa343031abfd0831626002c164"}
Sep 16 16:30:28.006862 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:28.006815    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv" podStartSLOduration=2.006779448 podStartE2EDuration="2.006779448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:30:28.005927438 +0000 UTC m=+2428.450619636" watchObservedRunningTime="2026-09-16 16:30:28.006779448 +0000 UTC m=+2428.451471634"
Sep 16 16:30:28.893898 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:28.893863    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv"
Sep 16 16:30:34.571279 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:34.571245    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:30:36.894342 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:36.894301    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv"
Sep 16 16:30:36.906771 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:36.906730    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv"
Sep 16 16:30:37.032225 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:37.032196    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv"
Sep 16 16:30:42.463534 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:42.463501    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:30:42.557525 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:42.557491    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:30:42.667245 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:42.667209    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:30:43.084752 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:43.084725    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:30:47.529275 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:30:47.529239    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:31:09.908147 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:31:09.908111    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:31:14.578390 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:31:14.578350    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:31:14.647867 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:31:14.647832    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:31:23.202498 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:31:23.202467    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:31:25.300733 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:31:25.300700    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:31:31.306505 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:31:31.306476    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:31:31.369106 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:31:31.369075    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:31:32.585348 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:31:32.585317    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:31:53.682714 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:31:53.682636    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv"]
Sep 16 16:31:53.692684 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:31:53.692646    2554 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 16:31:53.692684 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:31:53.692684 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 16:31:53.692684 ip-10-0-136-228 kubenswrapper[2554]:  > containerID="9fe772a814c247059081d4bd0bec09c74fbdd661b4ae3d3a595ce3ef660b5b5f" cmd=["/bin/sleep","15"]
Sep 16 16:31:53.693101 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:31:53.692702    2554 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv" message="" err=<
Sep 16 16:31:53.693101 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:31:53.693101 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 16:31:53.693101 ip-10-0-136-228 kubenswrapper[2554]:  >
Sep 16 16:31:53.693101 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:31:53.692718    2554 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 16:31:53.693101 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:31:53.693101 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 16:31:53.693101 ip-10-0-136-228 kubenswrapper[2554]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv" podUID="9f2bbe5b-56de-4fc6-8174-fe8d61242d1c" containerName="main" containerID="cri-o://9fe772a814c247059081d4bd0bec09c74fbdd661b4ae3d3a595ce3ef660b5b5f"
Sep 16 16:31:53.693101 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:31:53.692758    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv" podUID="9f2bbe5b-56de-4fc6-8174-fe8d61242d1c" containerName="main" containerID="cri-o://9fe772a814c247059081d4bd0bec09c74fbdd661b4ae3d3a595ce3ef660b5b5f" gracePeriod=60
Sep 16 16:31:53.835309 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:31:53.835271    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:31:53.893175 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:31:53.893141    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:31:54.037558 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:31:54.037532    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv"
Sep 16 16:31:54.143016 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:31:54.142982    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f2bbe5b-56de-4fc6-8174-fe8d61242d1c-model-cache\") pod \"9f2bbe5b-56de-4fc6-8174-fe8d61242d1c\" (UID: \"9f2bbe5b-56de-4fc6-8174-fe8d61242d1c\") "
Sep 16 16:31:54.143016 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:31:54.143016    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f2bbe5b-56de-4fc6-8174-fe8d61242d1c-tmp-dir\") pod \"9f2bbe5b-56de-4fc6-8174-fe8d61242d1c\" (UID: \"9f2bbe5b-56de-4fc6-8174-fe8d61242d1c\") "
Sep 16 16:31:54.143276 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:31:54.143065    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f2bbe5b-56de-4fc6-8174-fe8d61242d1c-tls-certs\") pod \"9f2bbe5b-56de-4fc6-8174-fe8d61242d1c\" (UID: \"9f2bbe5b-56de-4fc6-8174-fe8d61242d1c\") "
Sep 16 16:31:54.143276 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:31:54.143083    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f2bbe5b-56de-4fc6-8174-fe8d61242d1c-dshm\") pod \"9f2bbe5b-56de-4fc6-8174-fe8d61242d1c\" (UID: \"9f2bbe5b-56de-4fc6-8174-fe8d61242d1c\") "
Sep 16 16:31:54.143276 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:31:54.143116    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78hrg\" (UniqueName: \"kubernetes.io/projected/9f2bbe5b-56de-4fc6-8174-fe8d61242d1c-kube-api-access-78hrg\") pod \"9f2bbe5b-56de-4fc6-8174-fe8d61242d1c\" (UID: \"9f2bbe5b-56de-4fc6-8174-fe8d61242d1c\") "
Sep 16 16:31:54.143276 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:31:54.143131    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f2bbe5b-56de-4fc6-8174-fe8d61242d1c-home\") pod \"9f2bbe5b-56de-4fc6-8174-fe8d61242d1c\" (UID: \"9f2bbe5b-56de-4fc6-8174-fe8d61242d1c\") "
Sep 16 16:31:54.143472 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:31:54.143350    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2bbe5b-56de-4fc6-8174-fe8d61242d1c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9f2bbe5b-56de-4fc6-8174-fe8d61242d1c" (UID: "9f2bbe5b-56de-4fc6-8174-fe8d61242d1c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:31:54.143472 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:31:54.143365    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2bbe5b-56de-4fc6-8174-fe8d61242d1c-model-cache" (OuterVolumeSpecName: "model-cache") pod "9f2bbe5b-56de-4fc6-8174-fe8d61242d1c" (UID: "9f2bbe5b-56de-4fc6-8174-fe8d61242d1c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:31:54.143574 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:31:54.143548    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2bbe5b-56de-4fc6-8174-fe8d61242d1c-home" (OuterVolumeSpecName: "home") pod "9f2bbe5b-56de-4fc6-8174-fe8d61242d1c" (UID: "9f2bbe5b-56de-4fc6-8174-fe8d61242d1c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:31:54.145468 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:31:54.145430    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2bbe5b-56de-4fc6-8174-fe8d61242d1c-dshm" (OuterVolumeSpecName: "dshm") pod "9f2bbe5b-56de-4fc6-8174-fe8d61242d1c" (UID: "9f2bbe5b-56de-4fc6-8174-fe8d61242d1c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:31:54.145589 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:31:54.145494    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f2bbe5b-56de-4fc6-8174-fe8d61242d1c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9f2bbe5b-56de-4fc6-8174-fe8d61242d1c" (UID: "9f2bbe5b-56de-4fc6-8174-fe8d61242d1c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:31:54.145589 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:31:54.145505    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f2bbe5b-56de-4fc6-8174-fe8d61242d1c-kube-api-access-78hrg" (OuterVolumeSpecName: "kube-api-access-78hrg") pod "9f2bbe5b-56de-4fc6-8174-fe8d61242d1c" (UID: "9f2bbe5b-56de-4fc6-8174-fe8d61242d1c"). InnerVolumeSpecName "kube-api-access-78hrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:31:54.244426 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:31:54.244318    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78hrg\" (UniqueName: \"kubernetes.io/projected/9f2bbe5b-56de-4fc6-8174-fe8d61242d1c-kube-api-access-78hrg\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:31:54.244426 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:31:54.244363    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f2bbe5b-56de-4fc6-8174-fe8d61242d1c-home\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:31:54.244426 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:31:54.244379    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f2bbe5b-56de-4fc6-8174-fe8d61242d1c-model-cache\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:31:54.244426 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:31:54.244394    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f2bbe5b-56de-4fc6-8174-fe8d61242d1c-tmp-dir\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:31:54.244426 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:31:54.244404    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f2bbe5b-56de-4fc6-8174-fe8d61242d1c-tls-certs\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:31:54.244426 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:31:54.244414    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f2bbe5b-56de-4fc6-8174-fe8d61242d1c-dshm\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:31:54.306865 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:31:54.306826    2554 generic.go:356] "Generic (PLEG): container finished" podID="9f2bbe5b-56de-4fc6-8174-fe8d61242d1c" containerID="9fe772a814c247059081d4bd0bec09c74fbdd661b4ae3d3a595ce3ef660b5b5f" exitCode=0
Sep 16 16:31:54.306865 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:31:54.306863    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv" event={"ID":"9f2bbe5b-56de-4fc6-8174-fe8d61242d1c","Type":"ContainerDied","Data":"9fe772a814c247059081d4bd0bec09c74fbdd661b4ae3d3a595ce3ef660b5b5f"}
Sep 16 16:31:54.307086 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:31:54.306878    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv"
Sep 16 16:31:54.307086 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:31:54.306899    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv" event={"ID":"9f2bbe5b-56de-4fc6-8174-fe8d61242d1c","Type":"ContainerDied","Data":"9715b40b7d369ae14fd8ffe01cf140baa570f0aa343031abfd0831626002c164"}
Sep 16 16:31:54.307086 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:31:54.306921    2554 scope.go:117] "RemoveContainer" containerID="9fe772a814c247059081d4bd0bec09c74fbdd661b4ae3d3a595ce3ef660b5b5f"
Sep 16 16:31:54.315808 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:31:54.315773    2554 scope.go:117] "RemoveContainer" containerID="9fe772a814c247059081d4bd0bec09c74fbdd661b4ae3d3a595ce3ef660b5b5f"
Sep 16 16:31:54.316101 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:31:54.316082    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fe772a814c247059081d4bd0bec09c74fbdd661b4ae3d3a595ce3ef660b5b5f\": container with ID starting with 9fe772a814c247059081d4bd0bec09c74fbdd661b4ae3d3a595ce3ef660b5b5f not found: ID does not exist" containerID="9fe772a814c247059081d4bd0bec09c74fbdd661b4ae3d3a595ce3ef660b5b5f"
Sep 16 16:31:54.316157 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:31:54.316110    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fe772a814c247059081d4bd0bec09c74fbdd661b4ae3d3a595ce3ef660b5b5f"} err="failed to get container status \"9fe772a814c247059081d4bd0bec09c74fbdd661b4ae3d3a595ce3ef660b5b5f\": rpc error: code = NotFound desc = could not find container \"9fe772a814c247059081d4bd0bec09c74fbdd661b4ae3d3a595ce3ef660b5b5f\": container with ID starting with 9fe772a814c247059081d4bd0bec09c74fbdd661b4ae3d3a595ce3ef660b5b5f not found: ID does not exist"
Sep 16 16:31:54.327586 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:31:54.327555    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv"]
Sep 16 16:31:54.332849 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:31:54.332823    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv"]
Sep 16 16:31:55.022787 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:31:55.022737    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-swwmv" podUID="9f2bbe5b-56de-4fc6-8174-fe8d61242d1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 16 16:31:56.075847 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:31:56.075817    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f2bbe5b-56de-4fc6-8174-fe8d61242d1c" path="/var/lib/kubelet/pods/9f2bbe5b-56de-4fc6-8174-fe8d61242d1c/volumes"
Sep 16 16:32:43.624234 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:32:43.624198    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:32:43.700335 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:32:43.700304    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:32:49.470065 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:32:49.470025    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:32:53.466720 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:32:53.466689    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8"]
Sep 16 16:32:53.467114 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:32:53.466958    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f2bbe5b-56de-4fc6-8174-fe8d61242d1c" containerName="main"
Sep 16 16:32:53.467114 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:32:53.466971    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f2bbe5b-56de-4fc6-8174-fe8d61242d1c" containerName="main"
Sep 16 16:32:53.467114 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:32:53.467029    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f2bbe5b-56de-4fc6-8174-fe8d61242d1c" containerName="main"
Sep 16 16:32:53.469660 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:32:53.469641    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8"
Sep 16 16:32:53.472506 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:32:53.472479    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 16 16:32:53.474069 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:32:53.474048    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 16 16:32:53.474173 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:32:53.474118    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 16 16:32:53.474173 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:32:53.474118    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-ph4nq\""
Sep 16 16:32:53.482937 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:32:53.482905    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8"]
Sep 16 16:32:53.546557 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:32:53.546519    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b6c1866-7918-4180-a719-db3bb8add488-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8\" (UID: \"7b6c1866-7918-4180-a719-db3bb8add488\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8"
Sep 16 16:32:53.546557 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:32:53.546561    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b6c1866-7918-4180-a719-db3bb8add488-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8\" (UID: \"7b6c1866-7918-4180-a719-db3bb8add488\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8"
Sep 16 16:32:53.546768 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:32:53.546580    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b6c1866-7918-4180-a719-db3bb8add488-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8\" (UID: \"7b6c1866-7918-4180-a719-db3bb8add488\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8"
Sep 16 16:32:53.546768 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:32:53.546682    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b6c1866-7918-4180-a719-db3bb8add488-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8\" (UID: \"7b6c1866-7918-4180-a719-db3bb8add488\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8"
Sep 16 16:32:53.546768 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:32:53.546716    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9h49\" (UniqueName: \"kubernetes.io/projected/7b6c1866-7918-4180-a719-db3bb8add488-kube-api-access-f9h49\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8\" (UID: \"7b6c1866-7918-4180-a719-db3bb8add488\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8"
Sep 16 16:32:53.546899 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:32:53.546772    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b6c1866-7918-4180-a719-db3bb8add488-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8\" (UID: \"7b6c1866-7918-4180-a719-db3bb8add488\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8"
Sep 16 16:32:53.647496 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:32:53.647458    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b6c1866-7918-4180-a719-db3bb8add488-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8\" (UID: \"7b6c1866-7918-4180-a719-db3bb8add488\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8"
Sep 16 16:32:53.647496 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:32:53.647508    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b6c1866-7918-4180-a719-db3bb8add488-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8\" (UID: \"7b6c1866-7918-4180-a719-db3bb8add488\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8"
Sep 16 16:32:53.647697 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:32:53.647528    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b6c1866-7918-4180-a719-db3bb8add488-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8\" (UID: \"7b6c1866-7918-4180-a719-db3bb8add488\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8"
Sep 16 16:32:53.647697 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:32:53.647633    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b6c1866-7918-4180-a719-db3bb8add488-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8\" (UID: \"7b6c1866-7918-4180-a719-db3bb8add488\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8"
Sep 16 16:32:53.647697 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:32:53.647692    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b6c1866-7918-4180-a719-db3bb8add488-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8\" (UID: \"7b6c1866-7918-4180-a719-db3bb8add488\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8"
Sep 16 16:32:53.647868 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:32:53.647726    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9h49\" (UniqueName: \"kubernetes.io/projected/7b6c1866-7918-4180-a719-db3bb8add488-kube-api-access-f9h49\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8\" (UID: \"7b6c1866-7918-4180-a719-db3bb8add488\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8"
Sep 16 16:32:53.647960 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:32:53.647939    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b6c1866-7918-4180-a719-db3bb8add488-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8\" (UID: \"7b6c1866-7918-4180-a719-db3bb8add488\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8"
Sep 16 16:32:53.648062 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:32:53.648041    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b6c1866-7918-4180-a719-db3bb8add488-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8\" (UID: \"7b6c1866-7918-4180-a719-db3bb8add488\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8"
Sep 16 16:32:53.648125 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:32:53.648051    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b6c1866-7918-4180-a719-db3bb8add488-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8\" (UID: \"7b6c1866-7918-4180-a719-db3bb8add488\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8"
Sep 16 16:32:53.649813 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:32:53.649781    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b6c1866-7918-4180-a719-db3bb8add488-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8\" (UID: \"7b6c1866-7918-4180-a719-db3bb8add488\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8"
Sep 16 16:32:53.650143 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:32:53.650124    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b6c1866-7918-4180-a719-db3bb8add488-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8\" (UID: \"7b6c1866-7918-4180-a719-db3bb8add488\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8"
Sep 16 16:32:53.656499 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:32:53.656480    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9h49\" (UniqueName: \"kubernetes.io/projected/7b6c1866-7918-4180-a719-db3bb8add488-kube-api-access-f9h49\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8\" (UID: \"7b6c1866-7918-4180-a719-db3bb8add488\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8"
Sep 16 16:32:53.781277 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:32:53.781180    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8"
Sep 16 16:32:54.050218 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:32:54.050189    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8"]
Sep 16 16:32:54.050320 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:32:54.050196    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:32:54.052775 ip-10-0-136-228 kubenswrapper[2554]: W0916 16:32:54.052747    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b6c1866_7918_4180_a719_db3bb8add488.slice/crio-7a97240d59d550385fac00077afa644b33b3bfdb603e409c16d22ab6f7f239cd WatchSource:0}: Error finding container 7a97240d59d550385fac00077afa644b33b3bfdb603e409c16d22ab6f7f239cd: Status 404 returned error can't find the container with id 7a97240d59d550385fac00077afa644b33b3bfdb603e409c16d22ab6f7f239cd
Sep 16 16:32:54.165523 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:32:54.165489    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:32:54.524784 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:32:54.524689    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8" event={"ID":"7b6c1866-7918-4180-a719-db3bb8add488","Type":"ContainerStarted","Data":"e14b7fbf6fe79f03819c9df1da6a4030f493c4d9a7de794411766d984b360895"}
Sep 16 16:32:54.524784 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:32:54.524761    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8" event={"ID":"7b6c1866-7918-4180-a719-db3bb8add488","Type":"ContainerStarted","Data":"7a97240d59d550385fac00077afa644b33b3bfdb603e409c16d22ab6f7f239cd"}
Sep 16 16:32:54.542435 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:32:54.542386    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8" podStartSLOduration=1.542369845 podStartE2EDuration="1.542369845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:32:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:32:54.541112694 +0000 UTC m=+2574.985804910" watchObservedRunningTime="2026-09-16 16:32:54.542369845 +0000 UTC m=+2574.987062016"
Sep 16 16:32:54.782043 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:32:54.782014    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8"
Sep 16 16:32:55.607351 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:32:55.607302    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:32:55.676183 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:32:55.676151    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:33:01.860311 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:33:01.860278    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:33:03.782067 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:33:03.782027    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8"
Sep 16 16:33:03.794365 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:33:03.794335    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8"
Sep 16 16:33:04.571779 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:33:04.571745    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8"
Sep 16 16:33:38.595416 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:33:38.595341    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:33:38.850418 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:33:38.850331    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:33:53.588868 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:33:53.588827    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:34:18.482430 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:34:18.482392    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:34:20.075725 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:34:20.075687    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:34:32.367313 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:34:32.367276    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:34:32.471726 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:34:32.471686    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:34:32.573501 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:34:32.573469    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:34:34.776132 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:34:34.776101    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:34:35.473700 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:34:35.473668    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:34:36.172322 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:34:36.172287    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:34:37.442236 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:34:37.442202    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:35:04.059655 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:35:04.059612    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:35:11.714629 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:35:11.714594    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8"]
Sep 16 16:35:11.724422 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:35:11.724384    2554 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 16:35:11.724422 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:35:11.724422 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 16:35:11.724422 ip-10-0-136-228 kubenswrapper[2554]:  > containerID="e14b7fbf6fe79f03819c9df1da6a4030f493c4d9a7de794411766d984b360895" cmd=["/bin/sleep","15"]
Sep 16 16:35:11.724847 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:35:11.724443    2554 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8" message="" err=<
Sep 16 16:35:11.724847 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:35:11.724847 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 16:35:11.724847 ip-10-0-136-228 kubenswrapper[2554]:  >
Sep 16 16:35:11.724847 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:35:11.724459    2554 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 16:35:11.724847 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:35:11.724847 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 16:35:11.724847 ip-10-0-136-228 kubenswrapper[2554]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8" podUID="7b6c1866-7918-4180-a719-db3bb8add488" containerName="main" containerID="cri-o://e14b7fbf6fe79f03819c9df1da6a4030f493c4d9a7de794411766d984b360895"
Sep 16 16:35:11.724847 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:35:11.724510    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8" podUID="7b6c1866-7918-4180-a719-db3bb8add488" containerName="main" containerID="cri-o://e14b7fbf6fe79f03819c9df1da6a4030f493c4d9a7de794411766d984b360895" gracePeriod=60
Sep 16 16:35:11.724847 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:35:11.724809    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:35:11.781733 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:35:11.781697    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:35:12.035559 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:35:12.035528    2554 generic.go:356] "Generic (PLEG): container finished" podID="7b6c1866-7918-4180-a719-db3bb8add488" containerID="e14b7fbf6fe79f03819c9df1da6a4030f493c4d9a7de794411766d984b360895" exitCode=0
Sep 16 16:35:12.035716 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:35:12.035574    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8" event={"ID":"7b6c1866-7918-4180-a719-db3bb8add488","Type":"ContainerDied","Data":"e14b7fbf6fe79f03819c9df1da6a4030f493c4d9a7de794411766d984b360895"}
Sep 16 16:35:12.062859 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:35:12.062830    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8"
Sep 16 16:35:12.192293 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:35:12.192253    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b6c1866-7918-4180-a719-db3bb8add488-tls-certs\") pod \"7b6c1866-7918-4180-a719-db3bb8add488\" (UID: \"7b6c1866-7918-4180-a719-db3bb8add488\") "
Sep 16 16:35:12.192293 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:35:12.192295    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b6c1866-7918-4180-a719-db3bb8add488-home\") pod \"7b6c1866-7918-4180-a719-db3bb8add488\" (UID: \"7b6c1866-7918-4180-a719-db3bb8add488\") "
Sep 16 16:35:12.192568 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:35:12.192336    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b6c1866-7918-4180-a719-db3bb8add488-dshm\") pod \"7b6c1866-7918-4180-a719-db3bb8add488\" (UID: \"7b6c1866-7918-4180-a719-db3bb8add488\") "
Sep 16 16:35:12.192568 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:35:12.192361    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9h49\" (UniqueName: \"kubernetes.io/projected/7b6c1866-7918-4180-a719-db3bb8add488-kube-api-access-f9h49\") pod \"7b6c1866-7918-4180-a719-db3bb8add488\" (UID: \"7b6c1866-7918-4180-a719-db3bb8add488\") "
Sep 16 16:35:12.192568 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:35:12.192393    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b6c1866-7918-4180-a719-db3bb8add488-model-cache\") pod \"7b6c1866-7918-4180-a719-db3bb8add488\" (UID: \"7b6c1866-7918-4180-a719-db3bb8add488\") "
Sep 16 16:35:12.192568 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:35:12.192417    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b6c1866-7918-4180-a719-db3bb8add488-tmp-dir\") pod \"7b6c1866-7918-4180-a719-db3bb8add488\" (UID: \"7b6c1866-7918-4180-a719-db3bb8add488\") "
Sep 16 16:35:12.192785 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:35:12.192653    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b6c1866-7918-4180-a719-db3bb8add488-home" (OuterVolumeSpecName: "home") pod "7b6c1866-7918-4180-a719-db3bb8add488" (UID: "7b6c1866-7918-4180-a719-db3bb8add488"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:35:12.192785 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:35:12.192678    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b6c1866-7918-4180-a719-db3bb8add488-model-cache" (OuterVolumeSpecName: "model-cache") pod "7b6c1866-7918-4180-a719-db3bb8add488" (UID: "7b6c1866-7918-4180-a719-db3bb8add488"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:35:12.192914 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:35:12.192831    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b6c1866-7918-4180-a719-db3bb8add488-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7b6c1866-7918-4180-a719-db3bb8add488" (UID: "7b6c1866-7918-4180-a719-db3bb8add488"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:35:12.194614 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:35:12.194588    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b6c1866-7918-4180-a719-db3bb8add488-dshm" (OuterVolumeSpecName: "dshm") pod "7b6c1866-7918-4180-a719-db3bb8add488" (UID: "7b6c1866-7918-4180-a719-db3bb8add488"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:35:12.194714 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:35:12.194655    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b6c1866-7918-4180-a719-db3bb8add488-kube-api-access-f9h49" (OuterVolumeSpecName: "kube-api-access-f9h49") pod "7b6c1866-7918-4180-a719-db3bb8add488" (UID: "7b6c1866-7918-4180-a719-db3bb8add488"). InnerVolumeSpecName "kube-api-access-f9h49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:35:12.194714 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:35:12.194664    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b6c1866-7918-4180-a719-db3bb8add488-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7b6c1866-7918-4180-a719-db3bb8add488" (UID: "7b6c1866-7918-4180-a719-db3bb8add488"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:35:12.293218 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:35:12.293192    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b6c1866-7918-4180-a719-db3bb8add488-dshm\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:35:12.293218 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:35:12.293218    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9h49\" (UniqueName: \"kubernetes.io/projected/7b6c1866-7918-4180-a719-db3bb8add488-kube-api-access-f9h49\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:35:12.293388 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:35:12.293229    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b6c1866-7918-4180-a719-db3bb8add488-model-cache\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:35:12.293388 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:35:12.293257    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b6c1866-7918-4180-a719-db3bb8add488-tmp-dir\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:35:12.293388 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:35:12.293265    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b6c1866-7918-4180-a719-db3bb8add488-tls-certs\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:35:12.293388 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:35:12.293273    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b6c1866-7918-4180-a719-db3bb8add488-home\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:35:13.040154 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:35:13.040122    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8"
Sep 16 16:35:13.040154 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:35:13.040137    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8" event={"ID":"7b6c1866-7918-4180-a719-db3bb8add488","Type":"ContainerDied","Data":"7a97240d59d550385fac00077afa644b33b3bfdb603e409c16d22ab6f7f239cd"}
Sep 16 16:35:13.040714 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:35:13.040201    2554 scope.go:117] "RemoveContainer" containerID="e14b7fbf6fe79f03819c9df1da6a4030f493c4d9a7de794411766d984b360895"
Sep 16 16:35:13.060710 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:35:13.060683    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8"]
Sep 16 16:35:13.064205 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:35:13.064181    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-t6nz8"]
Sep 16 16:35:14.076112 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:35:14.076070    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b6c1866-7918-4180-a719-db3bb8add488" path="/var/lib/kubelet/pods/7b6c1866-7918-4180-a719-db3bb8add488/volumes"
Sep 16 16:35:14.700658 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:35:14.700625    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:35:20.495617 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:35:20.495579    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:35:20.596668 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:35:20.596623    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:35:30.606629 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:35:30.606593    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:35:57.970237 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:35:57.970198    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:35:58.065728 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:35:58.065685    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:36:03.319417 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:03.319375    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:36:04.363243 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:04.363205    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:36:08.679028 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:08.678991    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:36:08.780626 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:08.780594    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:36:08.884193 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:08.884158    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:36:10.700850 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:10.700816    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:36:10.801884 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:10.801848    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:36:12.119654 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:12.119618    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:36:13.696595 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:13.696557    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:36:20.943753 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:20.943713    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:36:21.271678 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:21.271594    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l"]
Sep 16 16:36:21.271869 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:21.271855    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b6c1866-7918-4180-a719-db3bb8add488" containerName="main"
Sep 16 16:36:21.271925 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:21.271870    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b6c1866-7918-4180-a719-db3bb8add488" containerName="main"
Sep 16 16:36:21.271925 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:21.271923    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b6c1866-7918-4180-a719-db3bb8add488" containerName="main"
Sep 16 16:36:21.274626 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:21.274609    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l"
Sep 16 16:36:21.277601 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:21.277573    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 16 16:36:21.277601 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:21.277592    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 16 16:36:21.278871 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:21.278841    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-27sbj\""
Sep 16 16:36:21.279099 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:21.279083    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 16 16:36:21.284742 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:21.284718    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l"]
Sep 16 16:36:21.305308 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:21.305280    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:36:21.417878 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:21.417823    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83a192cf-f6d5-4972-96fb-7c4ade67a0d5-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l\" (UID: \"83a192cf-f6d5-4972-96fb-7c4ade67a0d5\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l"
Sep 16 16:36:21.418058 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:21.417897    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83a192cf-f6d5-4972-96fb-7c4ade67a0d5-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l\" (UID: \"83a192cf-f6d5-4972-96fb-7c4ade67a0d5\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l"
Sep 16 16:36:21.418058 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:21.417944    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83a192cf-f6d5-4972-96fb-7c4ade67a0d5-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l\" (UID: \"83a192cf-f6d5-4972-96fb-7c4ade67a0d5\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l"
Sep 16 16:36:21.418058 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:21.417966    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z49d\" (UniqueName: \"kubernetes.io/projected/83a192cf-f6d5-4972-96fb-7c4ade67a0d5-kube-api-access-8z49d\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l\" (UID: \"83a192cf-f6d5-4972-96fb-7c4ade67a0d5\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l"
Sep 16 16:36:21.418058 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:21.418004    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83a192cf-f6d5-4972-96fb-7c4ade67a0d5-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l\" (UID: \"83a192cf-f6d5-4972-96fb-7c4ade67a0d5\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l"
Sep 16 16:36:21.418058 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:21.418019    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83a192cf-f6d5-4972-96fb-7c4ade67a0d5-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l\" (UID: \"83a192cf-f6d5-4972-96fb-7c4ade67a0d5\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l"
Sep 16 16:36:21.519053 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:21.519009    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83a192cf-f6d5-4972-96fb-7c4ade67a0d5-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l\" (UID: \"83a192cf-f6d5-4972-96fb-7c4ade67a0d5\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l"
Sep 16 16:36:21.519053 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:21.519047    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z49d\" (UniqueName: \"kubernetes.io/projected/83a192cf-f6d5-4972-96fb-7c4ade67a0d5-kube-api-access-8z49d\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l\" (UID: \"83a192cf-f6d5-4972-96fb-7c4ade67a0d5\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l"
Sep 16 16:36:21.519288 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:21.519071    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83a192cf-f6d5-4972-96fb-7c4ade67a0d5-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l\" (UID: \"83a192cf-f6d5-4972-96fb-7c4ade67a0d5\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l"
Sep 16 16:36:21.519288 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:21.519179    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83a192cf-f6d5-4972-96fb-7c4ade67a0d5-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l\" (UID: \"83a192cf-f6d5-4972-96fb-7c4ade67a0d5\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l"
Sep 16 16:36:21.519288 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:21.519225    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83a192cf-f6d5-4972-96fb-7c4ade67a0d5-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l\" (UID: \"83a192cf-f6d5-4972-96fb-7c4ade67a0d5\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l"
Sep 16 16:36:21.519288 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:21.519265    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83a192cf-f6d5-4972-96fb-7c4ade67a0d5-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l\" (UID: \"83a192cf-f6d5-4972-96fb-7c4ade67a0d5\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l"
Sep 16 16:36:21.519548 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:21.519490    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83a192cf-f6d5-4972-96fb-7c4ade67a0d5-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l\" (UID: \"83a192cf-f6d5-4972-96fb-7c4ade67a0d5\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l"
Sep 16 16:36:21.519878 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:21.519850    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83a192cf-f6d5-4972-96fb-7c4ade67a0d5-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l\" (UID: \"83a192cf-f6d5-4972-96fb-7c4ade67a0d5\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l"
Sep 16 16:36:21.520001 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:21.519927    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83a192cf-f6d5-4972-96fb-7c4ade67a0d5-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l\" (UID: \"83a192cf-f6d5-4972-96fb-7c4ade67a0d5\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l"
Sep 16 16:36:21.521922 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:21.521869    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83a192cf-f6d5-4972-96fb-7c4ade67a0d5-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l\" (UID: \"83a192cf-f6d5-4972-96fb-7c4ade67a0d5\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l"
Sep 16 16:36:21.522030 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:21.522011    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83a192cf-f6d5-4972-96fb-7c4ade67a0d5-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l\" (UID: \"83a192cf-f6d5-4972-96fb-7c4ade67a0d5\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l"
Sep 16 16:36:21.532086 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:21.532052    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z49d\" (UniqueName: \"kubernetes.io/projected/83a192cf-f6d5-4972-96fb-7c4ade67a0d5-kube-api-access-8z49d\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l\" (UID: \"83a192cf-f6d5-4972-96fb-7c4ade67a0d5\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l"
Sep 16 16:36:21.563936 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:21.563900    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:36:21.585965 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:21.585931    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l"
Sep 16 16:36:21.856015 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:21.855988    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l"]
Sep 16 16:36:21.857940 ip-10-0-136-228 kubenswrapper[2554]: W0916 16:36:21.857909    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a192cf_f6d5_4972_96fb_7c4ade67a0d5.slice/crio-afffa3ee9f4ec4b69ccd1c43aeb3cfea9640efdcdd206d11073aee0b4d72edac WatchSource:0}: Error finding container afffa3ee9f4ec4b69ccd1c43aeb3cfea9640efdcdd206d11073aee0b4d72edac: Status 404 returned error can't find the container with id afffa3ee9f4ec4b69ccd1c43aeb3cfea9640efdcdd206d11073aee0b4d72edac
Sep 16 16:36:22.288733 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:22.288700    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l" event={"ID":"83a192cf-f6d5-4972-96fb-7c4ade67a0d5","Type":"ContainerStarted","Data":"1ab82c18a4486a8bd3cd04e6bb4cd980a70d10b70ed72133c59d2e89a80732ed"}
Sep 16 16:36:22.289123 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:22.288753    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l" event={"ID":"83a192cf-f6d5-4972-96fb-7c4ade67a0d5","Type":"ContainerStarted","Data":"afffa3ee9f4ec4b69ccd1c43aeb3cfea9640efdcdd206d11073aee0b4d72edac"}
Sep 16 16:36:22.308420 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:22.308370    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l" podStartSLOduration=1.3083528389999999 podStartE2EDuration="1.308352839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:36:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:36:22.307201863 +0000 UTC m=+2782.751894087" watchObservedRunningTime="2026-09-16 16:36:22.308352839 +0000 UTC m=+2782.753045016"
Sep 16 16:36:22.586242 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:22.586158    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l"
Sep 16 16:36:30.182511 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:30.182476    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:36:31.586505 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:31.586460    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l"
Sep 16 16:36:31.603585 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:31.603550    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l"
Sep 16 16:36:31.875052 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:31.874964    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:36:32.336315 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:32.336284    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l"
Sep 16 16:36:32.947677 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:32.947641    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:36:37.097812 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:37.097767    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:36:43.151237 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:43.151206    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:36:50.855211 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:50.855170    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:36:56.505730 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:56.505684    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:36:56.576288 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:36:56.576256    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:37:26.570771 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:37:26.570734    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:37:26.758398 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:37:26.758347    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:37:31.783883 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:37:31.783832    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:37:36.332787 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:37:36.332748    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:37:36.397478 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:37:36.397444    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:37:47.493936 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:37:47.493894    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:37:47.643749 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:37:47.643708    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:37:47.730588 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:37:47.730561    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:37:52.517920 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:37:52.517837    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:37:57.519459 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:37:57.519420    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:37:59.174329 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:37:59.174289    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:38:09.598108 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:38:09.598078    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:38:26.960177 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:38:26.960137    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:38:28.211692 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:38:28.211658    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:38:28.307623 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:38:28.307591    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:38:30.076925 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:38:30.076887    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:38:32.397446 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:38:32.397415    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:38:32.451423 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:38:32.451383    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:38:36.101091 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:38:36.101057    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:38:36.175878 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:38:36.175846    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:39:21.739347 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:39:21.739303    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l"]
Sep 16 16:39:21.796178 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:39:21.796151    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:39:21.949268 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:39:21.949226    2554 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 16:39:21.949268 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:39:21.949268 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 16:39:21.949268 ip-10-0-136-228 kubenswrapper[2554]:  > containerID="1ab82c18a4486a8bd3cd04e6bb4cd980a70d10b70ed72133c59d2e89a80732ed" cmd=["/bin/sleep","15"]
Sep 16 16:39:21.949541 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:39:21.949274    2554 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l" message="" err=<
Sep 16 16:39:21.949541 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:39:21.949541 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 16:39:21.949541 ip-10-0-136-228 kubenswrapper[2554]:  >
Sep 16 16:39:21.949541 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:39:21.949286    2554 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 16:39:21.949541 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:39:21.949541 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 16:39:21.949541 ip-10-0-136-228 kubenswrapper[2554]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l" podUID="83a192cf-f6d5-4972-96fb-7c4ade67a0d5" containerName="main" containerID="cri-o://1ab82c18a4486a8bd3cd04e6bb4cd980a70d10b70ed72133c59d2e89a80732ed"
Sep 16 16:39:21.949541 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:39:21.949317    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l" podUID="83a192cf-f6d5-4972-96fb-7c4ade67a0d5" containerName="main" containerID="cri-o://1ab82c18a4486a8bd3cd04e6bb4cd980a70d10b70ed72133c59d2e89a80732ed" gracePeriod=60
Sep 16 16:39:22.285237 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:39:22.285211    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l"
Sep 16 16:39:22.359518 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:39:22.359445    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83a192cf-f6d5-4972-96fb-7c4ade67a0d5-tmp-dir\") pod \"83a192cf-f6d5-4972-96fb-7c4ade67a0d5\" (UID: \"83a192cf-f6d5-4972-96fb-7c4ade67a0d5\") "
Sep 16 16:39:22.359518 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:39:22.359478    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83a192cf-f6d5-4972-96fb-7c4ade67a0d5-model-cache\") pod \"83a192cf-f6d5-4972-96fb-7c4ade67a0d5\" (UID: \"83a192cf-f6d5-4972-96fb-7c4ade67a0d5\") "
Sep 16 16:39:22.359518 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:39:22.359501    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8z49d\" (UniqueName: \"kubernetes.io/projected/83a192cf-f6d5-4972-96fb-7c4ade67a0d5-kube-api-access-8z49d\") pod \"83a192cf-f6d5-4972-96fb-7c4ade67a0d5\" (UID: \"83a192cf-f6d5-4972-96fb-7c4ade67a0d5\") "
Sep 16 16:39:22.359783 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:39:22.359528    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83a192cf-f6d5-4972-96fb-7c4ade67a0d5-tls-certs\") pod \"83a192cf-f6d5-4972-96fb-7c4ade67a0d5\" (UID: \"83a192cf-f6d5-4972-96fb-7c4ade67a0d5\") "
Sep 16 16:39:22.359783 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:39:22.359656    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83a192cf-f6d5-4972-96fb-7c4ade67a0d5-home\") pod \"83a192cf-f6d5-4972-96fb-7c4ade67a0d5\" (UID: \"83a192cf-f6d5-4972-96fb-7c4ade67a0d5\") "
Sep 16 16:39:22.359783 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:39:22.359688    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83a192cf-f6d5-4972-96fb-7c4ade67a0d5-dshm\") pod \"83a192cf-f6d5-4972-96fb-7c4ade67a0d5\" (UID: \"83a192cf-f6d5-4972-96fb-7c4ade67a0d5\") "
Sep 16 16:39:22.359783 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:39:22.359733    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a192cf-f6d5-4972-96fb-7c4ade67a0d5-model-cache" (OuterVolumeSpecName: "model-cache") pod "83a192cf-f6d5-4972-96fb-7c4ade67a0d5" (UID: "83a192cf-f6d5-4972-96fb-7c4ade67a0d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:39:22.359783 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:39:22.359763    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a192cf-f6d5-4972-96fb-7c4ade67a0d5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "83a192cf-f6d5-4972-96fb-7c4ade67a0d5" (UID: "83a192cf-f6d5-4972-96fb-7c4ade67a0d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:39:22.360057 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:39:22.359915    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a192cf-f6d5-4972-96fb-7c4ade67a0d5-home" (OuterVolumeSpecName: "home") pod "83a192cf-f6d5-4972-96fb-7c4ade67a0d5" (UID: "83a192cf-f6d5-4972-96fb-7c4ade67a0d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:39:22.360057 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:39:22.359990    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83a192cf-f6d5-4972-96fb-7c4ade67a0d5-tmp-dir\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:39:22.360057 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:39:22.360011    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83a192cf-f6d5-4972-96fb-7c4ade67a0d5-model-cache\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:39:22.361765 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:39:22.361739    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83a192cf-f6d5-4972-96fb-7c4ade67a0d5-kube-api-access-8z49d" (OuterVolumeSpecName: "kube-api-access-8z49d") pod "83a192cf-f6d5-4972-96fb-7c4ade67a0d5" (UID: "83a192cf-f6d5-4972-96fb-7c4ade67a0d5"). InnerVolumeSpecName "kube-api-access-8z49d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:39:22.361846 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:39:22.361773    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a192cf-f6d5-4972-96fb-7c4ade67a0d5-dshm" (OuterVolumeSpecName: "dshm") pod "83a192cf-f6d5-4972-96fb-7c4ade67a0d5" (UID: "83a192cf-f6d5-4972-96fb-7c4ade67a0d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:39:22.361886 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:39:22.361854    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83a192cf-f6d5-4972-96fb-7c4ade67a0d5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "83a192cf-f6d5-4972-96fb-7c4ade67a0d5" (UID: "83a192cf-f6d5-4972-96fb-7c4ade67a0d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:39:22.460643 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:39:22.460611    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z49d\" (UniqueName: \"kubernetes.io/projected/83a192cf-f6d5-4972-96fb-7c4ade67a0d5-kube-api-access-8z49d\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:39:22.460643 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:39:22.460641    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83a192cf-f6d5-4972-96fb-7c4ade67a0d5-tls-certs\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:39:22.460850 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:39:22.460651    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83a192cf-f6d5-4972-96fb-7c4ade67a0d5-home\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:39:22.460850 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:39:22.460659    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83a192cf-f6d5-4972-96fb-7c4ade67a0d5-dshm\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:39:22.950260 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:39:22.950222    2554 generic.go:356] "Generic (PLEG): container finished" podID="83a192cf-f6d5-4972-96fb-7c4ade67a0d5" containerID="1ab82c18a4486a8bd3cd04e6bb4cd980a70d10b70ed72133c59d2e89a80732ed" exitCode=0
Sep 16 16:39:22.950735 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:39:22.950284    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l" event={"ID":"83a192cf-f6d5-4972-96fb-7c4ade67a0d5","Type":"ContainerDied","Data":"1ab82c18a4486a8bd3cd04e6bb4cd980a70d10b70ed72133c59d2e89a80732ed"}
Sep 16 16:39:22.950735 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:39:22.950286    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l"
Sep 16 16:39:22.950735 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:39:22.950346    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l" event={"ID":"83a192cf-f6d5-4972-96fb-7c4ade67a0d5","Type":"ContainerDied","Data":"afffa3ee9f4ec4b69ccd1c43aeb3cfea9640efdcdd206d11073aee0b4d72edac"}
Sep 16 16:39:22.950735 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:39:22.950368    2554 scope.go:117] "RemoveContainer" containerID="1ab82c18a4486a8bd3cd04e6bb4cd980a70d10b70ed72133c59d2e89a80732ed"
Sep 16 16:39:22.959359 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:39:22.959340    2554 scope.go:117] "RemoveContainer" containerID="1ab82c18a4486a8bd3cd04e6bb4cd980a70d10b70ed72133c59d2e89a80732ed"
Sep 16 16:39:22.959633 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:39:22.959613    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ab82c18a4486a8bd3cd04e6bb4cd980a70d10b70ed72133c59d2e89a80732ed\": container with ID starting with 1ab82c18a4486a8bd3cd04e6bb4cd980a70d10b70ed72133c59d2e89a80732ed not found: ID does not exist" containerID="1ab82c18a4486a8bd3cd04e6bb4cd980a70d10b70ed72133c59d2e89a80732ed"
Sep 16 16:39:22.959677 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:39:22.959641    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ab82c18a4486a8bd3cd04e6bb4cd980a70d10b70ed72133c59d2e89a80732ed"} err="failed to get container status \"1ab82c18a4486a8bd3cd04e6bb4cd980a70d10b70ed72133c59d2e89a80732ed\": rpc error: code = NotFound desc = could not find container \"1ab82c18a4486a8bd3cd04e6bb4cd980a70d10b70ed72133c59d2e89a80732ed\": container with ID starting with 1ab82c18a4486a8bd3cd04e6bb4cd980a70d10b70ed72133c59d2e89a80732ed not found: ID does not exist"
Sep 16 16:39:22.971815 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:39:22.971777    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l"]
Sep 16 16:39:22.974913 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:39:22.974892    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699kh9l"]
Sep 16 16:39:23.186755 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:39:23.186722    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:39:24.076333 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:39:24.076304    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83a192cf-f6d5-4972-96fb-7c4ade67a0d5" path="/var/lib/kubelet/pods/83a192cf-f6d5-4972-96fb-7c4ade67a0d5/volumes"
Sep 16 16:40:06.676606 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:40:06.676573    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:40:06.776402 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:40:06.776363    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:40:07.089387 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:40:07.089360    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:40:08.186645 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:40:08.186607    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:40:08.314176 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:40:08.314141    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:40:11.651696 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:40:11.651656    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:40:14.687590 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:40:14.687556    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:40:20.626601 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:40:20.626565    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:40:20.726524 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:40:20.726496    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:40:27.720014 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:40:27.719983    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:40:48.816654 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:40:48.816621    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:40:52.918678 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:40:52.918594    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:40:55.434545 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:40:55.434515    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:40:55.535025 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:40:55.534995    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:41:06.384848 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:06.384812    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx"]
Sep 16 16:41:06.387181 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:06.385058    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83a192cf-f6d5-4972-96fb-7c4ade67a0d5" containerName="main"
Sep 16 16:41:06.387181 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:06.385069    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="83a192cf-f6d5-4972-96fb-7c4ade67a0d5" containerName="main"
Sep 16 16:41:06.387181 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:06.385124    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="83a192cf-f6d5-4972-96fb-7c4ade67a0d5" containerName="main"
Sep 16 16:41:06.388076 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:06.388058    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx"
Sep 16 16:41:06.391331 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:06.391301    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 16 16:41:06.391475 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:06.391301    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 16 16:41:06.391475 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:06.391301    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 16 16:41:06.391475 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:06.391405    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-t749t\""
Sep 16 16:41:06.391475 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:06.391317    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 16:41:06.391475 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:06.391319    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Sep 16 16:41:06.401308 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:06.401286    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx"]
Sep 16 16:41:06.412212 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:06.412189    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:41:06.502586 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:06.502546    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49ccf442-28a0-4b70-9533-b83e894bc580-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx\" (UID: \"49ccf442-28a0-4b70-9533-b83e894bc580\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx"
Sep 16 16:41:06.502586 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:06.502589    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49ccf442-28a0-4b70-9533-b83e894bc580-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx\" (UID: \"49ccf442-28a0-4b70-9533-b83e894bc580\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx"
Sep 16 16:41:06.502835 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:06.502617    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49ccf442-28a0-4b70-9533-b83e894bc580-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx\" (UID: \"49ccf442-28a0-4b70-9533-b83e894bc580\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx"
Sep 16 16:41:06.502835 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:06.502682    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49ccf442-28a0-4b70-9533-b83e894bc580-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx\" (UID: \"49ccf442-28a0-4b70-9533-b83e894bc580\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx"
Sep 16 16:41:06.502835 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:06.502737    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49ccf442-28a0-4b70-9533-b83e894bc580-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx\" (UID: \"49ccf442-28a0-4b70-9533-b83e894bc580\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx"
Sep 16 16:41:06.502835 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:06.502767    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgztk\" (UniqueName: \"kubernetes.io/projected/49ccf442-28a0-4b70-9533-b83e894bc580-kube-api-access-cgztk\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx\" (UID: \"49ccf442-28a0-4b70-9533-b83e894bc580\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx"
Sep 16 16:41:06.502998 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:06.502858    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49ccf442-28a0-4b70-9533-b83e894bc580-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx\" (UID: \"49ccf442-28a0-4b70-9533-b83e894bc580\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx"
Sep 16 16:41:06.502998 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:06.502900    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49ccf442-28a0-4b70-9533-b83e894bc580-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx\" (UID: \"49ccf442-28a0-4b70-9533-b83e894bc580\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx"
Sep 16 16:41:06.603880 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:06.603845    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49ccf442-28a0-4b70-9533-b83e894bc580-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx\" (UID: \"49ccf442-28a0-4b70-9533-b83e894bc580\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx"
Sep 16 16:41:06.604003 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:06.603900    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49ccf442-28a0-4b70-9533-b83e894bc580-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx\" (UID: \"49ccf442-28a0-4b70-9533-b83e894bc580\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx"
Sep 16 16:41:06.604003 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:06.603937    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49ccf442-28a0-4b70-9533-b83e894bc580-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx\" (UID: \"49ccf442-28a0-4b70-9533-b83e894bc580\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx"
Sep 16 16:41:06.604003 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:06.603964    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49ccf442-28a0-4b70-9533-b83e894bc580-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx\" (UID: \"49ccf442-28a0-4b70-9533-b83e894bc580\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx"
Sep 16 16:41:06.604141 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:06.604052    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49ccf442-28a0-4b70-9533-b83e894bc580-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx\" (UID: \"49ccf442-28a0-4b70-9533-b83e894bc580\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx"
Sep 16 16:41:06.604141 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:06.604089    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49ccf442-28a0-4b70-9533-b83e894bc580-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx\" (UID: \"49ccf442-28a0-4b70-9533-b83e894bc580\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx"
Sep 16 16:41:06.604141 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:06.604112    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49ccf442-28a0-4b70-9533-b83e894bc580-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx\" (UID: \"49ccf442-28a0-4b70-9533-b83e894bc580\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx"
Sep 16 16:41:06.604301 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:06.604139    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgztk\" (UniqueName: \"kubernetes.io/projected/49ccf442-28a0-4b70-9533-b83e894bc580-kube-api-access-cgztk\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx\" (UID: \"49ccf442-28a0-4b70-9533-b83e894bc580\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx"
Sep 16 16:41:06.604365 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:06.604325    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49ccf442-28a0-4b70-9533-b83e894bc580-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx\" (UID: \"49ccf442-28a0-4b70-9533-b83e894bc580\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx"
Sep 16 16:41:06.604365 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:06.604349    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49ccf442-28a0-4b70-9533-b83e894bc580-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx\" (UID: \"49ccf442-28a0-4b70-9533-b83e894bc580\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx"
Sep 16 16:41:06.604472 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:06.604406    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49ccf442-28a0-4b70-9533-b83e894bc580-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx\" (UID: \"49ccf442-28a0-4b70-9533-b83e894bc580\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx"
Sep 16 16:41:06.604528 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:06.604473    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49ccf442-28a0-4b70-9533-b83e894bc580-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx\" (UID: \"49ccf442-28a0-4b70-9533-b83e894bc580\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx"
Sep 16 16:41:06.604775 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:06.604743    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49ccf442-28a0-4b70-9533-b83e894bc580-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx\" (UID: \"49ccf442-28a0-4b70-9533-b83e894bc580\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx"
Sep 16 16:41:06.606326 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:06.606298    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49ccf442-28a0-4b70-9533-b83e894bc580-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx\" (UID: \"49ccf442-28a0-4b70-9533-b83e894bc580\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx"
Sep 16 16:41:06.606405 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:06.606393    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49ccf442-28a0-4b70-9533-b83e894bc580-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx\" (UID: \"49ccf442-28a0-4b70-9533-b83e894bc580\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx"
Sep 16 16:41:06.614340 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:06.614316    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgztk\" (UniqueName: \"kubernetes.io/projected/49ccf442-28a0-4b70-9533-b83e894bc580-kube-api-access-cgztk\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx\" (UID: \"49ccf442-28a0-4b70-9533-b83e894bc580\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx"
Sep 16 16:41:06.686316 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:06.686241    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:41:06.698346 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:06.698318    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx"
Sep 16 16:41:06.977678 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:06.977647    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx"]
Sep 16 16:41:06.979551 ip-10-0-136-228 kubenswrapper[2554]: W0916 16:41:06.979527    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ccf442_28a0_4b70_9533_b83e894bc580.slice/crio-fe67e237544529b140aab73c0748b2b82fcefff11a4cd8ab1fada5f9242173d4 WatchSource:0}: Error finding container fe67e237544529b140aab73c0748b2b82fcefff11a4cd8ab1fada5f9242173d4: Status 404 returned error can't find the container with id fe67e237544529b140aab73c0748b2b82fcefff11a4cd8ab1fada5f9242173d4
Sep 16 16:41:07.323350 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:07.323311    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx" event={"ID":"49ccf442-28a0-4b70-9533-b83e894bc580","Type":"ContainerStarted","Data":"14f1c1ba7a3505fca193d11826cf47724225e6000b412ec0dfd63aa08cf29a76"}
Sep 16 16:41:07.323548 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:07.323370    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx" event={"ID":"49ccf442-28a0-4b70-9533-b83e894bc580","Type":"ContainerStarted","Data":"fe67e237544529b140aab73c0748b2b82fcefff11a4cd8ab1fada5f9242173d4"}
Sep 16 16:41:10.337236 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:10.337198    2554 generic.go:356] "Generic (PLEG): container finished" podID="49ccf442-28a0-4b70-9533-b83e894bc580" containerID="14f1c1ba7a3505fca193d11826cf47724225e6000b412ec0dfd63aa08cf29a76" exitCode=0
Sep 16 16:41:10.337639 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:10.337234    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx" event={"ID":"49ccf442-28a0-4b70-9533-b83e894bc580","Type":"ContainerDied","Data":"14f1c1ba7a3505fca193d11826cf47724225e6000b412ec0dfd63aa08cf29a76"}
Sep 16 16:41:11.343315 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:11.343281    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx" event={"ID":"49ccf442-28a0-4b70-9533-b83e894bc580","Type":"ContainerStarted","Data":"71749e12f7c1c15a2e04ee63a3edfed76ece1298a6e1698d20bd057207d212ae"}
Sep 16 16:41:11.369843 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:11.369756    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx" podStartSLOduration=2.69721281 podStartE2EDuration="5.369729612s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.672516802s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:41:11.368364935 +0000 UTC m=+3071.813057133" watchObservedRunningTime="2026-09-16 16:41:11.369729612 +0000 UTC m=+3071.814421791"
Sep 16 16:41:11.699041 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:11.698954    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx"
Sep 16 16:41:16.698776 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:16.698736    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx"
Sep 16 16:41:16.700287 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:16.700258    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx" podUID="49ccf442-28a0-4b70-9533-b83e894bc580" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 16 16:41:26.699567 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:26.699521    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx" podUID="49ccf442-28a0-4b70-9533-b83e894bc580" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 16 16:41:28.521272 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:28.521237    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:41:36.699773 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:36.699728    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx" podUID="49ccf442-28a0-4b70-9533-b83e894bc580" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 16 16:41:43.363128 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:43.363077    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:41:46.699485 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:46.699433    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx" podUID="49ccf442-28a0-4b70-9533-b83e894bc580" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 16 16:41:56.698843 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:41:56.698779    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx" podUID="49ccf442-28a0-4b70-9533-b83e894bc580" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 16 16:42:03.573124 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:42:03.573084    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:42:06.458941 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:42:06.458897    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:42:06.563104 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:42:06.563072    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:42:06.699574 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:42:06.699539    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx" podUID="49ccf442-28a0-4b70-9533-b83e894bc580" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 16 16:42:06.781009 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:42:06.780975    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:42:06.923057 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:42:06.923016    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:42:09.977565 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:42:09.977535    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:42:10.830328 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:42:10.830290    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:42:11.541377 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:42:11.541346    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:42:16.699225 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:42:16.699181    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx" podUID="49ccf442-28a0-4b70-9533-b83e894bc580" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 16 16:42:26.708909 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:42:26.708834    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx"
Sep 16 16:42:26.716641 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:42:26.716613    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx"
Sep 16 16:42:29.967954 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:42:29.967920    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:42:42.423113 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:42:42.423076    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:42:48.130906 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:42:48.130873    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:42:48.230981 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:42:48.230951    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:43:11.252126 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:43:11.252089    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx"]
Sep 16 16:43:11.253872 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:43:11.253847    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:43:11.314549 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:43:11.314518    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:43:26.274853 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:43:26.274765    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx" podUID="49ccf442-28a0-4b70-9533-b83e894bc580" containerName="main" containerID="cri-o://71749e12f7c1c15a2e04ee63a3edfed76ece1298a6e1698d20bd057207d212ae" gracePeriod=45
Sep 16 16:43:26.710112 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:43:26.710077    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx" podUID="49ccf442-28a0-4b70-9533-b83e894bc580" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 16 16:43:26.865233 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:43:26.865203    2554 generic.go:356] "Generic (PLEG): container finished" podID="49ccf442-28a0-4b70-9533-b83e894bc580" containerID="71749e12f7c1c15a2e04ee63a3edfed76ece1298a6e1698d20bd057207d212ae" exitCode=0
Sep 16 16:43:26.865385 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:43:26.865270    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx" event={"ID":"49ccf442-28a0-4b70-9533-b83e894bc580","Type":"ContainerDied","Data":"71749e12f7c1c15a2e04ee63a3edfed76ece1298a6e1698d20bd057207d212ae"}
Sep 16 16:43:26.919997 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:43:26.919968    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx"
Sep 16 16:43:27.051131 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:43:27.051102    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49ccf442-28a0-4b70-9533-b83e894bc580-tls-certs\") pod \"49ccf442-28a0-4b70-9533-b83e894bc580\" (UID: \"49ccf442-28a0-4b70-9533-b83e894bc580\") "
Sep 16 16:43:27.051313 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:43:27.051155    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49ccf442-28a0-4b70-9533-b83e894bc580-home\") pod \"49ccf442-28a0-4b70-9533-b83e894bc580\" (UID: \"49ccf442-28a0-4b70-9533-b83e894bc580\") "
Sep 16 16:43:27.051313 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:43:27.051182    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49ccf442-28a0-4b70-9533-b83e894bc580-kserve-provision-location\") pod \"49ccf442-28a0-4b70-9533-b83e894bc580\" (UID: \"49ccf442-28a0-4b70-9533-b83e894bc580\") "
Sep 16 16:43:27.051313 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:43:27.051203    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49ccf442-28a0-4b70-9533-b83e894bc580-cabundle-cert\") pod \"49ccf442-28a0-4b70-9533-b83e894bc580\" (UID: \"49ccf442-28a0-4b70-9533-b83e894bc580\") "
Sep 16 16:43:27.051313 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:43:27.051224    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgztk\" (UniqueName: \"kubernetes.io/projected/49ccf442-28a0-4b70-9533-b83e894bc580-kube-api-access-cgztk\") pod \"49ccf442-28a0-4b70-9533-b83e894bc580\" (UID: \"49ccf442-28a0-4b70-9533-b83e894bc580\") "
Sep 16 16:43:27.051313 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:43:27.051256    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49ccf442-28a0-4b70-9533-b83e894bc580-dshm\") pod \"49ccf442-28a0-4b70-9533-b83e894bc580\" (UID: \"49ccf442-28a0-4b70-9533-b83e894bc580\") "
Sep 16 16:43:27.051587 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:43:27.051320    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49ccf442-28a0-4b70-9533-b83e894bc580-tmp-dir\") pod \"49ccf442-28a0-4b70-9533-b83e894bc580\" (UID: \"49ccf442-28a0-4b70-9533-b83e894bc580\") "
Sep 16 16:43:27.051587 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:43:27.051384    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49ccf442-28a0-4b70-9533-b83e894bc580-model-cache\") pod \"49ccf442-28a0-4b70-9533-b83e894bc580\" (UID: \"49ccf442-28a0-4b70-9533-b83e894bc580\") "
Sep 16 16:43:27.051712 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:43:27.051627    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49ccf442-28a0-4b70-9533-b83e894bc580-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "49ccf442-28a0-4b70-9533-b83e894bc580" (UID: "49ccf442-28a0-4b70-9533-b83e894bc580"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 16:43:27.051712 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:43:27.051674    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ccf442-28a0-4b70-9533-b83e894bc580-home" (OuterVolumeSpecName: "home") pod "49ccf442-28a0-4b70-9533-b83e894bc580" (UID: "49ccf442-28a0-4b70-9533-b83e894bc580"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:43:27.052013 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:43:27.051990    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ccf442-28a0-4b70-9533-b83e894bc580-model-cache" (OuterVolumeSpecName: "model-cache") pod "49ccf442-28a0-4b70-9533-b83e894bc580" (UID: "49ccf442-28a0-4b70-9533-b83e894bc580"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:43:27.052199 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:43:27.052181    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49ccf442-28a0-4b70-9533-b83e894bc580-model-cache\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:43:27.052279 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:43:27.052204    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49ccf442-28a0-4b70-9533-b83e894bc580-home\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:43:27.052279 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:43:27.052218    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49ccf442-28a0-4b70-9533-b83e894bc580-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:43:27.053646 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:43:27.053617    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49ccf442-28a0-4b70-9533-b83e894bc580-kube-api-access-cgztk" (OuterVolumeSpecName: "kube-api-access-cgztk") pod "49ccf442-28a0-4b70-9533-b83e894bc580" (UID: "49ccf442-28a0-4b70-9533-b83e894bc580"). InnerVolumeSpecName "kube-api-access-cgztk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:43:27.053749 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:43:27.053664    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ccf442-28a0-4b70-9533-b83e894bc580-dshm" (OuterVolumeSpecName: "dshm") pod "49ccf442-28a0-4b70-9533-b83e894bc580" (UID: "49ccf442-28a0-4b70-9533-b83e894bc580"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:43:27.053749 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:43:27.053694    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49ccf442-28a0-4b70-9533-b83e894bc580-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "49ccf442-28a0-4b70-9533-b83e894bc580" (UID: "49ccf442-28a0-4b70-9533-b83e894bc580"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:43:27.069396 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:43:27.069331    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ccf442-28a0-4b70-9533-b83e894bc580-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "49ccf442-28a0-4b70-9533-b83e894bc580" (UID: "49ccf442-28a0-4b70-9533-b83e894bc580"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:43:27.074218 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:43:27.074191    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ccf442-28a0-4b70-9533-b83e894bc580-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "49ccf442-28a0-4b70-9533-b83e894bc580" (UID: "49ccf442-28a0-4b70-9533-b83e894bc580"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:43:27.153527 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:43:27.153473    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49ccf442-28a0-4b70-9533-b83e894bc580-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:43:27.153527 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:43:27.153516    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgztk\" (UniqueName: \"kubernetes.io/projected/49ccf442-28a0-4b70-9533-b83e894bc580-kube-api-access-cgztk\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:43:27.153527 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:43:27.153530    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49ccf442-28a0-4b70-9533-b83e894bc580-dshm\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:43:27.153789 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:43:27.153544    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49ccf442-28a0-4b70-9533-b83e894bc580-tmp-dir\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:43:27.153789 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:43:27.153559    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49ccf442-28a0-4b70-9533-b83e894bc580-tls-certs\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:43:27.870232 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:43:27.870197    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx" event={"ID":"49ccf442-28a0-4b70-9533-b83e894bc580","Type":"ContainerDied","Data":"fe67e237544529b140aab73c0748b2b82fcefff11a4cd8ab1fada5f9242173d4"}
Sep 16 16:43:27.870657 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:43:27.870257    2554 scope.go:117] "RemoveContainer" containerID="71749e12f7c1c15a2e04ee63a3edfed76ece1298a6e1698d20bd057207d212ae"
Sep 16 16:43:27.870657 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:43:27.870204    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx"
Sep 16 16:43:27.878945 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:43:27.878922    2554 scope.go:117] "RemoveContainer" containerID="14f1c1ba7a3505fca193d11826cf47724225e6000b412ec0dfd63aa08cf29a76"
Sep 16 16:43:27.893050 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:43:27.893023    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx"]
Sep 16 16:43:27.897125 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:43:27.897103    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-599944dc97-757gx"]
Sep 16 16:43:28.077199 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:43:28.077168    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49ccf442-28a0-4b70-9533-b83e894bc580" path="/var/lib/kubelet/pods/49ccf442-28a0-4b70-9533-b83e894bc580/volumes"
Sep 16 16:43:46.459425 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:43:46.459386    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:43:51.451738 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:43:51.451707    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:43:51.547709 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:43:51.547675    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:43:56.476508 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:43:56.476470    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:44:01.118710 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:01.118675    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:44:01.381606 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:01.381528    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:44:01.477778 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:01.477743    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:44:06.228323 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:06.228289    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:44:07.753300 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:07.753267    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:44:11.922047 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:11.922012    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:44:13.575714 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:13.575678    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:44:13.675853 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:13.675808    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:44:18.167519 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:18.167487    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:44:18.847887 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:18.847789    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:44:21.479414 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:21.479377    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t"]
Sep 16 16:44:21.479888 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:21.479731    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49ccf442-28a0-4b70-9533-b83e894bc580" containerName="main"
Sep 16 16:44:21.479888 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:21.479749    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="49ccf442-28a0-4b70-9533-b83e894bc580" containerName="main"
Sep 16 16:44:21.479888 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:21.479768    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49ccf442-28a0-4b70-9533-b83e894bc580" containerName="storage-initializer"
Sep 16 16:44:21.479888 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:21.479776    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="49ccf442-28a0-4b70-9533-b83e894bc580" containerName="storage-initializer"
Sep 16 16:44:21.479888 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:21.479874    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="49ccf442-28a0-4b70-9533-b83e894bc580" containerName="main"
Sep 16 16:44:21.482759 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:21.482736    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t"
Sep 16 16:44:21.485599 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:21.485579    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 16 16:44:21.485726 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:21.485602    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 16 16:44:21.485726 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:21.485642    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Sep 16 16:44:21.485726 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:21.485675    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-zqm6x\""
Sep 16 16:44:21.485926 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:21.485780    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 16:44:21.486834 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:21.486814    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 16 16:44:21.494559 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:21.494538    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t"]
Sep 16 16:44:21.509335 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:21.509311    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:44:21.524731 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:21.524707    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fea9ae87-7c0c-416f-860c-5f4a72a2925f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t\" (UID: \"fea9ae87-7c0c-416f-860c-5f4a72a2925f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t"
Sep 16 16:44:21.524846 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:21.524742    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fea9ae87-7c0c-416f-860c-5f4a72a2925f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t\" (UID: \"fea9ae87-7c0c-416f-860c-5f4a72a2925f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t"
Sep 16 16:44:21.524846 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:21.524784    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fea9ae87-7c0c-416f-860c-5f4a72a2925f-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t\" (UID: \"fea9ae87-7c0c-416f-860c-5f4a72a2925f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t"
Sep 16 16:44:21.524928 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:21.524847    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fea9ae87-7c0c-416f-860c-5f4a72a2925f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t\" (UID: \"fea9ae87-7c0c-416f-860c-5f4a72a2925f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t"
Sep 16 16:44:21.524928 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:21.524878    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fea9ae87-7c0c-416f-860c-5f4a72a2925f-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t\" (UID: \"fea9ae87-7c0c-416f-860c-5f4a72a2925f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t"
Sep 16 16:44:21.524928 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:21.524895    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsp4x\" (UniqueName: \"kubernetes.io/projected/fea9ae87-7c0c-416f-860c-5f4a72a2925f-kube-api-access-fsp4x\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t\" (UID: \"fea9ae87-7c0c-416f-860c-5f4a72a2925f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t"
Sep 16 16:44:21.524928 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:21.524916    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fea9ae87-7c0c-416f-860c-5f4a72a2925f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t\" (UID: \"fea9ae87-7c0c-416f-860c-5f4a72a2925f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t"
Sep 16 16:44:21.525059 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:21.524995    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fea9ae87-7c0c-416f-860c-5f4a72a2925f-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t\" (UID: \"fea9ae87-7c0c-416f-860c-5f4a72a2925f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t"
Sep 16 16:44:21.625287 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:21.625259    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fea9ae87-7c0c-416f-860c-5f4a72a2925f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t\" (UID: \"fea9ae87-7c0c-416f-860c-5f4a72a2925f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t"
Sep 16 16:44:21.625418 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:21.625292    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fea9ae87-7c0c-416f-860c-5f4a72a2925f-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t\" (UID: \"fea9ae87-7c0c-416f-860c-5f4a72a2925f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t"
Sep 16 16:44:21.625418 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:21.625310    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fea9ae87-7c0c-416f-860c-5f4a72a2925f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t\" (UID: \"fea9ae87-7c0c-416f-860c-5f4a72a2925f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t"
Sep 16 16:44:21.625418 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:21.625334    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fea9ae87-7c0c-416f-860c-5f4a72a2925f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t\" (UID: \"fea9ae87-7c0c-416f-860c-5f4a72a2925f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t"
Sep 16 16:44:21.625418 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:21.625364    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fea9ae87-7c0c-416f-860c-5f4a72a2925f-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t\" (UID: \"fea9ae87-7c0c-416f-860c-5f4a72a2925f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t"
Sep 16 16:44:21.625418 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:21.625406    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fea9ae87-7c0c-416f-860c-5f4a72a2925f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t\" (UID: \"fea9ae87-7c0c-416f-860c-5f4a72a2925f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t"
Sep 16 16:44:21.625671 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:21.625441    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fea9ae87-7c0c-416f-860c-5f4a72a2925f-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t\" (UID: \"fea9ae87-7c0c-416f-860c-5f4a72a2925f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t"
Sep 16 16:44:21.625671 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:21.625526    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsp4x\" (UniqueName: \"kubernetes.io/projected/fea9ae87-7c0c-416f-860c-5f4a72a2925f-kube-api-access-fsp4x\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t\" (UID: \"fea9ae87-7c0c-416f-860c-5f4a72a2925f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t"
Sep 16 16:44:21.625774 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:21.625712    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fea9ae87-7c0c-416f-860c-5f4a72a2925f-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t\" (UID: \"fea9ae87-7c0c-416f-860c-5f4a72a2925f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t"
Sep 16 16:44:21.625774 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:21.625731    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fea9ae87-7c0c-416f-860c-5f4a72a2925f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t\" (UID: \"fea9ae87-7c0c-416f-860c-5f4a72a2925f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t"
Sep 16 16:44:21.625904 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:21.625827    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fea9ae87-7c0c-416f-860c-5f4a72a2925f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t\" (UID: \"fea9ae87-7c0c-416f-860c-5f4a72a2925f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t"
Sep 16 16:44:21.626020 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:21.626000    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fea9ae87-7c0c-416f-860c-5f4a72a2925f-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t\" (UID: \"fea9ae87-7c0c-416f-860c-5f4a72a2925f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t"
Sep 16 16:44:21.626198 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:21.626181    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fea9ae87-7c0c-416f-860c-5f4a72a2925f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t\" (UID: \"fea9ae87-7c0c-416f-860c-5f4a72a2925f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t"
Sep 16 16:44:21.627710 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:21.627691    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fea9ae87-7c0c-416f-860c-5f4a72a2925f-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t\" (UID: \"fea9ae87-7c0c-416f-860c-5f4a72a2925f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t"
Sep 16 16:44:21.627907 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:21.627888    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fea9ae87-7c0c-416f-860c-5f4a72a2925f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t\" (UID: \"fea9ae87-7c0c-416f-860c-5f4a72a2925f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t"
Sep 16 16:44:21.633439 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:21.633411    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsp4x\" (UniqueName: \"kubernetes.io/projected/fea9ae87-7c0c-416f-860c-5f4a72a2925f-kube-api-access-fsp4x\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t\" (UID: \"fea9ae87-7c0c-416f-860c-5f4a72a2925f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t"
Sep 16 16:44:21.785648 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:21.785614    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:44:21.793821 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:21.793770    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t"
Sep 16 16:44:22.069018 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:22.068990    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t"]
Sep 16 16:44:22.071527 ip-10-0-136-228 kubenswrapper[2554]: W0916 16:44:22.071496    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfea9ae87_7c0c_416f_860c_5f4a72a2925f.slice/crio-7108fe3cd3078c6f0cc62bc5d3b0bbb675c0d9dc26dfcc72a58f1997a65d4bcd WatchSource:0}: Error finding container 7108fe3cd3078c6f0cc62bc5d3b0bbb675c0d9dc26dfcc72a58f1997a65d4bcd: Status 404 returned error can't find the container with id 7108fe3cd3078c6f0cc62bc5d3b0bbb675c0d9dc26dfcc72a58f1997a65d4bcd
Sep 16 16:44:23.079461 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:23.079422    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t" event={"ID":"fea9ae87-7c0c-416f-860c-5f4a72a2925f","Type":"ContainerStarted","Data":"8e49874cb040fcec604f8f3a6eed4fe17f3598602d5e5d3b9ef7bdd1d6882c87"}
Sep 16 16:44:23.079864 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:23.079477    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t" event={"ID":"fea9ae87-7c0c-416f-860c-5f4a72a2925f","Type":"ContainerStarted","Data":"7108fe3cd3078c6f0cc62bc5d3b0bbb675c0d9dc26dfcc72a58f1997a65d4bcd"}
Sep 16 16:44:24.842837 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:24.842808    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:44:25.089448 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:25.089359    2554 generic.go:356] "Generic (PLEG): container finished" podID="fea9ae87-7c0c-416f-860c-5f4a72a2925f" containerID="8e49874cb040fcec604f8f3a6eed4fe17f3598602d5e5d3b9ef7bdd1d6882c87" exitCode=0
Sep 16 16:44:25.089448 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:25.089420    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t" event={"ID":"fea9ae87-7c0c-416f-860c-5f4a72a2925f","Type":"ContainerDied","Data":"8e49874cb040fcec604f8f3a6eed4fe17f3598602d5e5d3b9ef7bdd1d6882c87"}
Sep 16 16:44:26.094236 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:26.094200    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t" event={"ID":"fea9ae87-7c0c-416f-860c-5f4a72a2925f","Type":"ContainerStarted","Data":"d02964d42b1c6b1f744018275445f7e64527a4f044b1ef2433af31254ce54365"}
Sep 16 16:44:26.118876 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:26.118810    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t" podStartSLOduration=2.430379485 podStartE2EDuration="5.118772668s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.688393183s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:44:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:44:26.11555633 +0000 UTC m=+3266.560248564" watchObservedRunningTime="2026-09-16 16:44:26.118772668 +0000 UTC m=+3266.563464847"
Sep 16 16:44:26.794620 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:26.794585    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t"
Sep 16 16:44:31.794176 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:31.794137    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t"
Sep 16 16:44:31.795832 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:31.795778    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t" podUID="fea9ae87-7c0c-416f-860c-5f4a72a2925f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Sep 16 16:44:38.658137 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:38.658101    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:44:41.794784 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:41.794732    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t" podUID="fea9ae87-7c0c-416f-860c-5f4a72a2925f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Sep 16 16:44:44.312871 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:44.312843    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:44:49.527973 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:49.527934    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:44:49.628436 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:49.628404    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:44:51.794301 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:44:51.794252    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t" podUID="fea9ae87-7c0c-416f-860c-5f4a72a2925f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Sep 16 16:45:01.795206 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:01.795156    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t" podUID="fea9ae87-7c0c-416f-860c-5f4a72a2925f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Sep 16 16:45:11.794606 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:11.794558    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t" podUID="fea9ae87-7c0c-416f-860c-5f4a72a2925f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Sep 16 16:45:21.795127 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:21.795073    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t" podUID="fea9ae87-7c0c-416f-860c-5f4a72a2925f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Sep 16 16:45:30.866135 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:30.866098    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:45:31.765787 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:31.765750    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:45:31.795068 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:31.795031    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t" podUID="fea9ae87-7c0c-416f-860c-5f4a72a2925f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Sep 16 16:45:41.804223 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:41.804190    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t"
Sep 16 16:45:41.811945 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:41.811916    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t"
Sep 16 16:45:43.691434 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:43.691396    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t"]
Sep 16 16:45:43.693374 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:43.693325    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:45:43.783896 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:43.783865    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:45:58.713952 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:58.713888    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t" podUID="fea9ae87-7c0c-416f-860c-5f4a72a2925f" containerName="main" containerID="cri-o://d02964d42b1c6b1f744018275445f7e64527a4f044b1ef2433af31254ce54365" gracePeriod=45
Sep 16 16:45:58.805667 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:58.805619    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t" podUID="fea9ae87-7c0c-416f-860c-5f4a72a2925f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Sep 16 16:45:59.345786 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:59.345763    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t"
Sep 16 16:45:59.457027 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:59.456946    2554 generic.go:356] "Generic (PLEG): container finished" podID="fea9ae87-7c0c-416f-860c-5f4a72a2925f" containerID="d02964d42b1c6b1f744018275445f7e64527a4f044b1ef2433af31254ce54365" exitCode=0
Sep 16 16:45:59.457027 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:59.456991    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t" event={"ID":"fea9ae87-7c0c-416f-860c-5f4a72a2925f","Type":"ContainerDied","Data":"d02964d42b1c6b1f744018275445f7e64527a4f044b1ef2433af31254ce54365"}
Sep 16 16:45:59.457250 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:59.457044    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t" event={"ID":"fea9ae87-7c0c-416f-860c-5f4a72a2925f","Type":"ContainerDied","Data":"7108fe3cd3078c6f0cc62bc5d3b0bbb675c0d9dc26dfcc72a58f1997a65d4bcd"}
Sep 16 16:45:59.457250 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:59.457067    2554 scope.go:117] "RemoveContainer" containerID="d02964d42b1c6b1f744018275445f7e64527a4f044b1ef2433af31254ce54365"
Sep 16 16:45:59.457250 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:59.457005    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t"
Sep 16 16:45:59.464995 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:59.464978    2554 scope.go:117] "RemoveContainer" containerID="8e49874cb040fcec604f8f3a6eed4fe17f3598602d5e5d3b9ef7bdd1d6882c87"
Sep 16 16:45:59.471063 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:59.471038    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fea9ae87-7c0c-416f-860c-5f4a72a2925f-tls-certs\") pod \"fea9ae87-7c0c-416f-860c-5f4a72a2925f\" (UID: \"fea9ae87-7c0c-416f-860c-5f4a72a2925f\") "
Sep 16 16:45:59.471173 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:59.471074    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fea9ae87-7c0c-416f-860c-5f4a72a2925f-tmp-dir\") pod \"fea9ae87-7c0c-416f-860c-5f4a72a2925f\" (UID: \"fea9ae87-7c0c-416f-860c-5f4a72a2925f\") "
Sep 16 16:45:59.471173 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:59.471103    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fea9ae87-7c0c-416f-860c-5f4a72a2925f-kserve-provision-location\") pod \"fea9ae87-7c0c-416f-860c-5f4a72a2925f\" (UID: \"fea9ae87-7c0c-416f-860c-5f4a72a2925f\") "
Sep 16 16:45:59.471173 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:59.471126    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fea9ae87-7c0c-416f-860c-5f4a72a2925f-dshm\") pod \"fea9ae87-7c0c-416f-860c-5f4a72a2925f\" (UID: \"fea9ae87-7c0c-416f-860c-5f4a72a2925f\") "
Sep 16 16:45:59.471348 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:59.471189    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fea9ae87-7c0c-416f-860c-5f4a72a2925f-cabundle-cert\") pod \"fea9ae87-7c0c-416f-860c-5f4a72a2925f\" (UID: \"fea9ae87-7c0c-416f-860c-5f4a72a2925f\") "
Sep 16 16:45:59.471348 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:59.471219    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fea9ae87-7c0c-416f-860c-5f4a72a2925f-model-cache\") pod \"fea9ae87-7c0c-416f-860c-5f4a72a2925f\" (UID: \"fea9ae87-7c0c-416f-860c-5f4a72a2925f\") "
Sep 16 16:45:59.471348 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:59.471261    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fsp4x\" (UniqueName: \"kubernetes.io/projected/fea9ae87-7c0c-416f-860c-5f4a72a2925f-kube-api-access-fsp4x\") pod \"fea9ae87-7c0c-416f-860c-5f4a72a2925f\" (UID: \"fea9ae87-7c0c-416f-860c-5f4a72a2925f\") "
Sep 16 16:45:59.471348 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:59.471302    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fea9ae87-7c0c-416f-860c-5f4a72a2925f-home\") pod \"fea9ae87-7c0c-416f-860c-5f4a72a2925f\" (UID: \"fea9ae87-7c0c-416f-860c-5f4a72a2925f\") "
Sep 16 16:45:59.471706 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:59.471663    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea9ae87-7c0c-416f-860c-5f4a72a2925f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fea9ae87-7c0c-416f-860c-5f4a72a2925f" (UID: "fea9ae87-7c0c-416f-860c-5f4a72a2925f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:45:59.471890 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:59.471859    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea9ae87-7c0c-416f-860c-5f4a72a2925f-home" (OuterVolumeSpecName: "home") pod "fea9ae87-7c0c-416f-860c-5f4a72a2925f" (UID: "fea9ae87-7c0c-416f-860c-5f4a72a2925f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:45:59.471890 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:59.471869    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fea9ae87-7c0c-416f-860c-5f4a72a2925f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fea9ae87-7c0c-416f-860c-5f4a72a2925f" (UID: "fea9ae87-7c0c-416f-860c-5f4a72a2925f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 16:45:59.472263 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:59.472235    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea9ae87-7c0c-416f-860c-5f4a72a2925f-model-cache" (OuterVolumeSpecName: "model-cache") pod "fea9ae87-7c0c-416f-860c-5f4a72a2925f" (UID: "fea9ae87-7c0c-416f-860c-5f4a72a2925f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:45:59.472663 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:59.472639    2554 scope.go:117] "RemoveContainer" containerID="d02964d42b1c6b1f744018275445f7e64527a4f044b1ef2433af31254ce54365"
Sep 16 16:45:59.472990 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:45:59.472970    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d02964d42b1c6b1f744018275445f7e64527a4f044b1ef2433af31254ce54365\": container with ID starting with d02964d42b1c6b1f744018275445f7e64527a4f044b1ef2433af31254ce54365 not found: ID does not exist" containerID="d02964d42b1c6b1f744018275445f7e64527a4f044b1ef2433af31254ce54365"
Sep 16 16:45:59.473091 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:59.472998    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d02964d42b1c6b1f744018275445f7e64527a4f044b1ef2433af31254ce54365"} err="failed to get container status \"d02964d42b1c6b1f744018275445f7e64527a4f044b1ef2433af31254ce54365\": rpc error: code = NotFound desc = could not find container \"d02964d42b1c6b1f744018275445f7e64527a4f044b1ef2433af31254ce54365\": container with ID starting with d02964d42b1c6b1f744018275445f7e64527a4f044b1ef2433af31254ce54365 not found: ID does not exist"
Sep 16 16:45:59.473091 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:59.473016    2554 scope.go:117] "RemoveContainer" containerID="8e49874cb040fcec604f8f3a6eed4fe17f3598602d5e5d3b9ef7bdd1d6882c87"
Sep 16 16:45:59.473360 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:45:59.473303    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e49874cb040fcec604f8f3a6eed4fe17f3598602d5e5d3b9ef7bdd1d6882c87\": container with ID starting with 8e49874cb040fcec604f8f3a6eed4fe17f3598602d5e5d3b9ef7bdd1d6882c87 not found: ID does not exist" containerID="8e49874cb040fcec604f8f3a6eed4fe17f3598602d5e5d3b9ef7bdd1d6882c87"
Sep 16 16:45:59.473360 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:59.473341    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e49874cb040fcec604f8f3a6eed4fe17f3598602d5e5d3b9ef7bdd1d6882c87"} err="failed to get container status \"8e49874cb040fcec604f8f3a6eed4fe17f3598602d5e5d3b9ef7bdd1d6882c87\": rpc error: code = NotFound desc = could not find container \"8e49874cb040fcec604f8f3a6eed4fe17f3598602d5e5d3b9ef7bdd1d6882c87\": container with ID starting with 8e49874cb040fcec604f8f3a6eed4fe17f3598602d5e5d3b9ef7bdd1d6882c87 not found: ID does not exist"
Sep 16 16:45:59.473538 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:59.473473    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fea9ae87-7c0c-416f-860c-5f4a72a2925f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fea9ae87-7c0c-416f-860c-5f4a72a2925f" (UID: "fea9ae87-7c0c-416f-860c-5f4a72a2925f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:45:59.473685 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:59.473665    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea9ae87-7c0c-416f-860c-5f4a72a2925f-dshm" (OuterVolumeSpecName: "dshm") pod "fea9ae87-7c0c-416f-860c-5f4a72a2925f" (UID: "fea9ae87-7c0c-416f-860c-5f4a72a2925f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:45:59.473948 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:59.473923    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fea9ae87-7c0c-416f-860c-5f4a72a2925f-kube-api-access-fsp4x" (OuterVolumeSpecName: "kube-api-access-fsp4x") pod "fea9ae87-7c0c-416f-860c-5f4a72a2925f" (UID: "fea9ae87-7c0c-416f-860c-5f4a72a2925f"). InnerVolumeSpecName "kube-api-access-fsp4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:45:59.493471 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:59.493443    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea9ae87-7c0c-416f-860c-5f4a72a2925f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fea9ae87-7c0c-416f-860c-5f4a72a2925f" (UID: "fea9ae87-7c0c-416f-860c-5f4a72a2925f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:45:59.571809 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:59.571768    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fea9ae87-7c0c-416f-860c-5f4a72a2925f-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:45:59.571809 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:59.571803    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fea9ae87-7c0c-416f-860c-5f4a72a2925f-model-cache\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:45:59.571809 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:59.571814    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsp4x\" (UniqueName: \"kubernetes.io/projected/fea9ae87-7c0c-416f-860c-5f4a72a2925f-kube-api-access-fsp4x\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:45:59.571990 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:59.571823    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fea9ae87-7c0c-416f-860c-5f4a72a2925f-home\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:45:59.571990 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:59.571831    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fea9ae87-7c0c-416f-860c-5f4a72a2925f-tls-certs\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:45:59.571990 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:59.571840    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fea9ae87-7c0c-416f-860c-5f4a72a2925f-tmp-dir\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:45:59.571990 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:59.571848    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fea9ae87-7c0c-416f-860c-5f4a72a2925f-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:45:59.571990 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:59.571856    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fea9ae87-7c0c-416f-860c-5f4a72a2925f-dshm\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:45:59.780078 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:59.780046    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t"]
Sep 16 16:45:59.784664 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:45:59.784641    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6d4985f674-2gg9t"]
Sep 16 16:46:00.077919 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:00.077890    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fea9ae87-7c0c-416f-860c-5f4a72a2925f" path="/var/lib/kubelet/pods/fea9ae87-7c0c-416f-860c-5f4a72a2925f/volumes"
Sep 16 16:46:06.241119 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:06.241076    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:46:06.341642 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:06.341599    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:46:34.871227 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:34.871172    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:46:34.927116 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:34.927064    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:46:35.846515 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:35.846467    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:46:35.947458 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:35.947402    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:46:36.048412 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:36.048381    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:46:39.567186 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:39.567154    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:46:41.767579 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:41.767531    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:46:41.825487 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:41.825417    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:46:45.454760 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:45.454721    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:46:45.556086 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:45.556053    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:46:46.110077 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:46.110045    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85779c986f-hswjk"]
Sep 16 16:46:46.110305 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:46.110293    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fea9ae87-7c0c-416f-860c-5f4a72a2925f" containerName="storage-initializer"
Sep 16 16:46:46.110349 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:46.110306    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="fea9ae87-7c0c-416f-860c-5f4a72a2925f" containerName="storage-initializer"
Sep 16 16:46:46.110349 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:46.110325    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fea9ae87-7c0c-416f-860c-5f4a72a2925f" containerName="main"
Sep 16 16:46:46.110349 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:46.110330    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="fea9ae87-7c0c-416f-860c-5f4a72a2925f" containerName="main"
Sep 16 16:46:46.110448 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:46.110376    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="fea9ae87-7c0c-416f-860c-5f4a72a2925f" containerName="main"
Sep 16 16:46:46.112000 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:46.111983    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85779c986f-hswjk"
Sep 16 16:46:46.115008 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:46.114987    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-kserve-self-signed-certs\""
Sep 16 16:46:46.115126 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:46.115063    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"seaweedfs-s3-creds\""
Sep 16 16:46:46.115301 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:46.115285    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"openshift-service-ca.crt\""
Sep 16 16:46:46.115373 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:46.115313    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"default-dockercfg-p5skd\""
Sep 16 16:46:46.115457 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:46.115439    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"kube-root-ca.crt\""
Sep 16 16:46:46.128664 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:46.128640    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85779c986f-hswjk"]
Sep 16 16:46:46.134775 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:46.134754    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:46:46.209569 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:46.209526    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae39005d-19ea-4cba-a75f-884b2c1db905-model-cache\") pod \"e2e-del-blocked-svc-kserve-85779c986f-hswjk\" (UID: \"ae39005d-19ea-4cba-a75f-884b2c1db905\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85779c986f-hswjk"
Sep 16 16:46:46.209749 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:46.209575    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgdx5\" (UniqueName: \"kubernetes.io/projected/ae39005d-19ea-4cba-a75f-884b2c1db905-kube-api-access-pgdx5\") pod \"e2e-del-blocked-svc-kserve-85779c986f-hswjk\" (UID: \"ae39005d-19ea-4cba-a75f-884b2c1db905\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85779c986f-hswjk"
Sep 16 16:46:46.209749 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:46.209610    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae39005d-19ea-4cba-a75f-884b2c1db905-dshm\") pod \"e2e-del-blocked-svc-kserve-85779c986f-hswjk\" (UID: \"ae39005d-19ea-4cba-a75f-884b2c1db905\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85779c986f-hswjk"
Sep 16 16:46:46.209749 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:46.209628    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae39005d-19ea-4cba-a75f-884b2c1db905-home\") pod \"e2e-del-blocked-svc-kserve-85779c986f-hswjk\" (UID: \"ae39005d-19ea-4cba-a75f-884b2c1db905\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85779c986f-hswjk"
Sep 16 16:46:46.209749 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:46.209642    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae39005d-19ea-4cba-a75f-884b2c1db905-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-85779c986f-hswjk\" (UID: \"ae39005d-19ea-4cba-a75f-884b2c1db905\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85779c986f-hswjk"
Sep 16 16:46:46.209749 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:46.209661    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae39005d-19ea-4cba-a75f-884b2c1db905-tls-certs\") pod \"e2e-del-blocked-svc-kserve-85779c986f-hswjk\" (UID: \"ae39005d-19ea-4cba-a75f-884b2c1db905\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85779c986f-hswjk"
Sep 16 16:46:46.209956 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:46.209746    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae39005d-19ea-4cba-a75f-884b2c1db905-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-85779c986f-hswjk\" (UID: \"ae39005d-19ea-4cba-a75f-884b2c1db905\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85779c986f-hswjk"
Sep 16 16:46:46.310672 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:46.310637    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgdx5\" (UniqueName: \"kubernetes.io/projected/ae39005d-19ea-4cba-a75f-884b2c1db905-kube-api-access-pgdx5\") pod \"e2e-del-blocked-svc-kserve-85779c986f-hswjk\" (UID: \"ae39005d-19ea-4cba-a75f-884b2c1db905\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85779c986f-hswjk"
Sep 16 16:46:46.310672 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:46.310675    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae39005d-19ea-4cba-a75f-884b2c1db905-dshm\") pod \"e2e-del-blocked-svc-kserve-85779c986f-hswjk\" (UID: \"ae39005d-19ea-4cba-a75f-884b2c1db905\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85779c986f-hswjk"
Sep 16 16:46:46.310921 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:46.310694    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae39005d-19ea-4cba-a75f-884b2c1db905-home\") pod \"e2e-del-blocked-svc-kserve-85779c986f-hswjk\" (UID: \"ae39005d-19ea-4cba-a75f-884b2c1db905\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85779c986f-hswjk"
Sep 16 16:46:46.310921 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:46.310709    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae39005d-19ea-4cba-a75f-884b2c1db905-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-85779c986f-hswjk\" (UID: \"ae39005d-19ea-4cba-a75f-884b2c1db905\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85779c986f-hswjk"
Sep 16 16:46:46.310921 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:46.310839    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae39005d-19ea-4cba-a75f-884b2c1db905-tls-certs\") pod \"e2e-del-blocked-svc-kserve-85779c986f-hswjk\" (UID: \"ae39005d-19ea-4cba-a75f-884b2c1db905\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85779c986f-hswjk"
Sep 16 16:46:46.310921 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:46.310904    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae39005d-19ea-4cba-a75f-884b2c1db905-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-85779c986f-hswjk\" (UID: \"ae39005d-19ea-4cba-a75f-884b2c1db905\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85779c986f-hswjk"
Sep 16 16:46:46.311140 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:46.310939    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae39005d-19ea-4cba-a75f-884b2c1db905-model-cache\") pod \"e2e-del-blocked-svc-kserve-85779c986f-hswjk\" (UID: \"ae39005d-19ea-4cba-a75f-884b2c1db905\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85779c986f-hswjk"
Sep 16 16:46:46.311140 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:46.311108    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae39005d-19ea-4cba-a75f-884b2c1db905-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-85779c986f-hswjk\" (UID: \"ae39005d-19ea-4cba-a75f-884b2c1db905\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85779c986f-hswjk"
Sep 16 16:46:46.311256 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:46.311143    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae39005d-19ea-4cba-a75f-884b2c1db905-home\") pod \"e2e-del-blocked-svc-kserve-85779c986f-hswjk\" (UID: \"ae39005d-19ea-4cba-a75f-884b2c1db905\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85779c986f-hswjk"
Sep 16 16:46:46.311315 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:46.311281    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae39005d-19ea-4cba-a75f-884b2c1db905-model-cache\") pod \"e2e-del-blocked-svc-kserve-85779c986f-hswjk\" (UID: \"ae39005d-19ea-4cba-a75f-884b2c1db905\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85779c986f-hswjk"
Sep 16 16:46:46.311315 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:46.311287    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae39005d-19ea-4cba-a75f-884b2c1db905-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-85779c986f-hswjk\" (UID: \"ae39005d-19ea-4cba-a75f-884b2c1db905\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85779c986f-hswjk"
Sep 16 16:46:46.313106 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:46.313085    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae39005d-19ea-4cba-a75f-884b2c1db905-dshm\") pod \"e2e-del-blocked-svc-kserve-85779c986f-hswjk\" (UID: \"ae39005d-19ea-4cba-a75f-884b2c1db905\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85779c986f-hswjk"
Sep 16 16:46:46.313352 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:46.313331    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae39005d-19ea-4cba-a75f-884b2c1db905-tls-certs\") pod \"e2e-del-blocked-svc-kserve-85779c986f-hswjk\" (UID: \"ae39005d-19ea-4cba-a75f-884b2c1db905\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85779c986f-hswjk"
Sep 16 16:46:46.323573 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:46.323547    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgdx5\" (UniqueName: \"kubernetes.io/projected/ae39005d-19ea-4cba-a75f-884b2c1db905-kube-api-access-pgdx5\") pod \"e2e-del-blocked-svc-kserve-85779c986f-hswjk\" (UID: \"ae39005d-19ea-4cba-a75f-884b2c1db905\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85779c986f-hswjk"
Sep 16 16:46:46.421968 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:46.421880    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85779c986f-hswjk"
Sep 16 16:46:46.475769 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:46.475735    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:46:46.699264 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:46.699179    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85779c986f-hswjk"]
Sep 16 16:46:46.702945 ip-10-0-136-228 kubenswrapper[2554]: W0916 16:46:46.702919    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae39005d_19ea_4cba_a75f_884b2c1db905.slice/crio-c65c0f9f470953ba42fbc9b419fa977d04c7e4139e3323307f42348f2d81fd78 WatchSource:0}: Error finding container c65c0f9f470953ba42fbc9b419fa977d04c7e4139e3323307f42348f2d81fd78: Status 404 returned error can't find the container with id c65c0f9f470953ba42fbc9b419fa977d04c7e4139e3323307f42348f2d81fd78
Sep 16 16:46:47.623217 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:47.623182    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85779c986f-hswjk" event={"ID":"ae39005d-19ea-4cba-a75f-884b2c1db905","Type":"ContainerStarted","Data":"714fd107136ce5fd92f28ac2259ed33d22d7ab9094d09b80db9e494f30da3cdb"}
Sep 16 16:46:47.623631 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:47.623233    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85779c986f-hswjk" event={"ID":"ae39005d-19ea-4cba-a75f-884b2c1db905","Type":"ContainerStarted","Data":"c65c0f9f470953ba42fbc9b419fa977d04c7e4139e3323307f42348f2d81fd78"}
Sep 16 16:46:48.196273 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:48.196242    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:46:48.196745 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:48.196722    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85779c986f-hswjk"]
Sep 16 16:46:48.259034 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:48.258987    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:46:48.481576 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:48.481494    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:46:48.582477 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:48.582438    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:46:48.681479 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:48.681450    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:46:48.728388 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:46:48.728350    2554 secret.go:189] Couldn't get secret e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-self-signed-certs: secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found
Sep 16 16:46:48.728567 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:46:48.728424    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae39005d-19ea-4cba-a75f-884b2c1db905-tls-certs podName:ae39005d-19ea-4cba-a75f-884b2c1db905 nodeName:}" failed. No retries permitted until 2026-09-16 16:46:49.228406018 +0000 UTC m=+3409.673098173 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/ae39005d-19ea-4cba-a75f-884b2c1db905-tls-certs") pod "e2e-del-blocked-svc-kserve-85779c986f-hswjk" (UID: "ae39005d-19ea-4cba-a75f-884b2c1db905") : secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found
Sep 16 16:46:49.235410 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:46:49.235368    2554 secret.go:189] Couldn't get secret e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-self-signed-certs: secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found
Sep 16 16:46:49.235613 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:46:49.235518    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae39005d-19ea-4cba-a75f-884b2c1db905-tls-certs podName:ae39005d-19ea-4cba-a75f-884b2c1db905 nodeName:}" failed. No retries permitted until 2026-09-16 16:46:50.23543729 +0000 UTC m=+3410.680129459 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/ae39005d-19ea-4cba-a75f-884b2c1db905-tls-certs") pod "e2e-del-blocked-svc-kserve-85779c986f-hswjk" (UID: "ae39005d-19ea-4cba-a75f-884b2c1db905") : secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found
Sep 16 16:46:49.631863 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:49.631819    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85779c986f-hswjk" podUID="ae39005d-19ea-4cba-a75f-884b2c1db905" containerName="storage-initializer" containerID="cri-o://714fd107136ce5fd92f28ac2259ed33d22d7ab9094d09b80db9e494f30da3cdb" gracePeriod=60
Sep 16 16:46:50.243712 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:46:50.243673    2554 secret.go:189] Couldn't get secret e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-self-signed-certs: secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found
Sep 16 16:46:50.244107 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:46:50.243746    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae39005d-19ea-4cba-a75f-884b2c1db905-tls-certs podName:ae39005d-19ea-4cba-a75f-884b2c1db905 nodeName:}" failed. No retries permitted until 2026-09-16 16:46:52.243731836 +0000 UTC m=+3412.688423991 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/ae39005d-19ea-4cba-a75f-884b2c1db905-tls-certs") pod "e2e-del-blocked-svc-kserve-85779c986f-hswjk" (UID: "ae39005d-19ea-4cba-a75f-884b2c1db905") : secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found
Sep 16 16:46:50.776429 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:50.776404    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85779c986f-hswjk"
Sep 16 16:46:50.949604 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:50.949510    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgdx5\" (UniqueName: \"kubernetes.io/projected/ae39005d-19ea-4cba-a75f-884b2c1db905-kube-api-access-pgdx5\") pod \"ae39005d-19ea-4cba-a75f-884b2c1db905\" (UID: \"ae39005d-19ea-4cba-a75f-884b2c1db905\") "
Sep 16 16:46:50.949604 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:50.949577    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae39005d-19ea-4cba-a75f-884b2c1db905-home\") pod \"ae39005d-19ea-4cba-a75f-884b2c1db905\" (UID: \"ae39005d-19ea-4cba-a75f-884b2c1db905\") "
Sep 16 16:46:50.949876 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:50.949607    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae39005d-19ea-4cba-a75f-884b2c1db905-tmp-dir\") pod \"ae39005d-19ea-4cba-a75f-884b2c1db905\" (UID: \"ae39005d-19ea-4cba-a75f-884b2c1db905\") "
Sep 16 16:46:50.949876 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:50.949643    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae39005d-19ea-4cba-a75f-884b2c1db905-kserve-provision-location\") pod \"ae39005d-19ea-4cba-a75f-884b2c1db905\" (UID: \"ae39005d-19ea-4cba-a75f-884b2c1db905\") "
Sep 16 16:46:50.949876 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:50.949678    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae39005d-19ea-4cba-a75f-884b2c1db905-tls-certs\") pod \"ae39005d-19ea-4cba-a75f-884b2c1db905\" (UID: \"ae39005d-19ea-4cba-a75f-884b2c1db905\") "
Sep 16 16:46:50.949876 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:50.949709    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae39005d-19ea-4cba-a75f-884b2c1db905-model-cache\") pod \"ae39005d-19ea-4cba-a75f-884b2c1db905\" (UID: \"ae39005d-19ea-4cba-a75f-884b2c1db905\") "
Sep 16 16:46:50.949876 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:50.949744    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae39005d-19ea-4cba-a75f-884b2c1db905-dshm\") pod \"ae39005d-19ea-4cba-a75f-884b2c1db905\" (UID: \"ae39005d-19ea-4cba-a75f-884b2c1db905\") "
Sep 16 16:46:50.950149 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:50.949938    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae39005d-19ea-4cba-a75f-884b2c1db905-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ae39005d-19ea-4cba-a75f-884b2c1db905" (UID: "ae39005d-19ea-4cba-a75f-884b2c1db905"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:46:50.950149 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:50.949951    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae39005d-19ea-4cba-a75f-884b2c1db905-home" (OuterVolumeSpecName: "home") pod "ae39005d-19ea-4cba-a75f-884b2c1db905" (UID: "ae39005d-19ea-4cba-a75f-884b2c1db905"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:46:50.950149 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:50.950007    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae39005d-19ea-4cba-a75f-884b2c1db905-model-cache" (OuterVolumeSpecName: "model-cache") pod "ae39005d-19ea-4cba-a75f-884b2c1db905" (UID: "ae39005d-19ea-4cba-a75f-884b2c1db905"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:46:50.951919 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:50.951895    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae39005d-19ea-4cba-a75f-884b2c1db905-dshm" (OuterVolumeSpecName: "dshm") pod "ae39005d-19ea-4cba-a75f-884b2c1db905" (UID: "ae39005d-19ea-4cba-a75f-884b2c1db905"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:46:50.952028 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:50.951975    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae39005d-19ea-4cba-a75f-884b2c1db905-kube-api-access-pgdx5" (OuterVolumeSpecName: "kube-api-access-pgdx5") pod "ae39005d-19ea-4cba-a75f-884b2c1db905" (UID: "ae39005d-19ea-4cba-a75f-884b2c1db905"). InnerVolumeSpecName "kube-api-access-pgdx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:46:50.952028 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:50.951989    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae39005d-19ea-4cba-a75f-884b2c1db905-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ae39005d-19ea-4cba-a75f-884b2c1db905" (UID: "ae39005d-19ea-4cba-a75f-884b2c1db905"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:46:51.019285 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:51.019216    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae39005d-19ea-4cba-a75f-884b2c1db905-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ae39005d-19ea-4cba-a75f-884b2c1db905" (UID: "ae39005d-19ea-4cba-a75f-884b2c1db905"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:46:51.051278 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:51.051236    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae39005d-19ea-4cba-a75f-884b2c1db905-tls-certs\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:46:51.051278 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:51.051276    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae39005d-19ea-4cba-a75f-884b2c1db905-model-cache\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:46:51.051278 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:51.051285    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae39005d-19ea-4cba-a75f-884b2c1db905-dshm\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:46:51.051487 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:51.051295    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgdx5\" (UniqueName: \"kubernetes.io/projected/ae39005d-19ea-4cba-a75f-884b2c1db905-kube-api-access-pgdx5\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:46:51.051487 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:51.051304    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae39005d-19ea-4cba-a75f-884b2c1db905-home\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:46:51.051487 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:51.051313    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae39005d-19ea-4cba-a75f-884b2c1db905-tmp-dir\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:46:51.051487 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:51.051322    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae39005d-19ea-4cba-a75f-884b2c1db905-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:46:51.641517 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:51.641480    2554 generic.go:356] "Generic (PLEG): container finished" podID="ae39005d-19ea-4cba-a75f-884b2c1db905" containerID="714fd107136ce5fd92f28ac2259ed33d22d7ab9094d09b80db9e494f30da3cdb" exitCode=0
Sep 16 16:46:51.642075 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:51.641536    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85779c986f-hswjk"
Sep 16 16:46:51.642075 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:51.641567    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85779c986f-hswjk" event={"ID":"ae39005d-19ea-4cba-a75f-884b2c1db905","Type":"ContainerDied","Data":"714fd107136ce5fd92f28ac2259ed33d22d7ab9094d09b80db9e494f30da3cdb"}
Sep 16 16:46:51.642075 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:51.641630    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85779c986f-hswjk" event={"ID":"ae39005d-19ea-4cba-a75f-884b2c1db905","Type":"ContainerDied","Data":"c65c0f9f470953ba42fbc9b419fa977d04c7e4139e3323307f42348f2d81fd78"}
Sep 16 16:46:51.642075 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:51.641655    2554 scope.go:117] "RemoveContainer" containerID="714fd107136ce5fd92f28ac2259ed33d22d7ab9094d09b80db9e494f30da3cdb"
Sep 16 16:46:51.650297 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:51.650160    2554 scope.go:117] "RemoveContainer" containerID="714fd107136ce5fd92f28ac2259ed33d22d7ab9094d09b80db9e494f30da3cdb"
Sep 16 16:46:51.650403 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:46:51.650380    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"714fd107136ce5fd92f28ac2259ed33d22d7ab9094d09b80db9e494f30da3cdb\": container with ID starting with 714fd107136ce5fd92f28ac2259ed33d22d7ab9094d09b80db9e494f30da3cdb not found: ID does not exist" containerID="714fd107136ce5fd92f28ac2259ed33d22d7ab9094d09b80db9e494f30da3cdb"
Sep 16 16:46:51.650443 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:51.650411    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"714fd107136ce5fd92f28ac2259ed33d22d7ab9094d09b80db9e494f30da3cdb"} err="failed to get container status \"714fd107136ce5fd92f28ac2259ed33d22d7ab9094d09b80db9e494f30da3cdb\": rpc error: code = NotFound desc = could not find container \"714fd107136ce5fd92f28ac2259ed33d22d7ab9094d09b80db9e494f30da3cdb\": container with ID starting with 714fd107136ce5fd92f28ac2259ed33d22d7ab9094d09b80db9e494f30da3cdb not found: ID does not exist"
Sep 16 16:46:51.680935 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:51.680900    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85779c986f-hswjk"]
Sep 16 16:46:51.683606 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:51.683581    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85779c986f-hswjk"]
Sep 16 16:46:52.076420 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:52.076387    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae39005d-19ea-4cba-a75f-884b2c1db905" path="/var/lib/kubelet/pods/ae39005d-19ea-4cba-a75f-884b2c1db905/volumes"
Sep 16 16:46:52.735392 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:52.735359    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:46:52.836050 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:52.836016    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:46:52.952212 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:52.952179    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:46:53.555899 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:53.555864    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:46:56.089227 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:56.089189    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:46:56.189630 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:56.189599    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:46:57.186128 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:57.186095    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:46:57.431225 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:57.431190    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:46:57.532410 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:57.532376    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:46:59.274532 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:59.274497    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:46:59.374148 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:46:59.374119    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:47:02.481660 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:02.481630    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:47:02.576530 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:02.576499    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:47:02.685091 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:02.685062    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:47:04.264066 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:04.264033    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:47:04.360357 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:04.360312    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:47:06.877083 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:06.877050    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k"]
Sep 16 16:47:06.877555 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:06.877361    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae39005d-19ea-4cba-a75f-884b2c1db905" containerName="storage-initializer"
Sep 16 16:47:06.877555 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:06.877380    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae39005d-19ea-4cba-a75f-884b2c1db905" containerName="storage-initializer"
Sep 16 16:47:06.877555 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:06.877431    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae39005d-19ea-4cba-a75f-884b2c1db905" containerName="storage-initializer"
Sep 16 16:47:06.880853 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:06.880832    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k"
Sep 16 16:47:06.883889 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:06.883863    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-ms98d\""
Sep 16 16:47:06.884036 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:06.883876    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 16 16:47:06.884036 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:06.883933    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 16 16:47:06.884036 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:06.883931    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 16 16:47:06.890959 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:06.890934    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k"]
Sep 16 16:47:06.904930 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:06.904903    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:47:06.961625 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:06.961590    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30398541-3000-4728-8f04-62e27ce182e5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k\" (UID: \"30398541-3000-4728-8f04-62e27ce182e5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k"
Sep 16 16:47:06.961625 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:06.961626    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30398541-3000-4728-8f04-62e27ce182e5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k\" (UID: \"30398541-3000-4728-8f04-62e27ce182e5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k"
Sep 16 16:47:06.961864 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:06.961670    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30398541-3000-4728-8f04-62e27ce182e5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k\" (UID: \"30398541-3000-4728-8f04-62e27ce182e5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k"
Sep 16 16:47:06.961864 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:06.961688    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq7g7\" (UniqueName: \"kubernetes.io/projected/30398541-3000-4728-8f04-62e27ce182e5-kube-api-access-tq7g7\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k\" (UID: \"30398541-3000-4728-8f04-62e27ce182e5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k"
Sep 16 16:47:06.961864 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:06.961708    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30398541-3000-4728-8f04-62e27ce182e5-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k\" (UID: \"30398541-3000-4728-8f04-62e27ce182e5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k"
Sep 16 16:47:06.961864 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:06.961728    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30398541-3000-4728-8f04-62e27ce182e5-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k\" (UID: \"30398541-3000-4728-8f04-62e27ce182e5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k"
Sep 16 16:47:06.961864 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:06.961784    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30398541-3000-4728-8f04-62e27ce182e5-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k\" (UID: \"30398541-3000-4728-8f04-62e27ce182e5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k"
Sep 16 16:47:07.063127 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:07.063084    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30398541-3000-4728-8f04-62e27ce182e5-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k\" (UID: \"30398541-3000-4728-8f04-62e27ce182e5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k"
Sep 16 16:47:07.063335 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:07.063133    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30398541-3000-4728-8f04-62e27ce182e5-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k\" (UID: \"30398541-3000-4728-8f04-62e27ce182e5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k"
Sep 16 16:47:07.063335 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:07.063160    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30398541-3000-4728-8f04-62e27ce182e5-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k\" (UID: \"30398541-3000-4728-8f04-62e27ce182e5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k"
Sep 16 16:47:07.063335 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:07.063217    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30398541-3000-4728-8f04-62e27ce182e5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k\" (UID: \"30398541-3000-4728-8f04-62e27ce182e5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k"
Sep 16 16:47:07.063335 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:07.063242    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30398541-3000-4728-8f04-62e27ce182e5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k\" (UID: \"30398541-3000-4728-8f04-62e27ce182e5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k"
Sep 16 16:47:07.063335 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:07.063285    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30398541-3000-4728-8f04-62e27ce182e5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k\" (UID: \"30398541-3000-4728-8f04-62e27ce182e5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k"
Sep 16 16:47:07.063335 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:07.063317    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq7g7\" (UniqueName: \"kubernetes.io/projected/30398541-3000-4728-8f04-62e27ce182e5-kube-api-access-tq7g7\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k\" (UID: \"30398541-3000-4728-8f04-62e27ce182e5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k"
Sep 16 16:47:07.063630 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:07.063573    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30398541-3000-4728-8f04-62e27ce182e5-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k\" (UID: \"30398541-3000-4728-8f04-62e27ce182e5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k"
Sep 16 16:47:07.063630 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:07.063604    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30398541-3000-4728-8f04-62e27ce182e5-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k\" (UID: \"30398541-3000-4728-8f04-62e27ce182e5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k"
Sep 16 16:47:07.063728 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:07.063640    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30398541-3000-4728-8f04-62e27ce182e5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k\" (UID: \"30398541-3000-4728-8f04-62e27ce182e5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k"
Sep 16 16:47:07.063728 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:07.063709    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30398541-3000-4728-8f04-62e27ce182e5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k\" (UID: \"30398541-3000-4728-8f04-62e27ce182e5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k"
Sep 16 16:47:07.065505 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:07.065484    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30398541-3000-4728-8f04-62e27ce182e5-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k\" (UID: \"30398541-3000-4728-8f04-62e27ce182e5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k"
Sep 16 16:47:07.065780 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:07.065761    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30398541-3000-4728-8f04-62e27ce182e5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k\" (UID: \"30398541-3000-4728-8f04-62e27ce182e5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k"
Sep 16 16:47:07.071883 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:07.071858    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq7g7\" (UniqueName: \"kubernetes.io/projected/30398541-3000-4728-8f04-62e27ce182e5-kube-api-access-tq7g7\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k\" (UID: \"30398541-3000-4728-8f04-62e27ce182e5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k"
Sep 16 16:47:07.192045 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:07.191951    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k"
Sep 16 16:47:07.214377 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:07.214350    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:47:07.468024 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:07.467960    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k"]
Sep 16 16:47:07.512427 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:07.512405    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 16:47:07.701886 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:07.701848    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k" event={"ID":"30398541-3000-4728-8f04-62e27ce182e5","Type":"ContainerStarted","Data":"2f3006200ae9e52e8388bbf5207279c95ca3269834c28bf1fe6b12aa72ac5585"}
Sep 16 16:47:09.746218 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:09.746181    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:47:09.845298 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:09.845263    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:47:10.081589 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:10.081558    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:47:10.152060 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:10.152023    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:47:13.458073 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:13.458039    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:47:16.734998 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:16.734962    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k" event={"ID":"30398541-3000-4728-8f04-62e27ce182e5","Type":"ContainerStarted","Data":"f6385022c120bc0042fd116757b3e255179b4cf4fc949e70708288d8be94f888"}
Sep 16 16:47:17.740514 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:17.740478    2554 generic.go:356] "Generic (PLEG): container finished" podID="30398541-3000-4728-8f04-62e27ce182e5" containerID="f6385022c120bc0042fd116757b3e255179b4cf4fc949e70708288d8be94f888" exitCode=0
Sep 16 16:47:17.740918 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:17.740547    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k" event={"ID":"30398541-3000-4728-8f04-62e27ce182e5","Type":"ContainerDied","Data":"f6385022c120bc0042fd116757b3e255179b4cf4fc949e70708288d8be94f888"}
Sep 16 16:47:17.905698 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:17.905659    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:47:18.004044 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:18.003960    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:47:18.747473 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:18.747388    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k" event={"ID":"30398541-3000-4728-8f04-62e27ce182e5","Type":"ContainerStarted","Data":"a188d61cce58e34a7891170c434a2c9f2e6001e9f1f1587f69e73eb3c5b087b1"}
Sep 16 16:47:18.747473 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:18.747442    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k" event={"ID":"30398541-3000-4728-8f04-62e27ce182e5","Type":"ContainerStarted","Data":"4ee772c91c9dec2fecf6221efac4aa8e1caa0c9a9c7c3ef96eff0ec04cc066ba"}
Sep 16 16:47:18.771406 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:18.771347    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k" podStartSLOduration=3.134318754 podStartE2EDuration="12.771330227s" totalImagesPullingTime="8.451903069s" totalInitContainerRuntime="1.185108404s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:47:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:47:18.769129294 +0000 UTC m=+3439.213821491" watchObservedRunningTime="2026-09-16 16:47:18.771330227 +0000 UTC m=+3439.216022406"
Sep 16 16:47:19.192495 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:19.192455    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k"
Sep 16 16:47:20.731095 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:20.731050    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:47:20.974053 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:20.974003    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:47:27.192811 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:27.192770    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k"
Sep 16 16:47:27.194306 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:27.194280    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k" podUID="30398541-3000-4728-8f04-62e27ce182e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Sep 16 16:47:28.350490 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:28.350452    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:47:37.192969 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:37.192924    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k" podUID="30398541-3000-4728-8f04-62e27ce182e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Sep 16 16:47:42.159000 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:42.158965    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:47:46.175109 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:46.175074    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:47:47.192409 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:47.192368    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k" podUID="30398541-3000-4728-8f04-62e27ce182e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Sep 16 16:47:57.193248 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:47:57.193178    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k" podUID="30398541-3000-4728-8f04-62e27ce182e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Sep 16 16:48:07.193143 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:48:07.193096    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k" podUID="30398541-3000-4728-8f04-62e27ce182e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Sep 16 16:48:17.193491 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:48:17.193435    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k" podUID="30398541-3000-4728-8f04-62e27ce182e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Sep 16 16:48:27.193260 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:48:27.193164    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k" podUID="30398541-3000-4728-8f04-62e27ce182e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Sep 16 16:48:37.193135 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:48:37.193088    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k" podUID="30398541-3000-4728-8f04-62e27ce182e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Sep 16 16:48:46.448377 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:48:46.448341    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:48:47.193463 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:48:47.193413    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k" podUID="30398541-3000-4728-8f04-62e27ce182e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Sep 16 16:48:57.193373 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:48:57.193334    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k" podUID="30398541-3000-4728-8f04-62e27ce182e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Sep 16 16:49:07.193344 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:49:07.193295    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k" podUID="30398541-3000-4728-8f04-62e27ce182e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Sep 16 16:49:12.708166 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:49:12.708127    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:49:12.771494 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:49:12.771464    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:49:12.827635 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:49:12.827604    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:49:17.202756 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:49:17.202727    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k"
Sep 16 16:49:17.210723 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:49:17.210694    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k"
Sep 16 16:49:41.720084 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:49:41.720051    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:49:43.268477 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:49:43.268445    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:49:44.790967 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:49:44.790931    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:49:44.798006 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:49:44.797978    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k"]
Sep 16 16:49:44.798297 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:49:44.798245    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k" podUID="30398541-3000-4728-8f04-62e27ce182e5" containerName="modelcar" containerID="cri-o://a188d61cce58e34a7891170c434a2c9f2e6001e9f1f1587f69e73eb3c5b087b1" gracePeriod=60
Sep 16 16:49:44.920250 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:49:44.920216    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:49:45.310925 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:49:45.310857    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k_30398541-3000-4728-8f04-62e27ce182e5/modelcar/0.log"
Sep 16 16:49:45.311615 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:49:45.311593    2554 generic.go:356] "Generic (PLEG): container finished" podID="30398541-3000-4728-8f04-62e27ce182e5" containerID="a188d61cce58e34a7891170c434a2c9f2e6001e9f1f1587f69e73eb3c5b087b1" exitCode=143
Sep 16 16:49:45.311713 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:49:45.311633    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k" event={"ID":"30398541-3000-4728-8f04-62e27ce182e5","Type":"ContainerDied","Data":"a188d61cce58e34a7891170c434a2c9f2e6001e9f1f1587f69e73eb3c5b087b1"}
Sep 16 16:49:52.337405 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:49:52.337324    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:49:59.819121 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:49:59.819074    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k" podUID="30398541-3000-4728-8f04-62e27ce182e5" containerName="main" containerID="cri-o://4ee772c91c9dec2fecf6221efac4aa8e1caa0c9a9c7c3ef96eff0ec04cc066ba" gracePeriod=45
Sep 16 16:50:00.203802 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:00.203695    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k" podUID="30398541-3000-4728-8f04-62e27ce182e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Sep 16 16:50:00.221316 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:00.221288    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k_30398541-3000-4728-8f04-62e27ce182e5/modelcar/0.log"
Sep 16 16:50:00.235676 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:00.235650    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k_30398541-3000-4728-8f04-62e27ce182e5/modelcar/0.log"
Sep 16 16:50:00.368550 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:00.368526    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k_30398541-3000-4728-8f04-62e27ce182e5/modelcar/0.log"
Sep 16 16:50:00.369165 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:00.369145    2554 generic.go:356] "Generic (PLEG): container finished" podID="30398541-3000-4728-8f04-62e27ce182e5" containerID="4ee772c91c9dec2fecf6221efac4aa8e1caa0c9a9c7c3ef96eff0ec04cc066ba" exitCode=0
Sep 16 16:50:00.369230 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:00.369201    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k" event={"ID":"30398541-3000-4728-8f04-62e27ce182e5","Type":"ContainerDied","Data":"4ee772c91c9dec2fecf6221efac4aa8e1caa0c9a9c7c3ef96eff0ec04cc066ba"}
Sep 16 16:50:01.069832 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:01.069809    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k_30398541-3000-4728-8f04-62e27ce182e5/modelcar/0.log"
Sep 16 16:50:01.071405 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:01.071388    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k"
Sep 16 16:50:01.212342 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:01.212246    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30398541-3000-4728-8f04-62e27ce182e5-tls-certs\") pod \"30398541-3000-4728-8f04-62e27ce182e5\" (UID: \"30398541-3000-4728-8f04-62e27ce182e5\") "
Sep 16 16:50:01.212342 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:01.212295    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30398541-3000-4728-8f04-62e27ce182e5-kserve-provision-location\") pod \"30398541-3000-4728-8f04-62e27ce182e5\" (UID: \"30398541-3000-4728-8f04-62e27ce182e5\") "
Sep 16 16:50:01.212342 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:01.212319    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30398541-3000-4728-8f04-62e27ce182e5-tmp-dir\") pod \"30398541-3000-4728-8f04-62e27ce182e5\" (UID: \"30398541-3000-4728-8f04-62e27ce182e5\") "
Sep 16 16:50:01.212342 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:01.212343    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30398541-3000-4728-8f04-62e27ce182e5-model-cache\") pod \"30398541-3000-4728-8f04-62e27ce182e5\" (UID: \"30398541-3000-4728-8f04-62e27ce182e5\") "
Sep 16 16:50:01.212686 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:01.212402    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tq7g7\" (UniqueName: \"kubernetes.io/projected/30398541-3000-4728-8f04-62e27ce182e5-kube-api-access-tq7g7\") pod \"30398541-3000-4728-8f04-62e27ce182e5\" (UID: \"30398541-3000-4728-8f04-62e27ce182e5\") "
Sep 16 16:50:01.212686 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:01.212447    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30398541-3000-4728-8f04-62e27ce182e5-home\") pod \"30398541-3000-4728-8f04-62e27ce182e5\" (UID: \"30398541-3000-4728-8f04-62e27ce182e5\") "
Sep 16 16:50:01.212686 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:01.212473    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30398541-3000-4728-8f04-62e27ce182e5-dshm\") pod \"30398541-3000-4728-8f04-62e27ce182e5\" (UID: \"30398541-3000-4728-8f04-62e27ce182e5\") "
Sep 16 16:50:01.212686 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:01.212602    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30398541-3000-4728-8f04-62e27ce182e5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "30398541-3000-4728-8f04-62e27ce182e5" (UID: "30398541-3000-4728-8f04-62e27ce182e5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:50:01.212686 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:01.212657    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30398541-3000-4728-8f04-62e27ce182e5-model-cache" (OuterVolumeSpecName: "model-cache") pod "30398541-3000-4728-8f04-62e27ce182e5" (UID: "30398541-3000-4728-8f04-62e27ce182e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:50:01.213155 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:01.213105    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30398541-3000-4728-8f04-62e27ce182e5-home" (OuterVolumeSpecName: "home") pod "30398541-3000-4728-8f04-62e27ce182e5" (UID: "30398541-3000-4728-8f04-62e27ce182e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:50:01.214633 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:01.214605    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30398541-3000-4728-8f04-62e27ce182e5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "30398541-3000-4728-8f04-62e27ce182e5" (UID: "30398541-3000-4728-8f04-62e27ce182e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:50:01.214727 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:01.214673    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30398541-3000-4728-8f04-62e27ce182e5-dshm" (OuterVolumeSpecName: "dshm") pod "30398541-3000-4728-8f04-62e27ce182e5" (UID: "30398541-3000-4728-8f04-62e27ce182e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:50:01.214727 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:01.214685    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30398541-3000-4728-8f04-62e27ce182e5-kube-api-access-tq7g7" (OuterVolumeSpecName: "kube-api-access-tq7g7") pod "30398541-3000-4728-8f04-62e27ce182e5" (UID: "30398541-3000-4728-8f04-62e27ce182e5"). InnerVolumeSpecName "kube-api-access-tq7g7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:50:01.226915 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:01.226887    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30398541-3000-4728-8f04-62e27ce182e5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "30398541-3000-4728-8f04-62e27ce182e5" (UID: "30398541-3000-4728-8f04-62e27ce182e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:50:01.313269 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:01.313237    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30398541-3000-4728-8f04-62e27ce182e5-home\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:50:01.313269 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:01.313265    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30398541-3000-4728-8f04-62e27ce182e5-dshm\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:50:01.313269 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:01.313274    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30398541-3000-4728-8f04-62e27ce182e5-tls-certs\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:50:01.313474 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:01.313285    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30398541-3000-4728-8f04-62e27ce182e5-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:50:01.313474 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:01.313294    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30398541-3000-4728-8f04-62e27ce182e5-tmp-dir\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:50:01.313474 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:01.313302    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30398541-3000-4728-8f04-62e27ce182e5-model-cache\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:50:01.313474 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:01.313311    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq7g7\" (UniqueName: \"kubernetes.io/projected/30398541-3000-4728-8f04-62e27ce182e5-kube-api-access-tq7g7\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:50:01.374253 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:01.374228    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k_30398541-3000-4728-8f04-62e27ce182e5/modelcar/0.log"
Sep 16 16:50:01.374999 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:01.374978    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k" event={"ID":"30398541-3000-4728-8f04-62e27ce182e5","Type":"ContainerDied","Data":"2f3006200ae9e52e8388bbf5207279c95ca3269834c28bf1fe6b12aa72ac5585"}
Sep 16 16:50:01.375098 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:01.375039    2554 scope.go:117] "RemoveContainer" containerID="a188d61cce58e34a7891170c434a2c9f2e6001e9f1f1587f69e73eb3c5b087b1"
Sep 16 16:50:01.375098 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:01.374983    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k"
Sep 16 16:50:01.383349 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:01.383331    2554 scope.go:117] "RemoveContainer" containerID="4ee772c91c9dec2fecf6221efac4aa8e1caa0c9a9c7c3ef96eff0ec04cc066ba"
Sep 16 16:50:01.390211 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:01.390197    2554 scope.go:117] "RemoveContainer" containerID="f6385022c120bc0042fd116757b3e255179b4cf4fc949e70708288d8be94f888"
Sep 16 16:50:01.395377 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:01.395355    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k"]
Sep 16 16:50:01.398783 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:01.398765    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dmxp6k"]
Sep 16 16:50:02.076121 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:02.076088    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30398541-3000-4728-8f04-62e27ce182e5" path="/var/lib/kubelet/pods/30398541-3000-4728-8f04-62e27ce182e5/volumes"
Sep 16 16:50:32.443689 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:32.443653    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:50:32.536133 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:32.536100    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:50:35.568021 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:35.567986    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5"]
Sep 16 16:50:35.568379 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:35.568309    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30398541-3000-4728-8f04-62e27ce182e5" containerName="modelcar"
Sep 16 16:50:35.568379 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:35.568324    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="30398541-3000-4728-8f04-62e27ce182e5" containerName="modelcar"
Sep 16 16:50:35.568379 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:35.568342    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30398541-3000-4728-8f04-62e27ce182e5" containerName="modelcar-init"
Sep 16 16:50:35.568379 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:35.568347    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="30398541-3000-4728-8f04-62e27ce182e5" containerName="modelcar-init"
Sep 16 16:50:35.568379 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:35.568355    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30398541-3000-4728-8f04-62e27ce182e5" containerName="main"
Sep 16 16:50:35.568379 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:35.568362    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="30398541-3000-4728-8f04-62e27ce182e5" containerName="main"
Sep 16 16:50:35.568580 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:35.568400    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="30398541-3000-4728-8f04-62e27ce182e5" containerName="main"
Sep 16 16:50:35.568580 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:35.568406    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="30398541-3000-4728-8f04-62e27ce182e5" containerName="modelcar"
Sep 16 16:50:35.573760 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:35.573742    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5"
Sep 16 16:50:35.577115 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:35.577090    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 16 16:50:35.577416 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:35.577396    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 16:50:35.577555 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:35.577536    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 16 16:50:35.577631 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:35.577618    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 16 16:50:35.577856 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:35.577837    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-kllwv\""
Sep 16 16:50:35.578424 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:35.578407    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"seaweedfs-s3-creds\""
Sep 16 16:50:35.584640 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:35.584617    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5"]
Sep 16 16:50:35.617227 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:35.617202    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:50:35.759841 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:35.759808    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d9a4106-66f3-40ca-a352-8265223e0702-dshm\") pod \"custom-route-timeout-test-kserve-695b897cc4-mdwx5\" (UID: \"4d9a4106-66f3-40ca-a352-8265223e0702\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5"
Sep 16 16:50:35.759976 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:35.759848    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d9a4106-66f3-40ca-a352-8265223e0702-home\") pod \"custom-route-timeout-test-kserve-695b897cc4-mdwx5\" (UID: \"4d9a4106-66f3-40ca-a352-8265223e0702\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5"
Sep 16 16:50:35.759976 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:35.759881    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d9a4106-66f3-40ca-a352-8265223e0702-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-695b897cc4-mdwx5\" (UID: \"4d9a4106-66f3-40ca-a352-8265223e0702\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5"
Sep 16 16:50:35.759976 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:35.759898    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d9a4106-66f3-40ca-a352-8265223e0702-cabundle-cert\") pod \"custom-route-timeout-test-kserve-695b897cc4-mdwx5\" (UID: \"4d9a4106-66f3-40ca-a352-8265223e0702\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5"
Sep 16 16:50:35.759976 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:35.759921    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d9a4106-66f3-40ca-a352-8265223e0702-model-cache\") pod \"custom-route-timeout-test-kserve-695b897cc4-mdwx5\" (UID: \"4d9a4106-66f3-40ca-a352-8265223e0702\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5"
Sep 16 16:50:35.759976 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:35.759958    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d9a4106-66f3-40ca-a352-8265223e0702-tls-certs\") pod \"custom-route-timeout-test-kserve-695b897cc4-mdwx5\" (UID: \"4d9a4106-66f3-40ca-a352-8265223e0702\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5"
Sep 16 16:50:35.760162 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:35.759977    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d9a4106-66f3-40ca-a352-8265223e0702-tmp-dir\") pod \"custom-route-timeout-test-kserve-695b897cc4-mdwx5\" (UID: \"4d9a4106-66f3-40ca-a352-8265223e0702\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5"
Sep 16 16:50:35.760162 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:35.759997    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n695\" (UniqueName: \"kubernetes.io/projected/4d9a4106-66f3-40ca-a352-8265223e0702-kube-api-access-8n695\") pod \"custom-route-timeout-test-kserve-695b897cc4-mdwx5\" (UID: \"4d9a4106-66f3-40ca-a352-8265223e0702\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5"
Sep 16 16:50:35.861494 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:35.861414    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d9a4106-66f3-40ca-a352-8265223e0702-dshm\") pod \"custom-route-timeout-test-kserve-695b897cc4-mdwx5\" (UID: \"4d9a4106-66f3-40ca-a352-8265223e0702\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5"
Sep 16 16:50:35.861494 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:35.861470    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d9a4106-66f3-40ca-a352-8265223e0702-home\") pod \"custom-route-timeout-test-kserve-695b897cc4-mdwx5\" (UID: \"4d9a4106-66f3-40ca-a352-8265223e0702\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5"
Sep 16 16:50:35.861709 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:35.861509    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d9a4106-66f3-40ca-a352-8265223e0702-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-695b897cc4-mdwx5\" (UID: \"4d9a4106-66f3-40ca-a352-8265223e0702\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5"
Sep 16 16:50:35.861709 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:35.861533    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d9a4106-66f3-40ca-a352-8265223e0702-cabundle-cert\") pod \"custom-route-timeout-test-kserve-695b897cc4-mdwx5\" (UID: \"4d9a4106-66f3-40ca-a352-8265223e0702\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5"
Sep 16 16:50:35.861709 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:35.861571    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d9a4106-66f3-40ca-a352-8265223e0702-model-cache\") pod \"custom-route-timeout-test-kserve-695b897cc4-mdwx5\" (UID: \"4d9a4106-66f3-40ca-a352-8265223e0702\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5"
Sep 16 16:50:35.861709 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:35.861608    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d9a4106-66f3-40ca-a352-8265223e0702-tls-certs\") pod \"custom-route-timeout-test-kserve-695b897cc4-mdwx5\" (UID: \"4d9a4106-66f3-40ca-a352-8265223e0702\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5"
Sep 16 16:50:35.861709 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:35.861639    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d9a4106-66f3-40ca-a352-8265223e0702-tmp-dir\") pod \"custom-route-timeout-test-kserve-695b897cc4-mdwx5\" (UID: \"4d9a4106-66f3-40ca-a352-8265223e0702\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5"
Sep 16 16:50:35.861709 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:35.861675    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n695\" (UniqueName: \"kubernetes.io/projected/4d9a4106-66f3-40ca-a352-8265223e0702-kube-api-access-8n695\") pod \"custom-route-timeout-test-kserve-695b897cc4-mdwx5\" (UID: \"4d9a4106-66f3-40ca-a352-8265223e0702\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5"
Sep 16 16:50:35.862049 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:35.861893    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d9a4106-66f3-40ca-a352-8265223e0702-home\") pod \"custom-route-timeout-test-kserve-695b897cc4-mdwx5\" (UID: \"4d9a4106-66f3-40ca-a352-8265223e0702\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5"
Sep 16 16:50:35.862192 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:35.862169    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d9a4106-66f3-40ca-a352-8265223e0702-tmp-dir\") pod \"custom-route-timeout-test-kserve-695b897cc4-mdwx5\" (UID: \"4d9a4106-66f3-40ca-a352-8265223e0702\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5"
Sep 16 16:50:35.862363 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:35.862315    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d9a4106-66f3-40ca-a352-8265223e0702-cabundle-cert\") pod \"custom-route-timeout-test-kserve-695b897cc4-mdwx5\" (UID: \"4d9a4106-66f3-40ca-a352-8265223e0702\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5"
Sep 16 16:50:35.862363 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:35.862328    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d9a4106-66f3-40ca-a352-8265223e0702-model-cache\") pod \"custom-route-timeout-test-kserve-695b897cc4-mdwx5\" (UID: \"4d9a4106-66f3-40ca-a352-8265223e0702\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5"
Sep 16 16:50:35.862629 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:35.862602    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d9a4106-66f3-40ca-a352-8265223e0702-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-695b897cc4-mdwx5\" (UID: \"4d9a4106-66f3-40ca-a352-8265223e0702\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5"
Sep 16 16:50:35.863905 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:35.863882    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d9a4106-66f3-40ca-a352-8265223e0702-dshm\") pod \"custom-route-timeout-test-kserve-695b897cc4-mdwx5\" (UID: \"4d9a4106-66f3-40ca-a352-8265223e0702\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5"
Sep 16 16:50:35.864439 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:35.864418    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d9a4106-66f3-40ca-a352-8265223e0702-tls-certs\") pod \"custom-route-timeout-test-kserve-695b897cc4-mdwx5\" (UID: \"4d9a4106-66f3-40ca-a352-8265223e0702\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5"
Sep 16 16:50:35.877076 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:35.877049    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n695\" (UniqueName: \"kubernetes.io/projected/4d9a4106-66f3-40ca-a352-8265223e0702-kube-api-access-8n695\") pod \"custom-route-timeout-test-kserve-695b897cc4-mdwx5\" (UID: \"4d9a4106-66f3-40ca-a352-8265223e0702\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5"
Sep 16 16:50:35.884927 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:35.884904    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5"
Sep 16 16:50:35.914836 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:35.914788    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:50:36.181886 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:36.181854    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5"]
Sep 16 16:50:36.183501 ip-10-0-136-228 kubenswrapper[2554]: W0916 16:50:36.183472    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d9a4106_66f3_40ca_a352_8265223e0702.slice/crio-63dc2a09ba397e9d113a1d488f271e07936c3e561d950adb8d17a90011e42ab6 WatchSource:0}: Error finding container 63dc2a09ba397e9d113a1d488f271e07936c3e561d950adb8d17a90011e42ab6: Status 404 returned error can't find the container with id 63dc2a09ba397e9d113a1d488f271e07936c3e561d950adb8d17a90011e42ab6
Sep 16 16:50:36.500054 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:36.499971    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5" event={"ID":"4d9a4106-66f3-40ca-a352-8265223e0702","Type":"ContainerStarted","Data":"915696e0ac2e80a4deccd15c4c48bd8aa716e2b215287a34ba14dd4fb9ce9ca6"}
Sep 16 16:50:36.500054 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:36.500022    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5" event={"ID":"4d9a4106-66f3-40ca-a352-8265223e0702","Type":"ContainerStarted","Data":"63dc2a09ba397e9d113a1d488f271e07936c3e561d950adb8d17a90011e42ab6"}
Sep 16 16:50:38.510027 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:38.509993    2554 generic.go:356] "Generic (PLEG): container finished" podID="4d9a4106-66f3-40ca-a352-8265223e0702" containerID="915696e0ac2e80a4deccd15c4c48bd8aa716e2b215287a34ba14dd4fb9ce9ca6" exitCode=0
Sep 16 16:50:38.510421 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:38.510064    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5" event={"ID":"4d9a4106-66f3-40ca-a352-8265223e0702","Type":"ContainerDied","Data":"915696e0ac2e80a4deccd15c4c48bd8aa716e2b215287a34ba14dd4fb9ce9ca6"}
Sep 16 16:50:38.849677 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:38.849639    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:50:39.516948 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:39.516908    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5" event={"ID":"4d9a4106-66f3-40ca-a352-8265223e0702","Type":"ContainerStarted","Data":"b13902a3742d63fea48fe4df463b93f4ab887932e4fc89021c867c5635c841a5"}
Sep 16 16:50:39.542762 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:39.542699    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5" podStartSLOduration=2.407630819 podStartE2EDuration="4.542680758s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.135049939s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:50:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:50:39.540849092 +0000 UTC m=+3639.985541293" watchObservedRunningTime="2026-09-16 16:50:39.542680758 +0000 UTC m=+3639.987372937"
Sep 16 16:50:39.885832 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:39.885782    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5"
Sep 16 16:50:43.002441 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:43.002403    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:50:44.445570 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:44.445537    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:50:44.514041 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:44.514003    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:50:45.886104 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:45.886065    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5"
Sep 16 16:50:45.887396 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:45.887365    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5" podUID="4d9a4106-66f3-40ca-a352-8265223e0702" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Sep 16 16:50:52.272247 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:52.272213    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:50:55.885887 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:55.885843    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5" podUID="4d9a4106-66f3-40ca-a352-8265223e0702" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Sep 16 16:50:58.375698 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:58.375664    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:50:58.436863 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:50:58.436829    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:51:05.885700 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:51:05.885661    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5" podUID="4d9a4106-66f3-40ca-a352-8265223e0702" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Sep 16 16:51:15.885370 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:51:15.885313    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5" podUID="4d9a4106-66f3-40ca-a352-8265223e0702" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Sep 16 16:51:25.885783 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:51:25.885691    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5" podUID="4d9a4106-66f3-40ca-a352-8265223e0702" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Sep 16 16:51:33.809584 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:51:33.809552    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:51:33.874214 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:51:33.874182    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:51:35.886332 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:51:35.886288    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5" podUID="4d9a4106-66f3-40ca-a352-8265223e0702" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Sep 16 16:51:45.886181 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:51:45.886141    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5" podUID="4d9a4106-66f3-40ca-a352-8265223e0702" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Sep 16 16:51:47.966500 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:51:47.966465    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:51:55.885699 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:51:55.885653    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5" podUID="4d9a4106-66f3-40ca-a352-8265223e0702" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Sep 16 16:52:05.885394 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:52:05.885348    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5" podUID="4d9a4106-66f3-40ca-a352-8265223e0702" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Sep 16 16:52:15.885956 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:52:15.885910    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5" podUID="4d9a4106-66f3-40ca-a352-8265223e0702" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Sep 16 16:52:23.184329 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:52:23.184292    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:52:23.279454 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:52:23.279419    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:52:25.886435 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:52:25.886391    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5" podUID="4d9a4106-66f3-40ca-a352-8265223e0702" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Sep 16 16:52:29.582279 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:52:29.582240    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:52:35.334650 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:52:35.334617    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:52:35.394341 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:52:35.394316    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:52:35.896134 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:52:35.896104    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5"
Sep 16 16:52:35.904157 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:52:35.904135    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5"
Sep 16 16:52:38.121948 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:52:38.121906    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:52:51.346118 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:52:51.346074    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5"]
Sep 16 16:52:51.351434 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:52:51.351406    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:52:51.413847 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:52:51.413818    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:53:06.368672 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:06.368614    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5" podUID="4d9a4106-66f3-40ca-a352-8265223e0702" containerName="main" containerID="cri-o://b13902a3742d63fea48fe4df463b93f4ab887932e4fc89021c867c5635c841a5" gracePeriod=45
Sep 16 16:53:06.897007 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:06.896974    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5" podUID="4d9a4106-66f3-40ca-a352-8265223e0702" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Sep 16 16:53:07.032494 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:07.032472    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5"
Sep 16 16:53:07.036232 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:07.036211    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d9a4106-66f3-40ca-a352-8265223e0702-tls-certs\") pod \"4d9a4106-66f3-40ca-a352-8265223e0702\" (UID: \"4d9a4106-66f3-40ca-a352-8265223e0702\") "
Sep 16 16:53:07.036298 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:07.036279    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8n695\" (UniqueName: \"kubernetes.io/projected/4d9a4106-66f3-40ca-a352-8265223e0702-kube-api-access-8n695\") pod \"4d9a4106-66f3-40ca-a352-8265223e0702\" (UID: \"4d9a4106-66f3-40ca-a352-8265223e0702\") "
Sep 16 16:53:07.036340 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:07.036314    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d9a4106-66f3-40ca-a352-8265223e0702-model-cache\") pod \"4d9a4106-66f3-40ca-a352-8265223e0702\" (UID: \"4d9a4106-66f3-40ca-a352-8265223e0702\") "
Sep 16 16:53:07.036382 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:07.036370    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d9a4106-66f3-40ca-a352-8265223e0702-dshm\") pod \"4d9a4106-66f3-40ca-a352-8265223e0702\" (UID: \"4d9a4106-66f3-40ca-a352-8265223e0702\") "
Sep 16 16:53:07.036416 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:07.036398    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d9a4106-66f3-40ca-a352-8265223e0702-home\") pod \"4d9a4106-66f3-40ca-a352-8265223e0702\" (UID: \"4d9a4106-66f3-40ca-a352-8265223e0702\") "
Sep 16 16:53:07.036450 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:07.036424    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d9a4106-66f3-40ca-a352-8265223e0702-kserve-provision-location\") pod \"4d9a4106-66f3-40ca-a352-8265223e0702\" (UID: \"4d9a4106-66f3-40ca-a352-8265223e0702\") "
Sep 16 16:53:07.036494 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:07.036482    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d9a4106-66f3-40ca-a352-8265223e0702-cabundle-cert\") pod \"4d9a4106-66f3-40ca-a352-8265223e0702\" (UID: \"4d9a4106-66f3-40ca-a352-8265223e0702\") "
Sep 16 16:53:07.036546 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:07.036506    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d9a4106-66f3-40ca-a352-8265223e0702-tmp-dir\") pod \"4d9a4106-66f3-40ca-a352-8265223e0702\" (UID: \"4d9a4106-66f3-40ca-a352-8265223e0702\") "
Sep 16 16:53:07.036830 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:07.036603    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d9a4106-66f3-40ca-a352-8265223e0702-model-cache" (OuterVolumeSpecName: "model-cache") pod "4d9a4106-66f3-40ca-a352-8265223e0702" (UID: "4d9a4106-66f3-40ca-a352-8265223e0702"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:53:07.036830 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:07.036781    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d9a4106-66f3-40ca-a352-8265223e0702-model-cache\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:53:07.037026 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:07.036936    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d9a4106-66f3-40ca-a352-8265223e0702-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4d9a4106-66f3-40ca-a352-8265223e0702" (UID: "4d9a4106-66f3-40ca-a352-8265223e0702"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 16:53:07.037197 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:07.037170    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d9a4106-66f3-40ca-a352-8265223e0702-home" (OuterVolumeSpecName: "home") pod "4d9a4106-66f3-40ca-a352-8265223e0702" (UID: "4d9a4106-66f3-40ca-a352-8265223e0702"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:53:07.038643 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:07.038573    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d9a4106-66f3-40ca-a352-8265223e0702-kube-api-access-8n695" (OuterVolumeSpecName: "kube-api-access-8n695") pod "4d9a4106-66f3-40ca-a352-8265223e0702" (UID: "4d9a4106-66f3-40ca-a352-8265223e0702"). InnerVolumeSpecName "kube-api-access-8n695". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:53:07.038743 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:07.038676    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d9a4106-66f3-40ca-a352-8265223e0702-dshm" (OuterVolumeSpecName: "dshm") pod "4d9a4106-66f3-40ca-a352-8265223e0702" (UID: "4d9a4106-66f3-40ca-a352-8265223e0702"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:53:07.038787 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:07.038753    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d9a4106-66f3-40ca-a352-8265223e0702-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4d9a4106-66f3-40ca-a352-8265223e0702" (UID: "4d9a4106-66f3-40ca-a352-8265223e0702"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:53:07.054677 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:07.054648    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d9a4106-66f3-40ca-a352-8265223e0702-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4d9a4106-66f3-40ca-a352-8265223e0702" (UID: "4d9a4106-66f3-40ca-a352-8265223e0702"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:53:07.058893 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:07.058867    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d9a4106-66f3-40ca-a352-8265223e0702-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4d9a4106-66f3-40ca-a352-8265223e0702" (UID: "4d9a4106-66f3-40ca-a352-8265223e0702"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:53:07.091093 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:07.091068    2554 generic.go:356] "Generic (PLEG): container finished" podID="4d9a4106-66f3-40ca-a352-8265223e0702" containerID="b13902a3742d63fea48fe4df463b93f4ab887932e4fc89021c867c5635c841a5" exitCode=0
Sep 16 16:53:07.091198 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:07.091103    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5" event={"ID":"4d9a4106-66f3-40ca-a352-8265223e0702","Type":"ContainerDied","Data":"b13902a3742d63fea48fe4df463b93f4ab887932e4fc89021c867c5635c841a5"}
Sep 16 16:53:07.091198 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:07.091120    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5"
Sep 16 16:53:07.091198 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:07.091141    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5" event={"ID":"4d9a4106-66f3-40ca-a352-8265223e0702","Type":"ContainerDied","Data":"63dc2a09ba397e9d113a1d488f271e07936c3e561d950adb8d17a90011e42ab6"}
Sep 16 16:53:07.091198 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:07.091165    2554 scope.go:117] "RemoveContainer" containerID="b13902a3742d63fea48fe4df463b93f4ab887932e4fc89021c867c5635c841a5"
Sep 16 16:53:07.099689 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:07.099672    2554 scope.go:117] "RemoveContainer" containerID="915696e0ac2e80a4deccd15c4c48bd8aa716e2b215287a34ba14dd4fb9ce9ca6"
Sep 16 16:53:07.106854 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:07.106832    2554 scope.go:117] "RemoveContainer" containerID="b13902a3742d63fea48fe4df463b93f4ab887932e4fc89021c867c5635c841a5"
Sep 16 16:53:07.107110 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:53:07.107092    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b13902a3742d63fea48fe4df463b93f4ab887932e4fc89021c867c5635c841a5\": container with ID starting with b13902a3742d63fea48fe4df463b93f4ab887932e4fc89021c867c5635c841a5 not found: ID does not exist" containerID="b13902a3742d63fea48fe4df463b93f4ab887932e4fc89021c867c5635c841a5"
Sep 16 16:53:07.107161 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:07.107117    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b13902a3742d63fea48fe4df463b93f4ab887932e4fc89021c867c5635c841a5"} err="failed to get container status \"b13902a3742d63fea48fe4df463b93f4ab887932e4fc89021c867c5635c841a5\": rpc error: code = NotFound desc = could not find container \"b13902a3742d63fea48fe4df463b93f4ab887932e4fc89021c867c5635c841a5\": container with ID starting with b13902a3742d63fea48fe4df463b93f4ab887932e4fc89021c867c5635c841a5 not found: ID does not exist"
Sep 16 16:53:07.107161 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:07.107131    2554 scope.go:117] "RemoveContainer" containerID="915696e0ac2e80a4deccd15c4c48bd8aa716e2b215287a34ba14dd4fb9ce9ca6"
Sep 16 16:53:07.107371 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:53:07.107351    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"915696e0ac2e80a4deccd15c4c48bd8aa716e2b215287a34ba14dd4fb9ce9ca6\": container with ID starting with 915696e0ac2e80a4deccd15c4c48bd8aa716e2b215287a34ba14dd4fb9ce9ca6 not found: ID does not exist" containerID="915696e0ac2e80a4deccd15c4c48bd8aa716e2b215287a34ba14dd4fb9ce9ca6"
Sep 16 16:53:07.107434 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:07.107380    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"915696e0ac2e80a4deccd15c4c48bd8aa716e2b215287a34ba14dd4fb9ce9ca6"} err="failed to get container status \"915696e0ac2e80a4deccd15c4c48bd8aa716e2b215287a34ba14dd4fb9ce9ca6\": rpc error: code = NotFound desc = could not find container \"915696e0ac2e80a4deccd15c4c48bd8aa716e2b215287a34ba14dd4fb9ce9ca6\": container with ID starting with 915696e0ac2e80a4deccd15c4c48bd8aa716e2b215287a34ba14dd4fb9ce9ca6 not found: ID does not exist"
Sep 16 16:53:07.118447 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:07.118425    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5"]
Sep 16 16:53:07.124706 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:07.124660    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-695b897cc4-mdwx5"]
Sep 16 16:53:07.137376 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:07.137354    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n695\" (UniqueName: \"kubernetes.io/projected/4d9a4106-66f3-40ca-a352-8265223e0702-kube-api-access-8n695\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:53:07.137477 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:07.137378    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d9a4106-66f3-40ca-a352-8265223e0702-dshm\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:53:07.137477 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:07.137393    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d9a4106-66f3-40ca-a352-8265223e0702-home\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:53:07.137477 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:07.137405    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d9a4106-66f3-40ca-a352-8265223e0702-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:53:07.137477 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:07.137418    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d9a4106-66f3-40ca-a352-8265223e0702-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:53:07.137477 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:07.137430    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d9a4106-66f3-40ca-a352-8265223e0702-tmp-dir\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:53:07.137477 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:07.137440    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d9a4106-66f3-40ca-a352-8265223e0702-tls-certs\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:53:08.075934 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:08.075894    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d9a4106-66f3-40ca-a352-8265223e0702" path="/var/lib/kubelet/pods/4d9a4106-66f3-40ca-a352-8265223e0702/volumes"
Sep 16 16:53:41.481418 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:41.481353    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:53:41.698248 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:41.698217    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:53:41.780015 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:41.779938    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:53:44.984315 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:44.984283    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:53:48.386870 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:48.386836    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:53:54.221124 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:54.221089    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:53:54.321550 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:54.321508    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:53:58.255519 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:58.255481    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544"]
Sep 16 16:53:58.256007 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:58.255898    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d9a4106-66f3-40ca-a352-8265223e0702" containerName="storage-initializer"
Sep 16 16:53:58.256007 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:58.255919    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d9a4106-66f3-40ca-a352-8265223e0702" containerName="storage-initializer"
Sep 16 16:53:58.256007 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:58.255934    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d9a4106-66f3-40ca-a352-8265223e0702" containerName="main"
Sep 16 16:53:58.256007 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:58.255943    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d9a4106-66f3-40ca-a352-8265223e0702" containerName="main"
Sep 16 16:53:58.256229 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:58.256015    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d9a4106-66f3-40ca-a352-8265223e0702" containerName="main"
Sep 16 16:53:58.259370 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:58.259345    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544"
Sep 16 16:53:58.262458 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:58.262429    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\""
Sep 16 16:53:58.262458 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:58.262443    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 16:53:58.262673 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:58.262641    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 16 16:53:58.262773 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:58.262727    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-4hrzn\""
Sep 16 16:53:58.262944 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:58.262928    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 16 16:53:58.263683 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:58.263660    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 16 16:53:58.271949 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:58.271926    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544"]
Sep 16 16:53:58.285325 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:58.285288    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84058966-ce0f-42ee-bdf1-729a6c66d43e-dshm\") pod \"router-with-refs-test-kserve-7fbd9694d6-tc544\" (UID: \"84058966-ce0f-42ee-bdf1-729a6c66d43e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544"
Sep 16 16:53:58.285672 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:58.285330    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84058966-ce0f-42ee-bdf1-729a6c66d43e-tmp-dir\") pod \"router-with-refs-test-kserve-7fbd9694d6-tc544\" (UID: \"84058966-ce0f-42ee-bdf1-729a6c66d43e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544"
Sep 16 16:53:58.285672 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:58.285397    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84058966-ce0f-42ee-bdf1-729a6c66d43e-home\") pod \"router-with-refs-test-kserve-7fbd9694d6-tc544\" (UID: \"84058966-ce0f-42ee-bdf1-729a6c66d43e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544"
Sep 16 16:53:58.285672 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:58.285455    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6hmw\" (UniqueName: \"kubernetes.io/projected/84058966-ce0f-42ee-bdf1-729a6c66d43e-kube-api-access-q6hmw\") pod \"router-with-refs-test-kserve-7fbd9694d6-tc544\" (UID: \"84058966-ce0f-42ee-bdf1-729a6c66d43e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544"
Sep 16 16:53:58.285672 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:58.285494    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84058966-ce0f-42ee-bdf1-729a6c66d43e-tls-certs\") pod \"router-with-refs-test-kserve-7fbd9694d6-tc544\" (UID: \"84058966-ce0f-42ee-bdf1-729a6c66d43e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544"
Sep 16 16:53:58.285672 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:58.285536    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84058966-ce0f-42ee-bdf1-729a6c66d43e-kserve-provision-location\") pod \"router-with-refs-test-kserve-7fbd9694d6-tc544\" (UID: \"84058966-ce0f-42ee-bdf1-729a6c66d43e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544"
Sep 16 16:53:58.285672 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:58.285568    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84058966-ce0f-42ee-bdf1-729a6c66d43e-model-cache\") pod \"router-with-refs-test-kserve-7fbd9694d6-tc544\" (UID: \"84058966-ce0f-42ee-bdf1-729a6c66d43e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544"
Sep 16 16:53:58.285672 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:58.285592    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84058966-ce0f-42ee-bdf1-729a6c66d43e-cabundle-cert\") pod \"router-with-refs-test-kserve-7fbd9694d6-tc544\" (UID: \"84058966-ce0f-42ee-bdf1-729a6c66d43e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544"
Sep 16 16:53:58.295691 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:58.295671    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:53:58.386944 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:58.386915    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84058966-ce0f-42ee-bdf1-729a6c66d43e-kserve-provision-location\") pod \"router-with-refs-test-kserve-7fbd9694d6-tc544\" (UID: \"84058966-ce0f-42ee-bdf1-729a6c66d43e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544"
Sep 16 16:53:58.387155 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:58.386950    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84058966-ce0f-42ee-bdf1-729a6c66d43e-model-cache\") pod \"router-with-refs-test-kserve-7fbd9694d6-tc544\" (UID: \"84058966-ce0f-42ee-bdf1-729a6c66d43e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544"
Sep 16 16:53:58.387155 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:58.386967    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84058966-ce0f-42ee-bdf1-729a6c66d43e-cabundle-cert\") pod \"router-with-refs-test-kserve-7fbd9694d6-tc544\" (UID: \"84058966-ce0f-42ee-bdf1-729a6c66d43e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544"
Sep 16 16:53:58.387155 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:58.387101    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84058966-ce0f-42ee-bdf1-729a6c66d43e-dshm\") pod \"router-with-refs-test-kserve-7fbd9694d6-tc544\" (UID: \"84058966-ce0f-42ee-bdf1-729a6c66d43e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544"
Sep 16 16:53:58.387155 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:58.387137    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84058966-ce0f-42ee-bdf1-729a6c66d43e-tmp-dir\") pod \"router-with-refs-test-kserve-7fbd9694d6-tc544\" (UID: \"84058966-ce0f-42ee-bdf1-729a6c66d43e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544"
Sep 16 16:53:58.387385 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:58.387166    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84058966-ce0f-42ee-bdf1-729a6c66d43e-home\") pod \"router-with-refs-test-kserve-7fbd9694d6-tc544\" (UID: \"84058966-ce0f-42ee-bdf1-729a6c66d43e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544"
Sep 16 16:53:58.387385 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:58.387206    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6hmw\" (UniqueName: \"kubernetes.io/projected/84058966-ce0f-42ee-bdf1-729a6c66d43e-kube-api-access-q6hmw\") pod \"router-with-refs-test-kserve-7fbd9694d6-tc544\" (UID: \"84058966-ce0f-42ee-bdf1-729a6c66d43e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544"
Sep 16 16:53:58.387385 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:58.387243    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84058966-ce0f-42ee-bdf1-729a6c66d43e-tls-certs\") pod \"router-with-refs-test-kserve-7fbd9694d6-tc544\" (UID: \"84058966-ce0f-42ee-bdf1-729a6c66d43e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544"
Sep 16 16:53:58.387385 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:58.387357    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84058966-ce0f-42ee-bdf1-729a6c66d43e-model-cache\") pod \"router-with-refs-test-kserve-7fbd9694d6-tc544\" (UID: \"84058966-ce0f-42ee-bdf1-729a6c66d43e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544"
Sep 16 16:53:58.387597 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:58.387467    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84058966-ce0f-42ee-bdf1-729a6c66d43e-kserve-provision-location\") pod \"router-with-refs-test-kserve-7fbd9694d6-tc544\" (UID: \"84058966-ce0f-42ee-bdf1-729a6c66d43e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544"
Sep 16 16:53:58.387597 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:58.387569    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84058966-ce0f-42ee-bdf1-729a6c66d43e-tmp-dir\") pod \"router-with-refs-test-kserve-7fbd9694d6-tc544\" (UID: \"84058966-ce0f-42ee-bdf1-729a6c66d43e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544"
Sep 16 16:53:58.387691 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:58.387634    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84058966-ce0f-42ee-bdf1-729a6c66d43e-home\") pod \"router-with-refs-test-kserve-7fbd9694d6-tc544\" (UID: \"84058966-ce0f-42ee-bdf1-729a6c66d43e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544"
Sep 16 16:53:58.387751 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:58.387734    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84058966-ce0f-42ee-bdf1-729a6c66d43e-cabundle-cert\") pod \"router-with-refs-test-kserve-7fbd9694d6-tc544\" (UID: \"84058966-ce0f-42ee-bdf1-729a6c66d43e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544"
Sep 16 16:53:58.389503 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:58.389483    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84058966-ce0f-42ee-bdf1-729a6c66d43e-dshm\") pod \"router-with-refs-test-kserve-7fbd9694d6-tc544\" (UID: \"84058966-ce0f-42ee-bdf1-729a6c66d43e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544"
Sep 16 16:53:58.389675 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:58.389658    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84058966-ce0f-42ee-bdf1-729a6c66d43e-tls-certs\") pod \"router-with-refs-test-kserve-7fbd9694d6-tc544\" (UID: \"84058966-ce0f-42ee-bdf1-729a6c66d43e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544"
Sep 16 16:53:58.395213 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:58.395185    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6hmw\" (UniqueName: \"kubernetes.io/projected/84058966-ce0f-42ee-bdf1-729a6c66d43e-kube-api-access-q6hmw\") pod \"router-with-refs-test-kserve-7fbd9694d6-tc544\" (UID: \"84058966-ce0f-42ee-bdf1-729a6c66d43e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544"
Sep 16 16:53:58.570774 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:58.570735    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544"
Sep 16 16:53:58.582826 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:58.582781    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:53:58.683329 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:58.683275    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:53:58.846767 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:58.846743    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544"]
Sep 16 16:53:58.849100 ip-10-0-136-228 kubenswrapper[2554]: W0916 16:53:58.849075    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84058966_ce0f_42ee_bdf1_729a6c66d43e.slice/crio-856813908e7e0c7744ad6966f58584502ae81b9160635dd937c559f37535dc2c WatchSource:0}: Error finding container 856813908e7e0c7744ad6966f58584502ae81b9160635dd937c559f37535dc2c: Status 404 returned error can't find the container with id 856813908e7e0c7744ad6966f58584502ae81b9160635dd937c559f37535dc2c
Sep 16 16:53:59.279504 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:59.279424    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544" event={"ID":"84058966-ce0f-42ee-bdf1-729a6c66d43e","Type":"ContainerStarted","Data":"6e5c1660c04a01f2812abe1590caa21ff08501ba240a10ee1c4e865975125b58"}
Sep 16 16:53:59.279504 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:53:59.279486    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544" event={"ID":"84058966-ce0f-42ee-bdf1-729a6c66d43e","Type":"ContainerStarted","Data":"856813908e7e0c7744ad6966f58584502ae81b9160635dd937c559f37535dc2c"}
Sep 16 16:54:01.287635 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:54:01.287602    2554 generic.go:356] "Generic (PLEG): container finished" podID="84058966-ce0f-42ee-bdf1-729a6c66d43e" containerID="6e5c1660c04a01f2812abe1590caa21ff08501ba240a10ee1c4e865975125b58" exitCode=0
Sep 16 16:54:01.288141 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:54:01.287645    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544" event={"ID":"84058966-ce0f-42ee-bdf1-729a6c66d43e","Type":"ContainerDied","Data":"6e5c1660c04a01f2812abe1590caa21ff08501ba240a10ee1c4e865975125b58"}
Sep 16 16:54:02.293751 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:54:02.293712    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544" event={"ID":"84058966-ce0f-42ee-bdf1-729a6c66d43e","Type":"ContainerStarted","Data":"30dd40c3671f679134d60193bcee46a4590070cc87f0bc7f28b54025d2b7d21b"}
Sep 16 16:54:02.316233 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:54:02.316168    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544" podStartSLOduration=2.187767049 podStartE2EDuration="4.316151541s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.128384492s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:54:02.314543693 +0000 UTC m=+3842.759235887" watchObservedRunningTime="2026-09-16 16:54:02.316151541 +0000 UTC m=+3842.760843719"
Sep 16 16:54:02.571230 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:54:02.571147    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544"
Sep 16 16:54:08.503551 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:54:08.503514    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:54:08.571949 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:54:08.571908    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544"
Sep 16 16:54:08.573488 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:54:08.573461    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544" podUID="84058966-ce0f-42ee-bdf1-729a6c66d43e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Sep 16 16:54:18.572054 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:54:18.572009    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544" podUID="84058966-ce0f-42ee-bdf1-729a6c66d43e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Sep 16 16:54:28.571931 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:54:28.571825    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544" podUID="84058966-ce0f-42ee-bdf1-729a6c66d43e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Sep 16 16:54:38.571471 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:54:38.571416    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544" podUID="84058966-ce0f-42ee-bdf1-729a6c66d43e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Sep 16 16:54:40.638660 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:54:40.638623    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:54:48.571977 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:54:48.571926    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544" podUID="84058966-ce0f-42ee-bdf1-729a6c66d43e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Sep 16 16:54:58.571654 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:54:58.571600    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544" podUID="84058966-ce0f-42ee-bdf1-729a6c66d43e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Sep 16 16:55:00.744132 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:55:00.744086    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:55:02.528013 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:55:02.527979    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:55:04.343461 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:55:04.343427    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:55:07.399658 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:55:07.399619    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:55:07.477757 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:55:07.477720    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:55:08.571530 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:55:08.571491    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544" podUID="84058966-ce0f-42ee-bdf1-729a6c66d43e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Sep 16 16:55:18.571490 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:55:18.571450    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544" podUID="84058966-ce0f-42ee-bdf1-729a6c66d43e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Sep 16 16:55:28.571552 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:55:28.571505    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544" podUID="84058966-ce0f-42ee-bdf1-729a6c66d43e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Sep 16 16:55:38.572115 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:55:38.572057    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544" podUID="84058966-ce0f-42ee-bdf1-729a6c66d43e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Sep 16 16:55:47.275840 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:55:47.275773    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:55:48.572066 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:55:48.572031    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544" podUID="84058966-ce0f-42ee-bdf1-729a6c66d43e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Sep 16 16:55:58.581548 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:55:58.581514    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544"
Sep 16 16:55:58.589434 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:55:58.589404    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544"
Sep 16 16:56:03.599273 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:03.599238    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:56:03.612903 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:03.612867    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544"]
Sep 16 16:56:03.662621 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:03.662586    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:56:18.634574 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:18.634535    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544" podUID="84058966-ce0f-42ee-bdf1-729a6c66d43e" containerName="main" containerID="cri-o://30dd40c3671f679134d60193bcee46a4590070cc87f0bc7f28b54025d2b7d21b" gracePeriod=45
Sep 16 16:56:19.294492 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:19.294469    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544"
Sep 16 16:56:19.471206 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:19.471123    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84058966-ce0f-42ee-bdf1-729a6c66d43e-kserve-provision-location\") pod \"84058966-ce0f-42ee-bdf1-729a6c66d43e\" (UID: \"84058966-ce0f-42ee-bdf1-729a6c66d43e\") "
Sep 16 16:56:19.471206 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:19.471175    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84058966-ce0f-42ee-bdf1-729a6c66d43e-tmp-dir\") pod \"84058966-ce0f-42ee-bdf1-729a6c66d43e\" (UID: \"84058966-ce0f-42ee-bdf1-729a6c66d43e\") "
Sep 16 16:56:19.471206 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:19.471197    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84058966-ce0f-42ee-bdf1-729a6c66d43e-home\") pod \"84058966-ce0f-42ee-bdf1-729a6c66d43e\" (UID: \"84058966-ce0f-42ee-bdf1-729a6c66d43e\") "
Sep 16 16:56:19.471463 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:19.471240    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84058966-ce0f-42ee-bdf1-729a6c66d43e-dshm\") pod \"84058966-ce0f-42ee-bdf1-729a6c66d43e\" (UID: \"84058966-ce0f-42ee-bdf1-729a6c66d43e\") "
Sep 16 16:56:19.471463 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:19.471278    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6hmw\" (UniqueName: \"kubernetes.io/projected/84058966-ce0f-42ee-bdf1-729a6c66d43e-kube-api-access-q6hmw\") pod \"84058966-ce0f-42ee-bdf1-729a6c66d43e\" (UID: \"84058966-ce0f-42ee-bdf1-729a6c66d43e\") "
Sep 16 16:56:19.471463 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:19.471321    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84058966-ce0f-42ee-bdf1-729a6c66d43e-tls-certs\") pod \"84058966-ce0f-42ee-bdf1-729a6c66d43e\" (UID: \"84058966-ce0f-42ee-bdf1-729a6c66d43e\") "
Sep 16 16:56:19.471463 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:19.471350    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84058966-ce0f-42ee-bdf1-729a6c66d43e-cabundle-cert\") pod \"84058966-ce0f-42ee-bdf1-729a6c66d43e\" (UID: \"84058966-ce0f-42ee-bdf1-729a6c66d43e\") "
Sep 16 16:56:19.471463 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:19.471377    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84058966-ce0f-42ee-bdf1-729a6c66d43e-model-cache\") pod \"84058966-ce0f-42ee-bdf1-729a6c66d43e\" (UID: \"84058966-ce0f-42ee-bdf1-729a6c66d43e\") "
Sep 16 16:56:19.471840 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:19.471768    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84058966-ce0f-42ee-bdf1-729a6c66d43e-model-cache" (OuterVolumeSpecName: "model-cache") pod "84058966-ce0f-42ee-bdf1-729a6c66d43e" (UID: "84058966-ce0f-42ee-bdf1-729a6c66d43e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:56:19.471937 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:19.471839    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84058966-ce0f-42ee-bdf1-729a6c66d43e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "84058966-ce0f-42ee-bdf1-729a6c66d43e" (UID: "84058966-ce0f-42ee-bdf1-729a6c66d43e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 16:56:19.472009 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:19.471978    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84058966-ce0f-42ee-bdf1-729a6c66d43e-home" (OuterVolumeSpecName: "home") pod "84058966-ce0f-42ee-bdf1-729a6c66d43e" (UID: "84058966-ce0f-42ee-bdf1-729a6c66d43e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:56:19.473579 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:19.473546    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84058966-ce0f-42ee-bdf1-729a6c66d43e-dshm" (OuterVolumeSpecName: "dshm") pod "84058966-ce0f-42ee-bdf1-729a6c66d43e" (UID: "84058966-ce0f-42ee-bdf1-729a6c66d43e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:56:19.473678 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:19.473642    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84058966-ce0f-42ee-bdf1-729a6c66d43e-kube-api-access-q6hmw" (OuterVolumeSpecName: "kube-api-access-q6hmw") pod "84058966-ce0f-42ee-bdf1-729a6c66d43e" (UID: "84058966-ce0f-42ee-bdf1-729a6c66d43e"). InnerVolumeSpecName "kube-api-access-q6hmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:56:19.474127 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:19.474105    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84058966-ce0f-42ee-bdf1-729a6c66d43e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "84058966-ce0f-42ee-bdf1-729a6c66d43e" (UID: "84058966-ce0f-42ee-bdf1-729a6c66d43e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:56:19.483842 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:19.483802    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84058966-ce0f-42ee-bdf1-729a6c66d43e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "84058966-ce0f-42ee-bdf1-729a6c66d43e" (UID: "84058966-ce0f-42ee-bdf1-729a6c66d43e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:56:19.498292 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:19.498270    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84058966-ce0f-42ee-bdf1-729a6c66d43e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "84058966-ce0f-42ee-bdf1-729a6c66d43e" (UID: "84058966-ce0f-42ee-bdf1-729a6c66d43e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:56:19.572090 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:19.572064    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84058966-ce0f-42ee-bdf1-729a6c66d43e-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:56:19.572090 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:19.572088    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84058966-ce0f-42ee-bdf1-729a6c66d43e-tmp-dir\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:56:19.572239 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:19.572098    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84058966-ce0f-42ee-bdf1-729a6c66d43e-home\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:56:19.572239 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:19.572106    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84058966-ce0f-42ee-bdf1-729a6c66d43e-dshm\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:56:19.572239 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:19.572113    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6hmw\" (UniqueName: \"kubernetes.io/projected/84058966-ce0f-42ee-bdf1-729a6c66d43e-kube-api-access-q6hmw\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:56:19.572239 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:19.572121    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84058966-ce0f-42ee-bdf1-729a6c66d43e-tls-certs\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:56:19.572239 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:19.572130    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84058966-ce0f-42ee-bdf1-729a6c66d43e-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:56:19.572239 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:19.572138    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84058966-ce0f-42ee-bdf1-729a6c66d43e-model-cache\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 16:56:19.837011 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:19.836981    2554 generic.go:356] "Generic (PLEG): container finished" podID="84058966-ce0f-42ee-bdf1-729a6c66d43e" containerID="30dd40c3671f679134d60193bcee46a4590070cc87f0bc7f28b54025d2b7d21b" exitCode=0
Sep 16 16:56:19.837387 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:19.837031    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544"
Sep 16 16:56:19.837387 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:19.837060    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544" event={"ID":"84058966-ce0f-42ee-bdf1-729a6c66d43e","Type":"ContainerDied","Data":"30dd40c3671f679134d60193bcee46a4590070cc87f0bc7f28b54025d2b7d21b"}
Sep 16 16:56:19.837387 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:19.837112    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544" event={"ID":"84058966-ce0f-42ee-bdf1-729a6c66d43e","Type":"ContainerDied","Data":"856813908e7e0c7744ad6966f58584502ae81b9160635dd937c559f37535dc2c"}
Sep 16 16:56:19.837387 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:19.837147    2554 scope.go:117] "RemoveContainer" containerID="30dd40c3671f679134d60193bcee46a4590070cc87f0bc7f28b54025d2b7d21b"
Sep 16 16:56:19.845656 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:19.845634    2554 scope.go:117] "RemoveContainer" containerID="6e5c1660c04a01f2812abe1590caa21ff08501ba240a10ee1c4e865975125b58"
Sep 16 16:56:19.852545 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:19.852529    2554 scope.go:117] "RemoveContainer" containerID="30dd40c3671f679134d60193bcee46a4590070cc87f0bc7f28b54025d2b7d21b"
Sep 16 16:56:19.852831 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:56:19.852805    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30dd40c3671f679134d60193bcee46a4590070cc87f0bc7f28b54025d2b7d21b\": container with ID starting with 30dd40c3671f679134d60193bcee46a4590070cc87f0bc7f28b54025d2b7d21b not found: ID does not exist" containerID="30dd40c3671f679134d60193bcee46a4590070cc87f0bc7f28b54025d2b7d21b"
Sep 16 16:56:19.852924 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:19.852838    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30dd40c3671f679134d60193bcee46a4590070cc87f0bc7f28b54025d2b7d21b"} err="failed to get container status \"30dd40c3671f679134d60193bcee46a4590070cc87f0bc7f28b54025d2b7d21b\": rpc error: code = NotFound desc = could not find container \"30dd40c3671f679134d60193bcee46a4590070cc87f0bc7f28b54025d2b7d21b\": container with ID starting with 30dd40c3671f679134d60193bcee46a4590070cc87f0bc7f28b54025d2b7d21b not found: ID does not exist"
Sep 16 16:56:19.852924 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:19.852855    2554 scope.go:117] "RemoveContainer" containerID="6e5c1660c04a01f2812abe1590caa21ff08501ba240a10ee1c4e865975125b58"
Sep 16 16:56:19.853114 ip-10-0-136-228 kubenswrapper[2554]: E0916 16:56:19.853095    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e5c1660c04a01f2812abe1590caa21ff08501ba240a10ee1c4e865975125b58\": container with ID starting with 6e5c1660c04a01f2812abe1590caa21ff08501ba240a10ee1c4e865975125b58 not found: ID does not exist" containerID="6e5c1660c04a01f2812abe1590caa21ff08501ba240a10ee1c4e865975125b58"
Sep 16 16:56:19.853165 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:19.853124    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e5c1660c04a01f2812abe1590caa21ff08501ba240a10ee1c4e865975125b58"} err="failed to get container status \"6e5c1660c04a01f2812abe1590caa21ff08501ba240a10ee1c4e865975125b58\": rpc error: code = NotFound desc = could not find container \"6e5c1660c04a01f2812abe1590caa21ff08501ba240a10ee1c4e865975125b58\": container with ID starting with 6e5c1660c04a01f2812abe1590caa21ff08501ba240a10ee1c4e865975125b58 not found: ID does not exist"
Sep 16 16:56:19.859115 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:19.859094    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544"]
Sep 16 16:56:19.863074 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:19.863052    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7fbd9694d6-tc544"]
Sep 16 16:56:20.076580 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:20.076553    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84058966-ce0f-42ee-bdf1-729a6c66d43e" path="/var/lib/kubelet/pods/84058966-ce0f-42ee-bdf1-729a6c66d43e/volumes"
Sep 16 16:56:51.901271 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:51.901238    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:56:53.680539 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:53.680504    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:56:53.777558 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:53.777518    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:56:58.731817 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:56:58.731760    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:57:04.313262 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.313221    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm"]
Sep 16 16:57:04.313706 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.313463    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84058966-ce0f-42ee-bdf1-729a6c66d43e" containerName="main"
Sep 16 16:57:04.313706 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.313476    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="84058966-ce0f-42ee-bdf1-729a6c66d43e" containerName="main"
Sep 16 16:57:04.313706 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.313489    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84058966-ce0f-42ee-bdf1-729a6c66d43e" containerName="storage-initializer"
Sep 16 16:57:04.313706 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.313495    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="84058966-ce0f-42ee-bdf1-729a6c66d43e" containerName="storage-initializer"
Sep 16 16:57:04.313706 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.313545    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="84058966-ce0f-42ee-bdf1-729a6c66d43e" containerName="main"
Sep 16 16:57:04.316103 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.316080    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm"
Sep 16 16:57:04.318845 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.318826    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 16 16:57:04.319060 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.319014    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 16 16:57:04.319060 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.319050    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 16:57:04.319175 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.319082    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-lgp9k\""
Sep 16 16:57:04.319175 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.319087    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 16 16:57:04.320266 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.320245    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 16 16:57:04.320266 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.320254    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-vdz4t\""
Sep 16 16:57:04.323392 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.323367    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq"]
Sep 16 16:57:04.325528 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.325508    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq"
Sep 16 16:57:04.334002 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.333977    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm"]
Sep 16 16:57:04.343617 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.343588    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq"]
Sep 16 16:57:04.360696 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.360670    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:57:04.394841 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.394780    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3bb8154-638e-4c93-98e3-41a56d965948-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq\" (UID: \"d3bb8154-638e-4c93-98e3-41a56d965948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq"
Sep 16 16:57:04.394841 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.394840    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3bb8154-638e-4c93-98e3-41a56d965948-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq\" (UID: \"d3bb8154-638e-4c93-98e3-41a56d965948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq"
Sep 16 16:57:04.395077 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.394864    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a22d4d0-042c-4fbb-b739-441ef61aaadf-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm\" (UID: \"4a22d4d0-042c-4fbb-b739-441ef61aaadf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm"
Sep 16 16:57:04.395077 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.394879    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3bb8154-638e-4c93-98e3-41a56d965948-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq\" (UID: \"d3bb8154-638e-4c93-98e3-41a56d965948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq"
Sep 16 16:57:04.395077 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.394895    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6r2l\" (UniqueName: \"kubernetes.io/projected/d3bb8154-638e-4c93-98e3-41a56d965948-kube-api-access-t6r2l\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq\" (UID: \"d3bb8154-638e-4c93-98e3-41a56d965948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq"
Sep 16 16:57:04.395077 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.394916    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a22d4d0-042c-4fbb-b739-441ef61aaadf-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm\" (UID: \"4a22d4d0-042c-4fbb-b739-441ef61aaadf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm"
Sep 16 16:57:04.395077 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.394971    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a22d4d0-042c-4fbb-b739-441ef61aaadf-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm\" (UID: \"4a22d4d0-042c-4fbb-b739-441ef61aaadf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm"
Sep 16 16:57:04.395077 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.394988    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a22d4d0-042c-4fbb-b739-441ef61aaadf-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm\" (UID: \"4a22d4d0-042c-4fbb-b739-441ef61aaadf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm"
Sep 16 16:57:04.395077 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.395060    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a22d4d0-042c-4fbb-b739-441ef61aaadf-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm\" (UID: \"4a22d4d0-042c-4fbb-b739-441ef61aaadf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm"
Sep 16 16:57:04.395392 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.395084    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a22d4d0-042c-4fbb-b739-441ef61aaadf-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm\" (UID: \"4a22d4d0-042c-4fbb-b739-441ef61aaadf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm"
Sep 16 16:57:04.395392 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.395107    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3bb8154-638e-4c93-98e3-41a56d965948-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq\" (UID: \"d3bb8154-638e-4c93-98e3-41a56d965948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq"
Sep 16 16:57:04.395392 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.395157    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3bb8154-638e-4c93-98e3-41a56d965948-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq\" (UID: \"d3bb8154-638e-4c93-98e3-41a56d965948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq"
Sep 16 16:57:04.395392 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.395199    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfd6l\" (UniqueName: \"kubernetes.io/projected/4a22d4d0-042c-4fbb-b739-441ef61aaadf-kube-api-access-tfd6l\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm\" (UID: \"4a22d4d0-042c-4fbb-b739-441ef61aaadf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm"
Sep 16 16:57:04.395392 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.395238    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3bb8154-638e-4c93-98e3-41a56d965948-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq\" (UID: \"d3bb8154-638e-4c93-98e3-41a56d965948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq"
Sep 16 16:57:04.395392 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.395276    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a22d4d0-042c-4fbb-b739-441ef61aaadf-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm\" (UID: \"4a22d4d0-042c-4fbb-b739-441ef61aaadf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm"
Sep 16 16:57:04.395392 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.395295    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3bb8154-638e-4c93-98e3-41a56d965948-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq\" (UID: \"d3bb8154-638e-4c93-98e3-41a56d965948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq"
Sep 16 16:57:04.496079 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.496043    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a22d4d0-042c-4fbb-b739-441ef61aaadf-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm\" (UID: \"4a22d4d0-042c-4fbb-b739-441ef61aaadf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm"
Sep 16 16:57:04.496079 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.496083    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a22d4d0-042c-4fbb-b739-441ef61aaadf-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm\" (UID: \"4a22d4d0-042c-4fbb-b739-441ef61aaadf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm"
Sep 16 16:57:04.496337 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.496101    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a22d4d0-042c-4fbb-b739-441ef61aaadf-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm\" (UID: \"4a22d4d0-042c-4fbb-b739-441ef61aaadf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm"
Sep 16 16:57:04.496337 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.496214    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a22d4d0-042c-4fbb-b739-441ef61aaadf-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm\" (UID: \"4a22d4d0-042c-4fbb-b739-441ef61aaadf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm"
Sep 16 16:57:04.496337 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.496247    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a22d4d0-042c-4fbb-b739-441ef61aaadf-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm\" (UID: \"4a22d4d0-042c-4fbb-b739-441ef61aaadf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm"
Sep 16 16:57:04.496337 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.496268    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3bb8154-638e-4c93-98e3-41a56d965948-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq\" (UID: \"d3bb8154-638e-4c93-98e3-41a56d965948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq"
Sep 16 16:57:04.496337 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.496289    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3bb8154-638e-4c93-98e3-41a56d965948-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq\" (UID: \"d3bb8154-638e-4c93-98e3-41a56d965948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq"
Sep 16 16:57:04.496337 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.496324    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfd6l\" (UniqueName: \"kubernetes.io/projected/4a22d4d0-042c-4fbb-b739-441ef61aaadf-kube-api-access-tfd6l\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm\" (UID: \"4a22d4d0-042c-4fbb-b739-441ef61aaadf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm"
Sep 16 16:57:04.496657 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.496360    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3bb8154-638e-4c93-98e3-41a56d965948-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq\" (UID: \"d3bb8154-638e-4c93-98e3-41a56d965948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq"
Sep 16 16:57:04.496657 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.496404    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a22d4d0-042c-4fbb-b739-441ef61aaadf-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm\" (UID: \"4a22d4d0-042c-4fbb-b739-441ef61aaadf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm"
Sep 16 16:57:04.496657 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.496439    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3bb8154-638e-4c93-98e3-41a56d965948-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq\" (UID: \"d3bb8154-638e-4c93-98e3-41a56d965948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq"
Sep 16 16:57:04.496657 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.496480    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3bb8154-638e-4c93-98e3-41a56d965948-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq\" (UID: \"d3bb8154-638e-4c93-98e3-41a56d965948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq"
Sep 16 16:57:04.496657 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.496507    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3bb8154-638e-4c93-98e3-41a56d965948-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq\" (UID: \"d3bb8154-638e-4c93-98e3-41a56d965948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq"
Sep 16 16:57:04.496657 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.496540    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a22d4d0-042c-4fbb-b739-441ef61aaadf-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm\" (UID: \"4a22d4d0-042c-4fbb-b739-441ef61aaadf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm"
Sep 16 16:57:04.496657 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.496545    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a22d4d0-042c-4fbb-b739-441ef61aaadf-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm\" (UID: \"4a22d4d0-042c-4fbb-b739-441ef61aaadf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm"
Sep 16 16:57:04.496657 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.496564    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3bb8154-638e-4c93-98e3-41a56d965948-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq\" (UID: \"d3bb8154-638e-4c93-98e3-41a56d965948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq"
Sep 16 16:57:04.496657 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.496589    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6r2l\" (UniqueName: \"kubernetes.io/projected/d3bb8154-638e-4c93-98e3-41a56d965948-kube-api-access-t6r2l\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq\" (UID: \"d3bb8154-638e-4c93-98e3-41a56d965948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq"
Sep 16 16:57:04.496657 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.496623    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a22d4d0-042c-4fbb-b739-441ef61aaadf-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm\" (UID: \"4a22d4d0-042c-4fbb-b739-441ef61aaadf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm"
Sep 16 16:57:04.496657 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.496643    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3bb8154-638e-4c93-98e3-41a56d965948-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq\" (UID: \"d3bb8154-638e-4c93-98e3-41a56d965948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq"
Sep 16 16:57:04.497287 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.496918    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a22d4d0-042c-4fbb-b739-441ef61aaadf-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm\" (UID: \"4a22d4d0-042c-4fbb-b739-441ef61aaadf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm"
Sep 16 16:57:04.497287 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.497000    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a22d4d0-042c-4fbb-b739-441ef61aaadf-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm\" (UID: \"4a22d4d0-042c-4fbb-b739-441ef61aaadf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm"
Sep 16 16:57:04.497287 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.497180    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3bb8154-638e-4c93-98e3-41a56d965948-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq\" (UID: \"d3bb8154-638e-4c93-98e3-41a56d965948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq"
Sep 16 16:57:04.497287 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.497231    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3bb8154-638e-4c93-98e3-41a56d965948-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq\" (UID: \"d3bb8154-638e-4c93-98e3-41a56d965948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq"
Sep 16 16:57:04.497287 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.497273    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a22d4d0-042c-4fbb-b739-441ef61aaadf-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm\" (UID: \"4a22d4d0-042c-4fbb-b739-441ef61aaadf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm"
Sep 16 16:57:04.497537 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.497518    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3bb8154-638e-4c93-98e3-41a56d965948-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq\" (UID: \"d3bb8154-638e-4c93-98e3-41a56d965948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq"
Sep 16 16:57:04.497595 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.497516    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3bb8154-638e-4c93-98e3-41a56d965948-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq\" (UID: \"d3bb8154-638e-4c93-98e3-41a56d965948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq"
Sep 16 16:57:04.498894 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.498870    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3bb8154-638e-4c93-98e3-41a56d965948-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq\" (UID: \"d3bb8154-638e-4c93-98e3-41a56d965948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq"
Sep 16 16:57:04.499341 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.499322    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a22d4d0-042c-4fbb-b739-441ef61aaadf-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm\" (UID: \"4a22d4d0-042c-4fbb-b739-441ef61aaadf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm"
Sep 16 16:57:04.499438 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.499407    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a22d4d0-042c-4fbb-b739-441ef61aaadf-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm\" (UID: \"4a22d4d0-042c-4fbb-b739-441ef61aaadf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm"
Sep 16 16:57:04.499498 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.499444    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3bb8154-638e-4c93-98e3-41a56d965948-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq\" (UID: \"d3bb8154-638e-4c93-98e3-41a56d965948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq"
Sep 16 16:57:04.504602 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.504580    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6r2l\" (UniqueName: \"kubernetes.io/projected/d3bb8154-638e-4c93-98e3-41a56d965948-kube-api-access-t6r2l\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq\" (UID: \"d3bb8154-638e-4c93-98e3-41a56d965948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq"
Sep 16 16:57:04.504855 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.504838    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfd6l\" (UniqueName: \"kubernetes.io/projected/4a22d4d0-042c-4fbb-b739-441ef61aaadf-kube-api-access-tfd6l\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm\" (UID: \"4a22d4d0-042c-4fbb-b739-441ef61aaadf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm"
Sep 16 16:57:04.626182 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.626088    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm"
Sep 16 16:57:04.635704 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.635520    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq"
Sep 16 16:57:04.636641 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.636613    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:57:04.907465 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.907376    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm"]
Sep 16 16:57:04.911580 ip-10-0-136-228 kubenswrapper[2554]: W0916 16:57:04.911555    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a22d4d0_042c_4fbb_b739_441ef61aaadf.slice/crio-1716e5eaa402041046be7d190471bbbea79d08d4d40d2e063c9bc2fb1c8bbc90 WatchSource:0}: Error finding container 1716e5eaa402041046be7d190471bbbea79d08d4d40d2e063c9bc2fb1c8bbc90: Status 404 returned error can't find the container with id 1716e5eaa402041046be7d190471bbbea79d08d4d40d2e063c9bc2fb1c8bbc90
Sep 16 16:57:04.923634 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:04.923605    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq"]
Sep 16 16:57:04.926393 ip-10-0-136-228 kubenswrapper[2554]: W0916 16:57:04.926370    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3bb8154_638e_4c93_98e3_41a56d965948.slice/crio-b075ed60920ab2b4b49f8c18cf02ea3552655eb776dd23948f636e83240ae716 WatchSource:0}: Error finding container b075ed60920ab2b4b49f8c18cf02ea3552655eb776dd23948f636e83240ae716: Status 404 returned error can't find the container with id b075ed60920ab2b4b49f8c18cf02ea3552655eb776dd23948f636e83240ae716
Sep 16 16:57:05.013173 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:05.013133    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq" event={"ID":"d3bb8154-638e-4c93-98e3-41a56d965948","Type":"ContainerStarted","Data":"beb0f9c0c3ecec106255cc9a7248ed1dcd71af199cf06f45dd1185a1985698c3"}
Sep 16 16:57:05.013373 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:05.013235    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq" event={"ID":"d3bb8154-638e-4c93-98e3-41a56d965948","Type":"ContainerStarted","Data":"b075ed60920ab2b4b49f8c18cf02ea3552655eb776dd23948f636e83240ae716"}
Sep 16 16:57:05.014974 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:05.014925    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm" event={"ID":"4a22d4d0-042c-4fbb-b739-441ef61aaadf","Type":"ContainerStarted","Data":"ffe2b4bb95d733b4213b00b03ecf7be7cba09bd01471c0c8782e8f7e7f279fe5"}
Sep 16 16:57:05.015251 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:05.014979    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm" event={"ID":"4a22d4d0-042c-4fbb-b739-441ef61aaadf","Type":"ContainerStarted","Data":"1716e5eaa402041046be7d190471bbbea79d08d4d40d2e063c9bc2fb1c8bbc90"}
Sep 16 16:57:05.015251 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:05.015018    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm"
Sep 16 16:57:06.020366 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:06.020319    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm" event={"ID":"4a22d4d0-042c-4fbb-b739-441ef61aaadf","Type":"ContainerStarted","Data":"67d71d40d5c1a4d679cbfe9718a2f23e479516fecc99a731a4ad1834fb2ac266"}
Sep 16 16:57:08.029480 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:08.029445    2554 generic.go:356] "Generic (PLEG): container finished" podID="d3bb8154-638e-4c93-98e3-41a56d965948" containerID="beb0f9c0c3ecec106255cc9a7248ed1dcd71af199cf06f45dd1185a1985698c3" exitCode=0
Sep 16 16:57:08.029946 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:08.029520    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq" event={"ID":"d3bb8154-638e-4c93-98e3-41a56d965948","Type":"ContainerDied","Data":"beb0f9c0c3ecec106255cc9a7248ed1dcd71af199cf06f45dd1185a1985698c3"}
Sep 16 16:57:08.031408 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:08.031387    2554 generic.go:356] "Generic (PLEG): container finished" podID="4a22d4d0-042c-4fbb-b739-441ef61aaadf" containerID="67d71d40d5c1a4d679cbfe9718a2f23e479516fecc99a731a4ad1834fb2ac266" exitCode=0
Sep 16 16:57:08.031514 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:08.031457    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm" event={"ID":"4a22d4d0-042c-4fbb-b739-441ef61aaadf","Type":"ContainerDied","Data":"67d71d40d5c1a4d679cbfe9718a2f23e479516fecc99a731a4ad1834fb2ac266"}
Sep 16 16:57:09.038454 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:09.038411    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm" event={"ID":"4a22d4d0-042c-4fbb-b739-441ef61aaadf","Type":"ContainerStarted","Data":"54bc974500241591b5f63f939006add20caeb01b58b96f20b51d66afadb9df65"}
Sep 16 16:57:09.040558 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:09.040534    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq" event={"ID":"d3bb8154-638e-4c93-98e3-41a56d965948","Type":"ContainerStarted","Data":"9919df28f4b4cf5be933b2305f6513a1112e58c1bbe4a8c0feada2aabeba7871"}
Sep 16 16:57:09.073687 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:09.073622    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm" podStartSLOduration=2.551040064 podStartE2EDuration="5.073601284s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.52256122s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:57:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:57:09.072433962 +0000 UTC m=+4029.517126142" watchObservedRunningTime="2026-09-16 16:57:09.073601284 +0000 UTC m=+4029.518293463"
Sep 16 16:57:09.096560 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:09.096483    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq" podStartSLOduration=2.5556368320000002 podStartE2EDuration="5.096462505s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.540825673s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:57:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:57:09.095653113 +0000 UTC m=+4029.540345312" watchObservedRunningTime="2026-09-16 16:57:09.096462505 +0000 UTC m=+4029.541154684"
Sep 16 16:57:12.277290 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:12.277255    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:57:12.377748 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:12.377711    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:57:12.479317 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:12.479274    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:57:12.579271 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:12.579237    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:57:12.691396 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:12.691361    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:57:12.791860 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:12.791829    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:57:14.627277 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:14.627243    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm"
Sep 16 16:57:14.627748 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:14.627329    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm"
Sep 16 16:57:14.628553 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:14.628515    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm" podUID="4a22d4d0-042c-4fbb-b739-441ef61aaadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8001/health\": dial tcp 10.132.0.42:8001: connect: connection refused"
Sep 16 16:57:14.635757 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:14.635732    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq"
Sep 16 16:57:14.635904 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:14.635816    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq"
Sep 16 16:57:14.636894 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:14.636865    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq" podUID="d3bb8154-638e-4c93-98e3-41a56d965948" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 16 16:57:14.647009 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:14.646987    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm"
Sep 16 16:57:15.101391 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:15.101348    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:57:15.201918 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:15.201879    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:57:15.736327 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:15.736292    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:57:17.502330 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:17.502296    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:57:19.665184 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:19.665156    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:57:19.764929 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:19.764896    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:57:20.214276 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:20.214239    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:57:21.482818 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:21.482768    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:57:24.626697 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:24.626463    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm" podUID="4a22d4d0-042c-4fbb-b739-441ef61aaadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8001/health\": dial tcp 10.132.0.42:8001: connect: connection refused"
Sep 16 16:57:24.636502 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:24.636459    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq" podUID="d3bb8154-638e-4c93-98e3-41a56d965948" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 16 16:57:27.778788 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:27.778744    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:57:34.626622 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:34.626564    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm" podUID="4a22d4d0-042c-4fbb-b739-441ef61aaadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8001/health\": dial tcp 10.132.0.42:8001: connect: connection refused"
Sep 16 16:57:34.636033 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:34.635989    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq" podUID="d3bb8154-638e-4c93-98e3-41a56d965948" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 16 16:57:41.462858 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:41.462776    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:57:42.038857 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:42.038815    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:57:42.104365 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:42.104332    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:57:44.626411 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:44.626372    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm" podUID="4a22d4d0-042c-4fbb-b739-441ef61aaadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8001/health\": dial tcp 10.132.0.42:8001: connect: connection refused"
Sep 16 16:57:44.636623 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:44.636418    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq" podUID="d3bb8154-638e-4c93-98e3-41a56d965948" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 16 16:57:49.623335 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:49.623256    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:57:54.626704 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:54.626648    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm" podUID="4a22d4d0-042c-4fbb-b739-441ef61aaadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8001/health\": dial tcp 10.132.0.42:8001: connect: connection refused"
Sep 16 16:57:54.636165 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:57:54.636125    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq" podUID="d3bb8154-638e-4c93-98e3-41a56d965948" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 16 16:58:04.627189 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:58:04.627142    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm" podUID="4a22d4d0-042c-4fbb-b739-441ef61aaadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8001/health\": dial tcp 10.132.0.42:8001: connect: connection refused"
Sep 16 16:58:04.635951 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:58:04.635911    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq" podUID="d3bb8154-638e-4c93-98e3-41a56d965948" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 16 16:58:14.627094 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:58:14.627048    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm" podUID="4a22d4d0-042c-4fbb-b739-441ef61aaadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8001/health\": dial tcp 10.132.0.42:8001: connect: connection refused"
Sep 16 16:58:14.636088 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:58:14.636040    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq" podUID="d3bb8154-638e-4c93-98e3-41a56d965948" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 16 16:58:24.626965 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:58:24.626908    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm" podUID="4a22d4d0-042c-4fbb-b739-441ef61aaadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8001/health\": dial tcp 10.132.0.42:8001: connect: connection refused"
Sep 16 16:58:24.636514 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:58:24.636484    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq" podUID="d3bb8154-638e-4c93-98e3-41a56d965948" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 16 16:58:34.626457 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:58:34.626409    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm" podUID="4a22d4d0-042c-4fbb-b739-441ef61aaadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8001/health\": dial tcp 10.132.0.42:8001: connect: connection refused"
Sep 16 16:58:34.636193 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:58:34.636161    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq" podUID="d3bb8154-638e-4c93-98e3-41a56d965948" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 16 16:58:44.627237 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:58:44.627186    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm" podUID="4a22d4d0-042c-4fbb-b739-441ef61aaadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8001/health\": dial tcp 10.132.0.42:8001: connect: connection refused"
Sep 16 16:58:44.636358 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:58:44.636321    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq" podUID="d3bb8154-638e-4c93-98e3-41a56d965948" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 16 16:58:54.627064 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:58:54.626971    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm" podUID="4a22d4d0-042c-4fbb-b739-441ef61aaadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8001/health\": dial tcp 10.132.0.42:8001: connect: connection refused"
Sep 16 16:58:54.636050 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:58:54.636016    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq" podUID="d3bb8154-638e-4c93-98e3-41a56d965948" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 16 16:59:04.627395 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:59:04.627352    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm" podUID="4a22d4d0-042c-4fbb-b739-441ef61aaadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8001/health\": dial tcp 10.132.0.42:8001: connect: connection refused"
Sep 16 16:59:04.636671 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:59:04.636640    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq" podUID="d3bb8154-638e-4c93-98e3-41a56d965948" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 16 16:59:14.626585 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:59:14.626525    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm" podUID="4a22d4d0-042c-4fbb-b739-441ef61aaadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8001/health\": dial tcp 10.132.0.42:8001: connect: connection refused"
Sep 16 16:59:14.636037 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:59:14.636006    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq" podUID="d3bb8154-638e-4c93-98e3-41a56d965948" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 16 16:59:24.626620 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:59:24.626568    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm" podUID="4a22d4d0-042c-4fbb-b739-441ef61aaadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8001/health\": dial tcp 10.132.0.42:8001: connect: connection refused"
Sep 16 16:59:24.635961 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:59:24.635922    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq" podUID="d3bb8154-638e-4c93-98e3-41a56d965948" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 16 16:59:26.567374 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:59:26.567339    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:59:26.627336 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:59:26.627304    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:59:34.627074 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:59:34.627024    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm" podUID="4a22d4d0-042c-4fbb-b739-441ef61aaadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8001/health\": dial tcp 10.132.0.42:8001: connect: connection refused"
Sep 16 16:59:34.636024 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:59:34.635987    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq" podUID="d3bb8154-638e-4c93-98e3-41a56d965948" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 16 16:59:44.636310 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:59:44.636280    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm"
Sep 16 16:59:44.645895 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:59:44.645867    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq"
Sep 16 16:59:44.648228 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:59:44.648205    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm"
Sep 16 16:59:44.653834 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:59:44.653815    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq"
Sep 16 16:59:47.225998 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:59:47.225962    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:59:47.327494 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:59:47.327464    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:59:54.101601 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:59:54.101561    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 16:59:54.112761 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:59:54.111852    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm"]
Sep 16 16:59:54.112761 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:59:54.112258    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq"]
Sep 16 16:59:54.165198 ip-10-0-136-228 kubenswrapper[2554]: I0916 16:59:54.165166    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:00:05.298918 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:05.298885    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:00:07.611611 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:07.611580    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:00:07.997593 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:07.997518    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:00:08.098245 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:08.098213    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:00:09.136686 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:09.136644    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm" podUID="4a22d4d0-042c-4fbb-b739-441ef61aaadf" containerName="main" containerID="cri-o://54bc974500241591b5f63f939006add20caeb01b58b96f20b51d66afadb9df65" gracePeriod=45
Sep 16 17:00:09.137282 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:09.136811    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq" podUID="d3bb8154-638e-4c93-98e3-41a56d965948" containerName="main" containerID="cri-o://9919df28f4b4cf5be933b2305f6513a1112e58c1bbe4a8c0feada2aabeba7871" gracePeriod=45
Sep 16 17:00:09.248215 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:09.248174    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:00:09.948909 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:09.948879    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm" podUID="4a22d4d0-042c-4fbb-b739-441ef61aaadf" containerName="llm-d-routing-sidecar" containerID="cri-o://ffe2b4bb95d733b4213b00b03ecf7be7cba09bd01471c0c8782e8f7e7f279fe5" gracePeriod=45
Sep 16 17:00:10.066385 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.066365    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq"
Sep 16 17:00:10.105938 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.105850    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3bb8154-638e-4c93-98e3-41a56d965948-tls-certs\") pod \"d3bb8154-638e-4c93-98e3-41a56d965948\" (UID: \"d3bb8154-638e-4c93-98e3-41a56d965948\") "
Sep 16 17:00:10.106109 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.105939    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3bb8154-638e-4c93-98e3-41a56d965948-tmp-dir\") pod \"d3bb8154-638e-4c93-98e3-41a56d965948\" (UID: \"d3bb8154-638e-4c93-98e3-41a56d965948\") "
Sep 16 17:00:10.106109 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.105968    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3bb8154-638e-4c93-98e3-41a56d965948-cabundle-cert\") pod \"d3bb8154-638e-4c93-98e3-41a56d965948\" (UID: \"d3bb8154-638e-4c93-98e3-41a56d965948\") "
Sep 16 17:00:10.106109 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.105998    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3bb8154-638e-4c93-98e3-41a56d965948-home\") pod \"d3bb8154-638e-4c93-98e3-41a56d965948\" (UID: \"d3bb8154-638e-4c93-98e3-41a56d965948\") "
Sep 16 17:00:10.106109 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.106029    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3bb8154-638e-4c93-98e3-41a56d965948-kserve-provision-location\") pod \"d3bb8154-638e-4c93-98e3-41a56d965948\" (UID: \"d3bb8154-638e-4c93-98e3-41a56d965948\") "
Sep 16 17:00:10.106109 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.106067    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3bb8154-638e-4c93-98e3-41a56d965948-model-cache\") pod \"d3bb8154-638e-4c93-98e3-41a56d965948\" (UID: \"d3bb8154-638e-4c93-98e3-41a56d965948\") "
Sep 16 17:00:10.106404 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.106129    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3bb8154-638e-4c93-98e3-41a56d965948-dshm\") pod \"d3bb8154-638e-4c93-98e3-41a56d965948\" (UID: \"d3bb8154-638e-4c93-98e3-41a56d965948\") "
Sep 16 17:00:10.106404 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.106159    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6r2l\" (UniqueName: \"kubernetes.io/projected/d3bb8154-638e-4c93-98e3-41a56d965948-kube-api-access-t6r2l\") pod \"d3bb8154-638e-4c93-98e3-41a56d965948\" (UID: \"d3bb8154-638e-4c93-98e3-41a56d965948\") "
Sep 16 17:00:10.106613 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.106579    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3bb8154-638e-4c93-98e3-41a56d965948-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d3bb8154-638e-4c93-98e3-41a56d965948" (UID: "d3bb8154-638e-4c93-98e3-41a56d965948"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:00:10.106817 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.106777    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3bb8154-638e-4c93-98e3-41a56d965948-model-cache" (OuterVolumeSpecName: "model-cache") pod "d3bb8154-638e-4c93-98e3-41a56d965948" (UID: "d3bb8154-638e-4c93-98e3-41a56d965948"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:00:10.108783 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.108755    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3bb8154-638e-4c93-98e3-41a56d965948-kube-api-access-t6r2l" (OuterVolumeSpecName: "kube-api-access-t6r2l") pod "d3bb8154-638e-4c93-98e3-41a56d965948" (UID: "d3bb8154-638e-4c93-98e3-41a56d965948"). InnerVolumeSpecName "kube-api-access-t6r2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:00:10.109061 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.108899    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3bb8154-638e-4c93-98e3-41a56d965948-home" (OuterVolumeSpecName: "home") pod "d3bb8154-638e-4c93-98e3-41a56d965948" (UID: "d3bb8154-638e-4c93-98e3-41a56d965948"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:00:10.109061 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.109005    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3bb8154-638e-4c93-98e3-41a56d965948-dshm" (OuterVolumeSpecName: "dshm") pod "d3bb8154-638e-4c93-98e3-41a56d965948" (UID: "d3bb8154-638e-4c93-98e3-41a56d965948"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:00:10.109061 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.109054    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3bb8154-638e-4c93-98e3-41a56d965948-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d3bb8154-638e-4c93-98e3-41a56d965948" (UID: "d3bb8154-638e-4c93-98e3-41a56d965948"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:00:10.125386 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.125335    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3bb8154-638e-4c93-98e3-41a56d965948-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d3bb8154-638e-4c93-98e3-41a56d965948" (UID: "d3bb8154-638e-4c93-98e3-41a56d965948"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:00:10.135984 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.135949    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3bb8154-638e-4c93-98e3-41a56d965948-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d3bb8154-638e-4c93-98e3-41a56d965948" (UID: "d3bb8154-638e-4c93-98e3-41a56d965948"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:00:10.184111 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.184086    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm"
Sep 16 17:00:10.206672 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.206643    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfd6l\" (UniqueName: \"kubernetes.io/projected/4a22d4d0-042c-4fbb-b739-441ef61aaadf-kube-api-access-tfd6l\") pod \"4a22d4d0-042c-4fbb-b739-441ef61aaadf\" (UID: \"4a22d4d0-042c-4fbb-b739-441ef61aaadf\") "
Sep 16 17:00:10.206821 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.206686    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a22d4d0-042c-4fbb-b739-441ef61aaadf-cabundle-cert\") pod \"4a22d4d0-042c-4fbb-b739-441ef61aaadf\" (UID: \"4a22d4d0-042c-4fbb-b739-441ef61aaadf\") "
Sep 16 17:00:10.206821 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.206710    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a22d4d0-042c-4fbb-b739-441ef61aaadf-tls-certs\") pod \"4a22d4d0-042c-4fbb-b739-441ef61aaadf\" (UID: \"4a22d4d0-042c-4fbb-b739-441ef61aaadf\") "
Sep 16 17:00:10.207002 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.206978    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a22d4d0-042c-4fbb-b739-441ef61aaadf-model-cache\") pod \"4a22d4d0-042c-4fbb-b739-441ef61aaadf\" (UID: \"4a22d4d0-042c-4fbb-b739-441ef61aaadf\") "
Sep 16 17:00:10.207087 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.207023    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a22d4d0-042c-4fbb-b739-441ef61aaadf-home\") pod \"4a22d4d0-042c-4fbb-b739-441ef61aaadf\" (UID: \"4a22d4d0-042c-4fbb-b739-441ef61aaadf\") "
Sep 16 17:00:10.207087 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.207052    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a22d4d0-042c-4fbb-b739-441ef61aaadf-tmp-dir\") pod \"4a22d4d0-042c-4fbb-b739-441ef61aaadf\" (UID: \"4a22d4d0-042c-4fbb-b739-441ef61aaadf\") "
Sep 16 17:00:10.207087 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.207078    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a22d4d0-042c-4fbb-b739-441ef61aaadf-kserve-provision-location\") pod \"4a22d4d0-042c-4fbb-b739-441ef61aaadf\" (UID: \"4a22d4d0-042c-4fbb-b739-441ef61aaadf\") "
Sep 16 17:00:10.208157 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.207114    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a22d4d0-042c-4fbb-b739-441ef61aaadf-dshm\") pod \"4a22d4d0-042c-4fbb-b739-441ef61aaadf\" (UID: \"4a22d4d0-042c-4fbb-b739-441ef61aaadf\") "
Sep 16 17:00:10.208157 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.207164    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a22d4d0-042c-4fbb-b739-441ef61aaadf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4a22d4d0-042c-4fbb-b739-441ef61aaadf" (UID: "4a22d4d0-042c-4fbb-b739-441ef61aaadf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:00:10.208157 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.207275    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a22d4d0-042c-4fbb-b739-441ef61aaadf-model-cache" (OuterVolumeSpecName: "model-cache") pod "4a22d4d0-042c-4fbb-b739-441ef61aaadf" (UID: "4a22d4d0-042c-4fbb-b739-441ef61aaadf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:00:10.208157 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.207483    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3bb8154-638e-4c93-98e3-41a56d965948-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:00:10.208157 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.207532    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3bb8154-638e-4c93-98e3-41a56d965948-model-cache\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:00:10.208157 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.207549    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a22d4d0-042c-4fbb-b739-441ef61aaadf-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:00:10.208157 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.207601    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3bb8154-638e-4c93-98e3-41a56d965948-dshm\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:00:10.208157 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.207616    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6r2l\" (UniqueName: \"kubernetes.io/projected/d3bb8154-638e-4c93-98e3-41a56d965948-kube-api-access-t6r2l\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:00:10.208157 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.207630    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a22d4d0-042c-4fbb-b739-441ef61aaadf-model-cache\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:00:10.208157 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.207646    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3bb8154-638e-4c93-98e3-41a56d965948-tls-certs\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:00:10.208157 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.207659    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3bb8154-638e-4c93-98e3-41a56d965948-tmp-dir\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:00:10.208157 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.207673    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3bb8154-638e-4c93-98e3-41a56d965948-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:00:10.208157 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.207686    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3bb8154-638e-4c93-98e3-41a56d965948-home\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:00:10.208819 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.208676    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a22d4d0-042c-4fbb-b739-441ef61aaadf-home" (OuterVolumeSpecName: "home") pod "4a22d4d0-042c-4fbb-b739-441ef61aaadf" (UID: "4a22d4d0-042c-4fbb-b739-441ef61aaadf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:00:10.209327 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.209214    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a22d4d0-042c-4fbb-b739-441ef61aaadf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4a22d4d0-042c-4fbb-b739-441ef61aaadf" (UID: "4a22d4d0-042c-4fbb-b739-441ef61aaadf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:00:10.209327 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.209303    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a22d4d0-042c-4fbb-b739-441ef61aaadf-kube-api-access-tfd6l" (OuterVolumeSpecName: "kube-api-access-tfd6l") pod "4a22d4d0-042c-4fbb-b739-441ef61aaadf" (UID: "4a22d4d0-042c-4fbb-b739-441ef61aaadf"). InnerVolumeSpecName "kube-api-access-tfd6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:00:10.209592 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.209571    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a22d4d0-042c-4fbb-b739-441ef61aaadf-dshm" (OuterVolumeSpecName: "dshm") pod "4a22d4d0-042c-4fbb-b739-441ef61aaadf" (UID: "4a22d4d0-042c-4fbb-b739-441ef61aaadf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:00:10.221049 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.221017    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a22d4d0-042c-4fbb-b739-441ef61aaadf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4a22d4d0-042c-4fbb-b739-441ef61aaadf" (UID: "4a22d4d0-042c-4fbb-b739-441ef61aaadf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:00:10.229407 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.229378    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a22d4d0-042c-4fbb-b739-441ef61aaadf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4a22d4d0-042c-4fbb-b739-441ef61aaadf" (UID: "4a22d4d0-042c-4fbb-b739-441ef61aaadf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:00:10.308740 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.308700    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a22d4d0-042c-4fbb-b739-441ef61aaadf-tls-certs\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:00:10.308740 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.308731    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a22d4d0-042c-4fbb-b739-441ef61aaadf-home\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:00:10.308740 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.308740    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a22d4d0-042c-4fbb-b739-441ef61aaadf-tmp-dir\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:00:10.308740 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.308749    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a22d4d0-042c-4fbb-b739-441ef61aaadf-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:00:10.309018 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.308759    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a22d4d0-042c-4fbb-b739-441ef61aaadf-dshm\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:00:10.309018 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.308769    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfd6l\" (UniqueName: \"kubernetes.io/projected/4a22d4d0-042c-4fbb-b739-441ef61aaadf-kube-api-access-tfd6l\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:00:10.802093 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.802062    2554 generic.go:356] "Generic (PLEG): container finished" podID="4a22d4d0-042c-4fbb-b739-441ef61aaadf" containerID="54bc974500241591b5f63f939006add20caeb01b58b96f20b51d66afadb9df65" exitCode=0
Sep 16 17:00:10.802093 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.802087    2554 generic.go:356] "Generic (PLEG): container finished" podID="4a22d4d0-042c-4fbb-b739-441ef61aaadf" containerID="ffe2b4bb95d733b4213b00b03ecf7be7cba09bd01471c0c8782e8f7e7f279fe5" exitCode=0
Sep 16 17:00:10.802319 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.802117    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm"
Sep 16 17:00:10.802319 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.802146    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm" event={"ID":"4a22d4d0-042c-4fbb-b739-441ef61aaadf","Type":"ContainerDied","Data":"54bc974500241591b5f63f939006add20caeb01b58b96f20b51d66afadb9df65"}
Sep 16 17:00:10.802319 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.802200    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm" event={"ID":"4a22d4d0-042c-4fbb-b739-441ef61aaadf","Type":"ContainerDied","Data":"ffe2b4bb95d733b4213b00b03ecf7be7cba09bd01471c0c8782e8f7e7f279fe5"}
Sep 16 17:00:10.802319 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.802217    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm" event={"ID":"4a22d4d0-042c-4fbb-b739-441ef61aaadf","Type":"ContainerDied","Data":"1716e5eaa402041046be7d190471bbbea79d08d4d40d2e063c9bc2fb1c8bbc90"}
Sep 16 17:00:10.802319 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.802244    2554 scope.go:117] "RemoveContainer" containerID="54bc974500241591b5f63f939006add20caeb01b58b96f20b51d66afadb9df65"
Sep 16 17:00:10.803923 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.803903    2554 generic.go:356] "Generic (PLEG): container finished" podID="d3bb8154-638e-4c93-98e3-41a56d965948" containerID="9919df28f4b4cf5be933b2305f6513a1112e58c1bbe4a8c0feada2aabeba7871" exitCode=0
Sep 16 17:00:10.804021 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.803934    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq" event={"ID":"d3bb8154-638e-4c93-98e3-41a56d965948","Type":"ContainerDied","Data":"9919df28f4b4cf5be933b2305f6513a1112e58c1bbe4a8c0feada2aabeba7871"}
Sep 16 17:00:10.804021 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.803972    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq"
Sep 16 17:00:10.804021 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.803987    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq" event={"ID":"d3bb8154-638e-4c93-98e3-41a56d965948","Type":"ContainerDied","Data":"b075ed60920ab2b4b49f8c18cf02ea3552655eb776dd23948f636e83240ae716"}
Sep 16 17:00:10.812423 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.812403    2554 scope.go:117] "RemoveContainer" containerID="67d71d40d5c1a4d679cbfe9718a2f23e479516fecc99a731a4ad1834fb2ac266"
Sep 16 17:00:10.819718 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.819702    2554 scope.go:117] "RemoveContainer" containerID="ffe2b4bb95d733b4213b00b03ecf7be7cba09bd01471c0c8782e8f7e7f279fe5"
Sep 16 17:00:10.826970 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.826723    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm"]
Sep 16 17:00:10.826970 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.826864    2554 scope.go:117] "RemoveContainer" containerID="54bc974500241591b5f63f939006add20caeb01b58b96f20b51d66afadb9df65"
Sep 16 17:00:10.827148 ip-10-0-136-228 kubenswrapper[2554]: E0916 17:00:10.827127    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54bc974500241591b5f63f939006add20caeb01b58b96f20b51d66afadb9df65\": container with ID starting with 54bc974500241591b5f63f939006add20caeb01b58b96f20b51d66afadb9df65 not found: ID does not exist" containerID="54bc974500241591b5f63f939006add20caeb01b58b96f20b51d66afadb9df65"
Sep 16 17:00:10.827191 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.827155    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54bc974500241591b5f63f939006add20caeb01b58b96f20b51d66afadb9df65"} err="failed to get container status \"54bc974500241591b5f63f939006add20caeb01b58b96f20b51d66afadb9df65\": rpc error: code = NotFound desc = could not find container \"54bc974500241591b5f63f939006add20caeb01b58b96f20b51d66afadb9df65\": container with ID starting with 54bc974500241591b5f63f939006add20caeb01b58b96f20b51d66afadb9df65 not found: ID does not exist"
Sep 16 17:00:10.827191 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.827173    2554 scope.go:117] "RemoveContainer" containerID="67d71d40d5c1a4d679cbfe9718a2f23e479516fecc99a731a4ad1834fb2ac266"
Sep 16 17:00:10.827454 ip-10-0-136-228 kubenswrapper[2554]: E0916 17:00:10.827437    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67d71d40d5c1a4d679cbfe9718a2f23e479516fecc99a731a4ad1834fb2ac266\": container with ID starting with 67d71d40d5c1a4d679cbfe9718a2f23e479516fecc99a731a4ad1834fb2ac266 not found: ID does not exist" containerID="67d71d40d5c1a4d679cbfe9718a2f23e479516fecc99a731a4ad1834fb2ac266"
Sep 16 17:00:10.827517 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.827462    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67d71d40d5c1a4d679cbfe9718a2f23e479516fecc99a731a4ad1834fb2ac266"} err="failed to get container status \"67d71d40d5c1a4d679cbfe9718a2f23e479516fecc99a731a4ad1834fb2ac266\": rpc error: code = NotFound desc = could not find container \"67d71d40d5c1a4d679cbfe9718a2f23e479516fecc99a731a4ad1834fb2ac266\": container with ID starting with 67d71d40d5c1a4d679cbfe9718a2f23e479516fecc99a731a4ad1834fb2ac266 not found: ID does not exist"
Sep 16 17:00:10.827517 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.827483    2554 scope.go:117] "RemoveContainer" containerID="ffe2b4bb95d733b4213b00b03ecf7be7cba09bd01471c0c8782e8f7e7f279fe5"
Sep 16 17:00:10.827746 ip-10-0-136-228 kubenswrapper[2554]: E0916 17:00:10.827729    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffe2b4bb95d733b4213b00b03ecf7be7cba09bd01471c0c8782e8f7e7f279fe5\": container with ID starting with ffe2b4bb95d733b4213b00b03ecf7be7cba09bd01471c0c8782e8f7e7f279fe5 not found: ID does not exist" containerID="ffe2b4bb95d733b4213b00b03ecf7be7cba09bd01471c0c8782e8f7e7f279fe5"
Sep 16 17:00:10.827835 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.827752    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffe2b4bb95d733b4213b00b03ecf7be7cba09bd01471c0c8782e8f7e7f279fe5"} err="failed to get container status \"ffe2b4bb95d733b4213b00b03ecf7be7cba09bd01471c0c8782e8f7e7f279fe5\": rpc error: code = NotFound desc = could not find container \"ffe2b4bb95d733b4213b00b03ecf7be7cba09bd01471c0c8782e8f7e7f279fe5\": container with ID starting with ffe2b4bb95d733b4213b00b03ecf7be7cba09bd01471c0c8782e8f7e7f279fe5 not found: ID does not exist"
Sep 16 17:00:10.827835 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.827774    2554 scope.go:117] "RemoveContainer" containerID="54bc974500241591b5f63f939006add20caeb01b58b96f20b51d66afadb9df65"
Sep 16 17:00:10.828072 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.828021    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54bc974500241591b5f63f939006add20caeb01b58b96f20b51d66afadb9df65"} err="failed to get container status \"54bc974500241591b5f63f939006add20caeb01b58b96f20b51d66afadb9df65\": rpc error: code = NotFound desc = could not find container \"54bc974500241591b5f63f939006add20caeb01b58b96f20b51d66afadb9df65\": container with ID starting with 54bc974500241591b5f63f939006add20caeb01b58b96f20b51d66afadb9df65 not found: ID does not exist"
Sep 16 17:00:10.828221 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.828200    2554 scope.go:117] "RemoveContainer" containerID="67d71d40d5c1a4d679cbfe9718a2f23e479516fecc99a731a4ad1834fb2ac266"
Sep 16 17:00:10.828600 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.828569    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67d71d40d5c1a4d679cbfe9718a2f23e479516fecc99a731a4ad1834fb2ac266"} err="failed to get container status \"67d71d40d5c1a4d679cbfe9718a2f23e479516fecc99a731a4ad1834fb2ac266\": rpc error: code = NotFound desc = could not find container \"67d71d40d5c1a4d679cbfe9718a2f23e479516fecc99a731a4ad1834fb2ac266\": container with ID starting with 67d71d40d5c1a4d679cbfe9718a2f23e479516fecc99a731a4ad1834fb2ac266 not found: ID does not exist"
Sep 16 17:00:10.828706 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.828596    2554 scope.go:117] "RemoveContainer" containerID="ffe2b4bb95d733b4213b00b03ecf7be7cba09bd01471c0c8782e8f7e7f279fe5"
Sep 16 17:00:10.829284 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.829183    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffe2b4bb95d733b4213b00b03ecf7be7cba09bd01471c0c8782e8f7e7f279fe5"} err="failed to get container status \"ffe2b4bb95d733b4213b00b03ecf7be7cba09bd01471c0c8782e8f7e7f279fe5\": rpc error: code = NotFound desc = could not find container \"ffe2b4bb95d733b4213b00b03ecf7be7cba09bd01471c0c8782e8f7e7f279fe5\": container with ID starting with ffe2b4bb95d733b4213b00b03ecf7be7cba09bd01471c0c8782e8f7e7f279fe5 not found: ID does not exist"
Sep 16 17:00:10.829284 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.829211    2554 scope.go:117] "RemoveContainer" containerID="9919df28f4b4cf5be933b2305f6513a1112e58c1bbe4a8c0feada2aabeba7871"
Sep 16 17:00:10.831169 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.831148    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-66556b59bb95vjm"]
Sep 16 17:00:10.836826 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.836810    2554 scope.go:117] "RemoveContainer" containerID="beb0f9c0c3ecec106255cc9a7248ed1dcd71af199cf06f45dd1185a1985698c3"
Sep 16 17:00:10.842294 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.842272    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq"]
Sep 16 17:00:10.844697 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.844681    2554 scope.go:117] "RemoveContainer" containerID="9919df28f4b4cf5be933b2305f6513a1112e58c1bbe4a8c0feada2aabeba7871"
Sep 16 17:00:10.845013 ip-10-0-136-228 kubenswrapper[2554]: E0916 17:00:10.844993    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9919df28f4b4cf5be933b2305f6513a1112e58c1bbe4a8c0feada2aabeba7871\": container with ID starting with 9919df28f4b4cf5be933b2305f6513a1112e58c1bbe4a8c0feada2aabeba7871 not found: ID does not exist" containerID="9919df28f4b4cf5be933b2305f6513a1112e58c1bbe4a8c0feada2aabeba7871"
Sep 16 17:00:10.845074 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.845017    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9919df28f4b4cf5be933b2305f6513a1112e58c1bbe4a8c0feada2aabeba7871"} err="failed to get container status \"9919df28f4b4cf5be933b2305f6513a1112e58c1bbe4a8c0feada2aabeba7871\": rpc error: code = NotFound desc = could not find container \"9919df28f4b4cf5be933b2305f6513a1112e58c1bbe4a8c0feada2aabeba7871\": container with ID starting with 9919df28f4b4cf5be933b2305f6513a1112e58c1bbe4a8c0feada2aabeba7871 not found: ID does not exist"
Sep 16 17:00:10.845074 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.845032    2554 scope.go:117] "RemoveContainer" containerID="beb0f9c0c3ecec106255cc9a7248ed1dcd71af199cf06f45dd1185a1985698c3"
Sep 16 17:00:10.845329 ip-10-0-136-228 kubenswrapper[2554]: E0916 17:00:10.845309    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beb0f9c0c3ecec106255cc9a7248ed1dcd71af199cf06f45dd1185a1985698c3\": container with ID starting with beb0f9c0c3ecec106255cc9a7248ed1dcd71af199cf06f45dd1185a1985698c3 not found: ID does not exist" containerID="beb0f9c0c3ecec106255cc9a7248ed1dcd71af199cf06f45dd1185a1985698c3"
Sep 16 17:00:10.845382 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.845337    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beb0f9c0c3ecec106255cc9a7248ed1dcd71af199cf06f45dd1185a1985698c3"} err="failed to get container status \"beb0f9c0c3ecec106255cc9a7248ed1dcd71af199cf06f45dd1185a1985698c3\": rpc error: code = NotFound desc = could not find container \"beb0f9c0c3ecec106255cc9a7248ed1dcd71af199cf06f45dd1185a1985698c3\": container with ID starting with beb0f9c0c3ecec106255cc9a7248ed1dcd71af199cf06f45dd1185a1985698c3 not found: ID does not exist"
Sep 16 17:00:10.845888 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:10.845756    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-78gcztq"]
Sep 16 17:00:11.724626 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:11.724595    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:00:11.887319 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:11.887277    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:00:12.080506 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:12.080471    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a22d4d0-042c-4fbb-b739-441ef61aaadf" path="/var/lib/kubelet/pods/4a22d4d0-042c-4fbb-b739-441ef61aaadf/volumes"
Sep 16 17:00:12.080974 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:12.080960    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3bb8154-638e-4c93-98e3-41a56d965948" path="/var/lib/kubelet/pods/d3bb8154-638e-4c93-98e3-41a56d965948/volumes"
Sep 16 17:00:12.124519 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:12.124488    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:00:18.268648 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:18.268613    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:00:18.363733 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:18.363702    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:00:18.915709 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:18.915677    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:00:18.987200 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:18.987168    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:00:32.172770 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:32.172695    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn"]
Sep 16 17:00:32.173119 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:32.172967    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a22d4d0-042c-4fbb-b739-441ef61aaadf" containerName="main"
Sep 16 17:00:32.173119 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:32.172981    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a22d4d0-042c-4fbb-b739-441ef61aaadf" containerName="main"
Sep 16 17:00:32.173119 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:32.172999    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3bb8154-638e-4c93-98e3-41a56d965948" containerName="storage-initializer"
Sep 16 17:00:32.173119 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:32.173006    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3bb8154-638e-4c93-98e3-41a56d965948" containerName="storage-initializer"
Sep 16 17:00:32.173119 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:32.173013    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a22d4d0-042c-4fbb-b739-441ef61aaadf" containerName="storage-initializer"
Sep 16 17:00:32.173119 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:32.173020    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a22d4d0-042c-4fbb-b739-441ef61aaadf" containerName="storage-initializer"
Sep 16 17:00:32.173119 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:32.173027    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a22d4d0-042c-4fbb-b739-441ef61aaadf" containerName="llm-d-routing-sidecar"
Sep 16 17:00:32.173119 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:32.173033    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a22d4d0-042c-4fbb-b739-441ef61aaadf" containerName="llm-d-routing-sidecar"
Sep 16 17:00:32.173119 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:32.173046    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3bb8154-638e-4c93-98e3-41a56d965948" containerName="main"
Sep 16 17:00:32.173119 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:32.173054    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3bb8154-638e-4c93-98e3-41a56d965948" containerName="main"
Sep 16 17:00:32.173119 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:32.173095    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a22d4d0-042c-4fbb-b739-441ef61aaadf" containerName="llm-d-routing-sidecar"
Sep 16 17:00:32.173119 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:32.173102    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a22d4d0-042c-4fbb-b739-441ef61aaadf" containerName="main"
Sep 16 17:00:32.173119 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:32.173110    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3bb8154-638e-4c93-98e3-41a56d965948" containerName="main"
Sep 16 17:00:32.175970 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:32.175953    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn"
Sep 16 17:00:32.178983 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:32.178953    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 16 17:00:32.179111 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:32.179016    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-rfdsk\""
Sep 16 17:00:32.179111 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:32.179045    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 16 17:00:32.179111 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:32.179018    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"lora-pvc-adapters-test-kserve-self-signed-certs\""
Sep 16 17:00:32.187709 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:32.187687    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn"]
Sep 16 17:00:32.200251 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:32.200230    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:00:32.265903 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:32.265870    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acacc924-9d50-4e79-89a9-aebfe4e2d050-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn\" (UID: \"acacc924-9d50-4e79-89a9-aebfe4e2d050\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn"
Sep 16 17:00:32.266080 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:32.265913    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acacc924-9d50-4e79-89a9-aebfe4e2d050-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn\" (UID: \"acacc924-9d50-4e79-89a9-aebfe4e2d050\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn"
Sep 16 17:00:32.266080 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:32.265944    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acacc924-9d50-4e79-89a9-aebfe4e2d050-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn\" (UID: \"acacc924-9d50-4e79-89a9-aebfe4e2d050\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn"
Sep 16 17:00:32.266080 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:32.265993    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acacc924-9d50-4e79-89a9-aebfe4e2d050-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn\" (UID: \"acacc924-9d50-4e79-89a9-aebfe4e2d050\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn"
Sep 16 17:00:32.266080 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:32.266019    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv8t5\" (UniqueName: \"kubernetes.io/projected/acacc924-9d50-4e79-89a9-aebfe4e2d050-kube-api-access-fv8t5\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn\" (UID: \"acacc924-9d50-4e79-89a9-aebfe4e2d050\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn"
Sep 16 17:00:32.266080 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:32.266039    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acacc924-9d50-4e79-89a9-aebfe4e2d050-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn\" (UID: \"acacc924-9d50-4e79-89a9-aebfe4e2d050\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn"
Sep 16 17:00:32.367222 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:32.367184    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acacc924-9d50-4e79-89a9-aebfe4e2d050-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn\" (UID: \"acacc924-9d50-4e79-89a9-aebfe4e2d050\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn"
Sep 16 17:00:32.367398 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:32.367231    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acacc924-9d50-4e79-89a9-aebfe4e2d050-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn\" (UID: \"acacc924-9d50-4e79-89a9-aebfe4e2d050\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn"
Sep 16 17:00:32.367398 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:32.367261    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv8t5\" (UniqueName: \"kubernetes.io/projected/acacc924-9d50-4e79-89a9-aebfe4e2d050-kube-api-access-fv8t5\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn\" (UID: \"acacc924-9d50-4e79-89a9-aebfe4e2d050\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn"
Sep 16 17:00:32.367398 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:32.367299    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acacc924-9d50-4e79-89a9-aebfe4e2d050-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn\" (UID: \"acacc924-9d50-4e79-89a9-aebfe4e2d050\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn"
Sep 16 17:00:32.367398 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:32.367333    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acacc924-9d50-4e79-89a9-aebfe4e2d050-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn\" (UID: \"acacc924-9d50-4e79-89a9-aebfe4e2d050\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn"
Sep 16 17:00:32.367398 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:32.367381    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acacc924-9d50-4e79-89a9-aebfe4e2d050-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn\" (UID: \"acacc924-9d50-4e79-89a9-aebfe4e2d050\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn"
Sep 16 17:00:32.367712 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:32.367650    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acacc924-9d50-4e79-89a9-aebfe4e2d050-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn\" (UID: \"acacc924-9d50-4e79-89a9-aebfe4e2d050\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn"
Sep 16 17:00:32.367712 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:32.367679    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acacc924-9d50-4e79-89a9-aebfe4e2d050-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn\" (UID: \"acacc924-9d50-4e79-89a9-aebfe4e2d050\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn"
Sep 16 17:00:32.367873 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:32.367851    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acacc924-9d50-4e79-89a9-aebfe4e2d050-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn\" (UID: \"acacc924-9d50-4e79-89a9-aebfe4e2d050\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn"
Sep 16 17:00:32.369596 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:32.369571    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acacc924-9d50-4e79-89a9-aebfe4e2d050-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn\" (UID: \"acacc924-9d50-4e79-89a9-aebfe4e2d050\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn"
Sep 16 17:00:32.369741 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:32.369723    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acacc924-9d50-4e79-89a9-aebfe4e2d050-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn\" (UID: \"acacc924-9d50-4e79-89a9-aebfe4e2d050\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn"
Sep 16 17:00:32.375502 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:32.375482    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv8t5\" (UniqueName: \"kubernetes.io/projected/acacc924-9d50-4e79-89a9-aebfe4e2d050-kube-api-access-fv8t5\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn\" (UID: \"acacc924-9d50-4e79-89a9-aebfe4e2d050\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn"
Sep 16 17:00:34.883335 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:34.883298    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-f6a115bc-fe2c-4d81-8231-1cd9cb25028e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0798b6fdf777dd2e5\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn\" (UID: \"acacc924-9d50-4e79-89a9-aebfe4e2d050\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn"
Sep 16 17:00:34.889504 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:34.889476    2554 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-f6a115bc-fe2c-4d81-8231-1cd9cb25028e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0798b6fdf777dd2e5\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn\" (UID: \"acacc924-9d50-4e79-89a9-aebfe4e2d050\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn"
Sep 16 17:00:34.984457 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:34.984421    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-f6a115bc-fe2c-4d81-8231-1cd9cb25028e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0798b6fdf777dd2e5\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn\" (UID: \"acacc924-9d50-4e79-89a9-aebfe4e2d050\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn"
Sep 16 17:00:34.984595 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:34.984520    2554 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-f6a115bc-fe2c-4d81-8231-1cd9cb25028e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0798b6fdf777dd2e5\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn\" (UID: \"acacc924-9d50-4e79-89a9-aebfe4e2d050\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn"
Sep 16 17:00:34.987563 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:34.987546    2554 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-f6a115bc-fe2c-4d81-8231-1cd9cb25028e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0798b6fdf777dd2e5\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn\" (UID: \"acacc924-9d50-4e79-89a9-aebfe4e2d050\") DevicePath \"csi-a525d0065135705af2c00989850bba3017699585805c261d976bce53b10f04b5\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn"
Sep 16 17:00:35.081041 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:35.081011    2554 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-f6a115bc-fe2c-4d81-8231-1cd9cb25028e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0798b6fdf777dd2e5\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn\" (UID: \"acacc924-9d50-4e79-89a9-aebfe4e2d050\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/9e751b0d9c270464a55c9c1a9c7fad2b709bde7e1c07c4c747f57d8ce0e87f02/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn"
Sep 16 17:00:35.099372 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:35.099346    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-f6a115bc-fe2c-4d81-8231-1cd9cb25028e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0798b6fdf777dd2e5\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn\" (UID: \"acacc924-9d50-4e79-89a9-aebfe4e2d050\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn"
Sep 16 17:00:35.185421 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:35.185345    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn"
Sep 16 17:00:35.459695 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:35.459666    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn"]
Sep 16 17:00:35.461684 ip-10-0-136-228 kubenswrapper[2554]: W0916 17:00:35.461645    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacacc924_9d50_4e79_89a9_aebfe4e2d050.slice/crio-a2b96f32bc6dc8fc78a33ac9092d06057ab8bfbd86aaf3505a572dd4044e1d78 WatchSource:0}: Error finding container a2b96f32bc6dc8fc78a33ac9092d06057ab8bfbd86aaf3505a572dd4044e1d78: Status 404 returned error can't find the container with id a2b96f32bc6dc8fc78a33ac9092d06057ab8bfbd86aaf3505a572dd4044e1d78
Sep 16 17:00:35.896025 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:35.895986    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn" event={"ID":"acacc924-9d50-4e79-89a9-aebfe4e2d050","Type":"ContainerStarted","Data":"ebcfde7d67560a36575d24193f1d05eafd48e5812d3f0aaeb04915b41554e64f"}
Sep 16 17:00:35.896524 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:35.896049    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn" event={"ID":"acacc924-9d50-4e79-89a9-aebfe4e2d050","Type":"ContainerStarted","Data":"a2b96f32bc6dc8fc78a33ac9092d06057ab8bfbd86aaf3505a572dd4044e1d78"}
Sep 16 17:00:35.916751 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:35.916705    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn" podStartSLOduration=3.916692841 podStartE2EDuration="3.916692841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:00:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:00:35.915345395 +0000 UTC m=+4236.360037602" watchObservedRunningTime="2026-09-16 17:00:35.916692841 +0000 UTC m=+4236.361385018"
Sep 16 17:00:36.185789 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:36.185703    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn"
Sep 16 17:00:44.236731 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:44.236698    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:00:44.335935 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:44.335899    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:00:45.186377 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:45.186338    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn"
Sep 16 17:00:45.187953 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:45.187917    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn" podUID="acacc924-9d50-4e79-89a9-aebfe4e2d050" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Sep 16 17:00:50.645840 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:50.645788    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:00:55.185878 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:55.185835    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn" podUID="acacc924-9d50-4e79-89a9-aebfe4e2d050" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Sep 16 17:00:56.348738 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:56.348708    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:00:56.411476 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:00:56.411441    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:01:05.186510 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:05.186472    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn" podUID="acacc924-9d50-4e79-89a9-aebfe4e2d050" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Sep 16 17:01:15.186787 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:15.186736    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn" podUID="acacc924-9d50-4e79-89a9-aebfe4e2d050" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Sep 16 17:01:25.186674 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:25.186632    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn" podUID="acacc924-9d50-4e79-89a9-aebfe4e2d050" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Sep 16 17:01:35.186086 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:35.186046    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn" podUID="acacc924-9d50-4e79-89a9-aebfe4e2d050" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Sep 16 17:01:36.958887 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:36.958822    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq"]
Sep 16 17:01:36.961263 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:36.961244    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq"
Sep 16 17:01:36.964495 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:36.964440    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 17:01:36.964615 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:36.964508    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 16 17:01:36.964739 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:36.964718    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-67t67\""
Sep 16 17:01:36.964843 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:36.964734    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 16 17:01:36.964843 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:36.964749    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 16 17:01:36.964843 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:36.964719    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 16 17:01:36.974161 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:36.974137    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq"]
Sep 16 17:01:36.995540 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:36.995517    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:01:37.064940 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:37.064899    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq\" (UID: \"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq"
Sep 16 17:01:37.065083 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:37.064959    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq\" (UID: \"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq"
Sep 16 17:01:37.065083 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:37.064988    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq\" (UID: \"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq"
Sep 16 17:01:37.065083 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:37.065070    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq\" (UID: \"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq"
Sep 16 17:01:37.065193 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:37.065125    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq\" (UID: \"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq"
Sep 16 17:01:37.065193 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:37.065172    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq\" (UID: \"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq"
Sep 16 17:01:37.065261 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:37.065196    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kpz2\" (UniqueName: \"kubernetes.io/projected/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-kube-api-access-9kpz2\") pod \"custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq\" (UID: \"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq"
Sep 16 17:01:37.065261 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:37.065215    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq\" (UID: \"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq"
Sep 16 17:01:37.166555 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:37.166521    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq\" (UID: \"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq"
Sep 16 17:01:37.166839 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:37.166807    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq\" (UID: \"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq"
Sep 16 17:01:37.166967 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:37.166873    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq\" (UID: \"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq"
Sep 16 17:01:37.166967 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:37.166903    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kpz2\" (UniqueName: \"kubernetes.io/projected/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-kube-api-access-9kpz2\") pod \"custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq\" (UID: \"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq"
Sep 16 17:01:37.166967 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:37.166919    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq\" (UID: \"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq"
Sep 16 17:01:37.167139 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:37.166971    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq\" (UID: \"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq"
Sep 16 17:01:37.167139 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:37.167024    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq\" (UID: \"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq"
Sep 16 17:01:37.167139 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:37.167062    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq\" (UID: \"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq"
Sep 16 17:01:37.167139 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:37.167088    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq\" (UID: \"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq"
Sep 16 17:01:37.167442 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:37.167418    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq\" (UID: \"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq"
Sep 16 17:01:37.167603 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:37.167539    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq\" (UID: \"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq"
Sep 16 17:01:37.168028 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:37.168003    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq\" (UID: \"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq"
Sep 16 17:01:37.168167 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:37.168146    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq\" (UID: \"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq"
Sep 16 17:01:37.169644 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:37.169623    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq\" (UID: \"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq"
Sep 16 17:01:37.169774 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:37.169755    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq\" (UID: \"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq"
Sep 16 17:01:37.175821 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:37.175782    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kpz2\" (UniqueName: \"kubernetes.io/projected/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-kube-api-access-9kpz2\") pod \"custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq\" (UID: \"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq"
Sep 16 17:01:37.213612 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:37.213534    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:01:37.271601 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:37.271570    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq"
Sep 16 17:01:37.557165 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:37.557126    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq"]
Sep 16 17:01:37.562922 ip-10-0-136-228 kubenswrapper[2554]: W0916 17:01:37.562891    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d20b8e0_3a26_41d9_b09c_6e9567d1dc94.slice/crio-6962540ebabe30068884afe9592b9290c2100f04e30dd90e32650220e09b6d0c WatchSource:0}: Error finding container 6962540ebabe30068884afe9592b9290c2100f04e30dd90e32650220e09b6d0c: Status 404 returned error can't find the container with id 6962540ebabe30068884afe9592b9290c2100f04e30dd90e32650220e09b6d0c
Sep 16 17:01:38.153638 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:38.153601    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq" event={"ID":"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94","Type":"ContainerStarted","Data":"fb649e3b59fd0d275939800ae87d01994831917f79fafbc5673070f6d0dbce3b"}
Sep 16 17:01:38.154077 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:38.153659    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq" event={"ID":"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94","Type":"ContainerStarted","Data":"6962540ebabe30068884afe9592b9290c2100f04e30dd90e32650220e09b6d0c"}
Sep 16 17:01:40.163695 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:40.163657    2554 generic.go:356] "Generic (PLEG): container finished" podID="5d20b8e0-3a26-41d9-b09c-6e9567d1dc94" containerID="fb649e3b59fd0d275939800ae87d01994831917f79fafbc5673070f6d0dbce3b" exitCode=0
Sep 16 17:01:40.164202 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:40.163731    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq" event={"ID":"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94","Type":"ContainerDied","Data":"fb649e3b59fd0d275939800ae87d01994831917f79fafbc5673070f6d0dbce3b"}
Sep 16 17:01:41.177023 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:41.176986    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq" event={"ID":"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94","Type":"ContainerStarted","Data":"8a92467a915b963e369042cf723630dbfae57dc8a8b44ecc27812fefb650ee20"}
Sep 16 17:01:41.202333 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:41.202280    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq" podStartSLOduration=2.897900934 podStartE2EDuration="5.202266069s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.304365135s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:01:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:01:41.200156052 +0000 UTC m=+4301.644848263" watchObservedRunningTime="2026-09-16 17:01:41.202266069 +0000 UTC m=+4301.646958246"
Sep 16 17:01:45.186692 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:45.186639    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn" podUID="acacc924-9d50-4e79-89a9-aebfe4e2d050" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Sep 16 17:01:47.272255 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:47.272213    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq"
Sep 16 17:01:47.272822 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:47.272281    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq"
Sep 16 17:01:47.273807 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:47.273759    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq" podUID="5d20b8e0-3a26-41d9-b09c-6e9567d1dc94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 16 17:01:47.660552 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:47.660513    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:01:55.196315 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:55.196231    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn"
Sep 16 17:01:55.204220 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:55.204191    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn"
Sep 16 17:01:57.272591 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:01:57.272554    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq" podUID="5d20b8e0-3a26-41d9-b09c-6e9567d1dc94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 16 17:02:07.272015 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:07.271967    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq" podUID="5d20b8e0-3a26-41d9-b09c-6e9567d1dc94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 16 17:02:17.273131 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:17.273081    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq" podUID="5d20b8e0-3a26-41d9-b09c-6e9567d1dc94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 16 17:02:22.602025 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:22.601992    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-a290d4b0_lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn_acacc924-9d50-4e79-89a9-aebfe4e2d050/main/0.log"
Sep 16 17:02:23.656073 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:23.656039    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn"]
Sep 16 17:02:23.672452 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:23.672410    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:02:27.272156 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:27.272111    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq" podUID="5d20b8e0-3a26-41d9-b09c-6e9567d1dc94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 16 17:02:37.272568 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:37.272518    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq" podUID="5d20b8e0-3a26-41d9-b09c-6e9567d1dc94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 16 17:02:38.677703 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:38.677661    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn" podUID="acacc924-9d50-4e79-89a9-aebfe4e2d050" containerName="main" containerID="cri-o://ebcfde7d67560a36575d24193f1d05eafd48e5812d3f0aaeb04915b41554e64f" gracePeriod=45
Sep 16 17:02:39.196780 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:39.196749    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn" podUID="acacc924-9d50-4e79-89a9-aebfe4e2d050" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Sep 16 17:02:39.342960 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:39.342931    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn"
Sep 16 17:02:39.433367 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:39.433281    2554 generic.go:356] "Generic (PLEG): container finished" podID="acacc924-9d50-4e79-89a9-aebfe4e2d050" containerID="ebcfde7d67560a36575d24193f1d05eafd48e5812d3f0aaeb04915b41554e64f" exitCode=0
Sep 16 17:02:39.433367 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:39.433327    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn" event={"ID":"acacc924-9d50-4e79-89a9-aebfe4e2d050","Type":"ContainerDied","Data":"ebcfde7d67560a36575d24193f1d05eafd48e5812d3f0aaeb04915b41554e64f"}
Sep 16 17:02:39.433367 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:39.433342    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn"
Sep 16 17:02:39.433603 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:39.433378    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn" event={"ID":"acacc924-9d50-4e79-89a9-aebfe4e2d050","Type":"ContainerDied","Data":"a2b96f32bc6dc8fc78a33ac9092d06057ab8bfbd86aaf3505a572dd4044e1d78"}
Sep 16 17:02:39.433603 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:39.433413    2554 scope.go:117] "RemoveContainer" containerID="ebcfde7d67560a36575d24193f1d05eafd48e5812d3f0aaeb04915b41554e64f"
Sep 16 17:02:39.441726 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:39.441707    2554 scope.go:117] "RemoveContainer" containerID="ebcfde7d67560a36575d24193f1d05eafd48e5812d3f0aaeb04915b41554e64f"
Sep 16 17:02:39.442090 ip-10-0-136-228 kubenswrapper[2554]: E0916 17:02:39.442065    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebcfde7d67560a36575d24193f1d05eafd48e5812d3f0aaeb04915b41554e64f\": container with ID starting with ebcfde7d67560a36575d24193f1d05eafd48e5812d3f0aaeb04915b41554e64f not found: ID does not exist" containerID="ebcfde7d67560a36575d24193f1d05eafd48e5812d3f0aaeb04915b41554e64f"
Sep 16 17:02:39.442159 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:39.442104    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebcfde7d67560a36575d24193f1d05eafd48e5812d3f0aaeb04915b41554e64f"} err="failed to get container status \"ebcfde7d67560a36575d24193f1d05eafd48e5812d3f0aaeb04915b41554e64f\": rpc error: code = NotFound desc = could not find container \"ebcfde7d67560a36575d24193f1d05eafd48e5812d3f0aaeb04915b41554e64f\": container with ID starting with ebcfde7d67560a36575d24193f1d05eafd48e5812d3f0aaeb04915b41554e64f not found: ID does not exist"
Sep 16 17:02:39.453367 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:39.453346    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acacc924-9d50-4e79-89a9-aebfe4e2d050-model-cache\") pod \"acacc924-9d50-4e79-89a9-aebfe4e2d050\" (UID: \"acacc924-9d50-4e79-89a9-aebfe4e2d050\") "
Sep 16 17:02:39.453458 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:39.453381    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fv8t5\" (UniqueName: \"kubernetes.io/projected/acacc924-9d50-4e79-89a9-aebfe4e2d050-kube-api-access-fv8t5\") pod \"acacc924-9d50-4e79-89a9-aebfe4e2d050\" (UID: \"acacc924-9d50-4e79-89a9-aebfe4e2d050\") "
Sep 16 17:02:39.453458 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:39.453406    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acacc924-9d50-4e79-89a9-aebfe4e2d050-tmp-dir\") pod \"acacc924-9d50-4e79-89a9-aebfe4e2d050\" (UID: \"acacc924-9d50-4e79-89a9-aebfe4e2d050\") "
Sep 16 17:02:39.453458 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:39.453432    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acacc924-9d50-4e79-89a9-aebfe4e2d050-dshm\") pod \"acacc924-9d50-4e79-89a9-aebfe4e2d050\" (UID: \"acacc924-9d50-4e79-89a9-aebfe4e2d050\") "
Sep 16 17:02:39.453614 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:39.453468    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acacc924-9d50-4e79-89a9-aebfe4e2d050-home\") pod \"acacc924-9d50-4e79-89a9-aebfe4e2d050\" (UID: \"acacc924-9d50-4e79-89a9-aebfe4e2d050\") "
Sep 16 17:02:39.453614 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:39.453502    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acacc924-9d50-4e79-89a9-aebfe4e2d050-tls-certs\") pod \"acacc924-9d50-4e79-89a9-aebfe4e2d050\" (UID: \"acacc924-9d50-4e79-89a9-aebfe4e2d050\") "
Sep 16 17:02:39.453614 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:39.453608    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0798b6fdf777dd2e5\") pod \"acacc924-9d50-4e79-89a9-aebfe4e2d050\" (UID: \"acacc924-9d50-4e79-89a9-aebfe4e2d050\") "
Sep 16 17:02:39.453771 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:39.453616    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acacc924-9d50-4e79-89a9-aebfe4e2d050-model-cache" (OuterVolumeSpecName: "model-cache") pod "acacc924-9d50-4e79-89a9-aebfe4e2d050" (UID: "acacc924-9d50-4e79-89a9-aebfe4e2d050"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:02:39.453852 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:39.453818    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acacc924-9d50-4e79-89a9-aebfe4e2d050-model-cache\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:02:39.454115 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:39.454079    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acacc924-9d50-4e79-89a9-aebfe4e2d050-home" (OuterVolumeSpecName: "home") pod "acacc924-9d50-4e79-89a9-aebfe4e2d050" (UID: "acacc924-9d50-4e79-89a9-aebfe4e2d050"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:02:39.456073 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:39.456031    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acacc924-9d50-4e79-89a9-aebfe4e2d050-kube-api-access-fv8t5" (OuterVolumeSpecName: "kube-api-access-fv8t5") pod "acacc924-9d50-4e79-89a9-aebfe4e2d050" (UID: "acacc924-9d50-4e79-89a9-aebfe4e2d050"). InnerVolumeSpecName "kube-api-access-fv8t5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:02:39.456073 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:39.456035    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acacc924-9d50-4e79-89a9-aebfe4e2d050-dshm" (OuterVolumeSpecName: "dshm") pod "acacc924-9d50-4e79-89a9-aebfe4e2d050" (UID: "acacc924-9d50-4e79-89a9-aebfe4e2d050"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:02:39.456248 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:39.456135    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acacc924-9d50-4e79-89a9-aebfe4e2d050-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "acacc924-9d50-4e79-89a9-aebfe4e2d050" (UID: "acacc924-9d50-4e79-89a9-aebfe4e2d050"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:02:39.462890 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:39.462865    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0798b6fdf777dd2e5" (OuterVolumeSpecName: "kserve-pvc-source") pod "acacc924-9d50-4e79-89a9-aebfe4e2d050" (UID: "acacc924-9d50-4e79-89a9-aebfe4e2d050"). InnerVolumeSpecName "pvc-f6a115bc-fe2c-4d81-8231-1cd9cb25028e". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 16 17:02:39.469329 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:39.469302    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acacc924-9d50-4e79-89a9-aebfe4e2d050-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "acacc924-9d50-4e79-89a9-aebfe4e2d050" (UID: "acacc924-9d50-4e79-89a9-aebfe4e2d050"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:02:39.554123 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:39.554089    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv8t5\" (UniqueName: \"kubernetes.io/projected/acacc924-9d50-4e79-89a9-aebfe4e2d050-kube-api-access-fv8t5\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:02:39.554123 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:39.554116    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acacc924-9d50-4e79-89a9-aebfe4e2d050-tmp-dir\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:02:39.554123 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:39.554127    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acacc924-9d50-4e79-89a9-aebfe4e2d050-dshm\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:02:39.554356 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:39.554135    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acacc924-9d50-4e79-89a9-aebfe4e2d050-home\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:02:39.554356 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:39.554143    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acacc924-9d50-4e79-89a9-aebfe4e2d050-tls-certs\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:02:39.554356 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:39.554172    2554 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-f6a115bc-fe2c-4d81-8231-1cd9cb25028e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0798b6fdf777dd2e5\") on node \"ip-10-0-136-228.ec2.internal\" "
Sep 16 17:02:39.583249 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:39.583224    2554 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-f6a115bc-fe2c-4d81-8231-1cd9cb25028e" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0798b6fdf777dd2e5") on node "ip-10-0-136-228.ec2.internal"
Sep 16 17:02:39.654522 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:39.654485    2554 reconciler_common.go:299] "Volume detached for volume \"pvc-f6a115bc-fe2c-4d81-8231-1cd9cb25028e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0798b6fdf777dd2e5\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"csi-a525d0065135705af2c00989850bba3017699585805c261d976bce53b10f04b5\""
Sep 16 17:02:39.758576 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:39.758544    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn"]
Sep 16 17:02:39.762630 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:39.762608    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-n9rtn"]
Sep 16 17:02:40.076342 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:40.076312    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acacc924-9d50-4e79-89a9-aebfe4e2d050" path="/var/lib/kubelet/pods/acacc924-9d50-4e79-89a9-aebfe4e2d050/volumes"
Sep 16 17:02:43.899017 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:43.898953    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:02:43.996440 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:43.996408    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:02:44.096434 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:44.096387    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:02:47.273019 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:47.272971    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq" podUID="5d20b8e0-3a26-41d9-b09c-6e9567d1dc94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 16 17:02:49.554610 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:49.554565    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:02:49.612105 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:49.612057    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:02:50.186345 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:50.186281    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:02:56.046584 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:56.046543    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:02:56.146925 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:56.146879    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:02:56.727305 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:56.727265    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:02:57.272177 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:57.272139    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq" podUID="5d20b8e0-3a26-41d9-b09c-6e9567d1dc94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 16 17:02:57.818738 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:02:57.818673    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:03:01.164358 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:01.164320    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr"]
Sep 16 17:03:01.164773 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:01.164752    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acacc924-9d50-4e79-89a9-aebfe4e2d050" containerName="main"
Sep 16 17:03:01.164840 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:01.164773    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="acacc924-9d50-4e79-89a9-aebfe4e2d050" containerName="main"
Sep 16 17:03:01.164924 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:01.164893    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="acacc924-9d50-4e79-89a9-aebfe4e2d050" containerName="main"
Sep 16 17:03:01.167506 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:01.167484    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr"
Sep 16 17:03:01.170439 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:01.170405    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"seaweedfs-s3-creds\""
Sep 16 17:03:01.170573 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:01.170415    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-vzmf9\""
Sep 16 17:03:01.170754 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:01.170737    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 17:03:01.170860 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:01.170830    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 16 17:03:01.170941 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:01.170921    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 16 17:03:01.171010 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:01.170736    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 16 17:03:01.178376 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:01.178355    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr"]
Sep 16 17:03:01.200712 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:01.200680    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:03:01.223002 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:01.222982    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-tls-certs\") pod \"tls-verification-test-kserve-5bc4ff64-mnpsr\" (UID: \"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr"
Sep 16 17:03:01.223100 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:01.223020    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-home\") pod \"tls-verification-test-kserve-5bc4ff64-mnpsr\" (UID: \"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr"
Sep 16 17:03:01.223100 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:01.223044    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-model-cache\") pod \"tls-verification-test-kserve-5bc4ff64-mnpsr\" (UID: \"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr"
Sep 16 17:03:01.223100 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:01.223063    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-tmp-dir\") pod \"tls-verification-test-kserve-5bc4ff64-mnpsr\" (UID: \"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr"
Sep 16 17:03:01.223100 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:01.223092    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bnmq\" (UniqueName: \"kubernetes.io/projected/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-kube-api-access-6bnmq\") pod \"tls-verification-test-kserve-5bc4ff64-mnpsr\" (UID: \"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr"
Sep 16 17:03:01.223263 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:01.223117    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-cabundle-cert\") pod \"tls-verification-test-kserve-5bc4ff64-mnpsr\" (UID: \"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr"
Sep 16 17:03:01.223263 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:01.223132    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-kserve-provision-location\") pod \"tls-verification-test-kserve-5bc4ff64-mnpsr\" (UID: \"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr"
Sep 16 17:03:01.223263 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:01.223152    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-dshm\") pod \"tls-verification-test-kserve-5bc4ff64-mnpsr\" (UID: \"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr"
Sep 16 17:03:01.324057 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:01.324025    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-kserve-provision-location\") pod \"tls-verification-test-kserve-5bc4ff64-mnpsr\" (UID: \"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr"
Sep 16 17:03:01.324057 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:01.324062    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-dshm\") pod \"tls-verification-test-kserve-5bc4ff64-mnpsr\" (UID: \"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr"
Sep 16 17:03:01.324312 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:01.324096    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-tls-certs\") pod \"tls-verification-test-kserve-5bc4ff64-mnpsr\" (UID: \"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr"
Sep 16 17:03:01.324312 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:01.324126    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-home\") pod \"tls-verification-test-kserve-5bc4ff64-mnpsr\" (UID: \"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr"
Sep 16 17:03:01.324312 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:01.324263    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-model-cache\") pod \"tls-verification-test-kserve-5bc4ff64-mnpsr\" (UID: \"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr"
Sep 16 17:03:01.324499 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:01.324484    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-home\") pod \"tls-verification-test-kserve-5bc4ff64-mnpsr\" (UID: \"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr"
Sep 16 17:03:01.324572 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:01.324549    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-kserve-provision-location\") pod \"tls-verification-test-kserve-5bc4ff64-mnpsr\" (UID: \"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr"
Sep 16 17:03:01.324635 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:01.324613    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-tmp-dir\") pod \"tls-verification-test-kserve-5bc4ff64-mnpsr\" (UID: \"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr"
Sep 16 17:03:01.324694 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:01.324657    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bnmq\" (UniqueName: \"kubernetes.io/projected/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-kube-api-access-6bnmq\") pod \"tls-verification-test-kserve-5bc4ff64-mnpsr\" (UID: \"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr"
Sep 16 17:03:01.324694 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:01.324679    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-cabundle-cert\") pod \"tls-verification-test-kserve-5bc4ff64-mnpsr\" (UID: \"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr"
Sep 16 17:03:01.324824 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:01.324620    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-model-cache\") pod \"tls-verification-test-kserve-5bc4ff64-mnpsr\" (UID: \"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr"
Sep 16 17:03:01.324891 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:01.324853    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-tmp-dir\") pod \"tls-verification-test-kserve-5bc4ff64-mnpsr\" (UID: \"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr"
Sep 16 17:03:01.325276 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:01.325259    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-cabundle-cert\") pod \"tls-verification-test-kserve-5bc4ff64-mnpsr\" (UID: \"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr"
Sep 16 17:03:01.327008 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:01.326983    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-dshm\") pod \"tls-verification-test-kserve-5bc4ff64-mnpsr\" (UID: \"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr"
Sep 16 17:03:01.327208 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:01.327190    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-tls-certs\") pod \"tls-verification-test-kserve-5bc4ff64-mnpsr\" (UID: \"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr"
Sep 16 17:03:01.333847 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:01.333823    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bnmq\" (UniqueName: \"kubernetes.io/projected/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-kube-api-access-6bnmq\") pod \"tls-verification-test-kserve-5bc4ff64-mnpsr\" (UID: \"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr"
Sep 16 17:03:01.451590 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:01.451515    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:03:01.480971 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:01.480930    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr"
Sep 16 17:03:01.756914 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:01.756888    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr"]
Sep 16 17:03:01.758976 ip-10-0-136-228 kubenswrapper[2554]: W0916 17:03:01.758935    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1972bb3c_af5c_4913_8e0e_eadd9ddceb1e.slice/crio-c273d8ea576ee514e03afd0a73757ba7bb987a19d05585a5cd6ee4019e6c06b4 WatchSource:0}: Error finding container c273d8ea576ee514e03afd0a73757ba7bb987a19d05585a5cd6ee4019e6c06b4: Status 404 returned error can't find the container with id c273d8ea576ee514e03afd0a73757ba7bb987a19d05585a5cd6ee4019e6c06b4
Sep 16 17:03:02.529488 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:02.529444    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr" event={"ID":"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e","Type":"ContainerStarted","Data":"0afc608aa690204c5f0d2501b979832053aaddb87669f02122b4a9c0e9a62b0e"}
Sep 16 17:03:02.529913 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:02.529517    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr" event={"ID":"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e","Type":"ContainerStarted","Data":"c273d8ea576ee514e03afd0a73757ba7bb987a19d05585a5cd6ee4019e6c06b4"}
Sep 16 17:03:04.539431 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:04.539401    2554 generic.go:356] "Generic (PLEG): container finished" podID="1972bb3c-af5c-4913-8e0e-eadd9ddceb1e" containerID="0afc608aa690204c5f0d2501b979832053aaddb87669f02122b4a9c0e9a62b0e" exitCode=0
Sep 16 17:03:04.539832 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:04.539458    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr" event={"ID":"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e","Type":"ContainerDied","Data":"0afc608aa690204c5f0d2501b979832053aaddb87669f02122b4a9c0e9a62b0e"}
Sep 16 17:03:05.546660 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:05.546619    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr" event={"ID":"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e","Type":"ContainerStarted","Data":"90bf5f131d5896829cc3530fa384cc94b2957cfe7c0b96213266f4b984bdb6bc"}
Sep 16 17:03:05.570426 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:05.570361    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr" podStartSLOduration=2.274264105 podStartE2EDuration="4.570337342s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.296073237s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:03:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:03:05.568321467 +0000 UTC m=+4386.013013656" watchObservedRunningTime="2026-09-16 17:03:05.570337342 +0000 UTC m=+4386.015029515"
Sep 16 17:03:06.481969 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:06.481930    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr"
Sep 16 17:03:07.273007 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:07.272963    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq" podUID="5d20b8e0-3a26-41d9-b09c-6e9567d1dc94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 16 17:03:08.264647 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:08.264614    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:03:11.481634 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:11.481583    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr"
Sep 16 17:03:11.482900 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:11.482871    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr" podUID="1972bb3c-af5c-4913-8e0e-eadd9ddceb1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 16 17:03:14.827290 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:14.827249    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:03:17.272329 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:17.272286    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq" podUID="5d20b8e0-3a26-41d9-b09c-6e9567d1dc94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 16 17:03:21.482305 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:21.482207    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr" podUID="1972bb3c-af5c-4913-8e0e-eadd9ddceb1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 16 17:03:26.432064 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:26.431970    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:03:26.584461 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:26.584419    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:03:26.684202 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:26.684112    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:03:27.272606 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:27.272565    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq" podUID="5d20b8e0-3a26-41d9-b09c-6e9567d1dc94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 16 17:03:31.482082 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:31.482043    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr" podUID="1972bb3c-af5c-4913-8e0e-eadd9ddceb1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 16 17:03:37.282984 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:37.282951    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq"
Sep 16 17:03:37.291634 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:37.291608    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq"
Sep 16 17:03:41.481392 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:41.481341    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr" podUID="1972bb3c-af5c-4913-8e0e-eadd9ddceb1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 16 17:03:46.221543 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:46.221503    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:03:46.236452 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:46.236418    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq"]
Sep 16 17:03:46.319385 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:46.319351    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:03:51.481740 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:03:51.481698    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr" podUID="1972bb3c-af5c-4913-8e0e-eadd9ddceb1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 16 17:04:01.258815 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:01.258727    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq" podUID="5d20b8e0-3a26-41d9-b09c-6e9567d1dc94" containerName="main" containerID="cri-o://8a92467a915b963e369042cf723630dbfae57dc8a8b44ecc27812fefb650ee20" gracePeriod=45
Sep 16 17:04:01.481734 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:01.481687    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr" podUID="1972bb3c-af5c-4913-8e0e-eadd9ddceb1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 16 17:04:02.010919 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:02.010889    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq"
Sep 16 17:04:02.137936 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:02.137895    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-home\") pod \"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94\" (UID: \"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94\") "
Sep 16 17:04:02.138117 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:02.137986    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kpz2\" (UniqueName: \"kubernetes.io/projected/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-kube-api-access-9kpz2\") pod \"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94\" (UID: \"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94\") "
Sep 16 17:04:02.138117 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:02.138017    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-model-cache\") pod \"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94\" (UID: \"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94\") "
Sep 16 17:04:02.138117 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:02.138066    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-tls-certs\") pod \"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94\" (UID: \"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94\") "
Sep 16 17:04:02.138117 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:02.138090    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-dshm\") pod \"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94\" (UID: \"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94\") "
Sep 16 17:04:02.138339 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:02.138121    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-tmp-dir\") pod \"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94\" (UID: \"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94\") "
Sep 16 17:04:02.138339 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:02.138144    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-kserve-provision-location\") pod \"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94\" (UID: \"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94\") "
Sep 16 17:04:02.138339 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:02.138178    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-cabundle-cert\") pod \"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94\" (UID: \"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94\") "
Sep 16 17:04:02.138339 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:02.138296    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-model-cache" (OuterVolumeSpecName: "model-cache") pod "5d20b8e0-3a26-41d9-b09c-6e9567d1dc94" (UID: "5d20b8e0-3a26-41d9-b09c-6e9567d1dc94"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:04:02.138564 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:02.138476    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-model-cache\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:04:02.138623 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:02.138577    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-home" (OuterVolumeSpecName: "home") pod "5d20b8e0-3a26-41d9-b09c-6e9567d1dc94" (UID: "5d20b8e0-3a26-41d9-b09c-6e9567d1dc94"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:04:02.138912 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:02.138887    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5d20b8e0-3a26-41d9-b09c-6e9567d1dc94" (UID: "5d20b8e0-3a26-41d9-b09c-6e9567d1dc94"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:04:02.141085 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:02.141053    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-dshm" (OuterVolumeSpecName: "dshm") pod "5d20b8e0-3a26-41d9-b09c-6e9567d1dc94" (UID: "5d20b8e0-3a26-41d9-b09c-6e9567d1dc94"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:04:02.141197 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:02.141150    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5d20b8e0-3a26-41d9-b09c-6e9567d1dc94" (UID: "5d20b8e0-3a26-41d9-b09c-6e9567d1dc94"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:04:02.141264 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:02.141199    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-kube-api-access-9kpz2" (OuterVolumeSpecName: "kube-api-access-9kpz2") pod "5d20b8e0-3a26-41d9-b09c-6e9567d1dc94" (UID: "5d20b8e0-3a26-41d9-b09c-6e9567d1dc94"). InnerVolumeSpecName "kube-api-access-9kpz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:04:02.150953 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:02.150925    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5d20b8e0-3a26-41d9-b09c-6e9567d1dc94" (UID: "5d20b8e0-3a26-41d9-b09c-6e9567d1dc94"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:04:02.158472 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:02.158444    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5d20b8e0-3a26-41d9-b09c-6e9567d1dc94" (UID: "5d20b8e0-3a26-41d9-b09c-6e9567d1dc94"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:04:02.239725 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:02.239686    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:04:02.239725 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:02.239720    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-home\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:04:02.239725 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:02.239729    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kpz2\" (UniqueName: \"kubernetes.io/projected/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-kube-api-access-9kpz2\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:04:02.239974 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:02.239739    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-tls-certs\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:04:02.239974 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:02.239748    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-dshm\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:04:02.239974 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:02.239758    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-tmp-dir\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:04:02.239974 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:02.239765    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:04:02.792980 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:02.792942    2554 generic.go:356] "Generic (PLEG): container finished" podID="5d20b8e0-3a26-41d9-b09c-6e9567d1dc94" containerID="8a92467a915b963e369042cf723630dbfae57dc8a8b44ecc27812fefb650ee20" exitCode=0
Sep 16 17:04:02.793389 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:02.793004    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq"
Sep 16 17:04:02.793389 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:02.793031    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq" event={"ID":"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94","Type":"ContainerDied","Data":"8a92467a915b963e369042cf723630dbfae57dc8a8b44ecc27812fefb650ee20"}
Sep 16 17:04:02.793389 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:02.793089    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq" event={"ID":"5d20b8e0-3a26-41d9-b09c-6e9567d1dc94","Type":"ContainerDied","Data":"6962540ebabe30068884afe9592b9290c2100f04e30dd90e32650220e09b6d0c"}
Sep 16 17:04:02.793389 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:02.793113    2554 scope.go:117] "RemoveContainer" containerID="8a92467a915b963e369042cf723630dbfae57dc8a8b44ecc27812fefb650ee20"
Sep 16 17:04:02.802121 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:02.802102    2554 scope.go:117] "RemoveContainer" containerID="fb649e3b59fd0d275939800ae87d01994831917f79fafbc5673070f6d0dbce3b"
Sep 16 17:04:02.809724 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:02.809704    2554 scope.go:117] "RemoveContainer" containerID="8a92467a915b963e369042cf723630dbfae57dc8a8b44ecc27812fefb650ee20"
Sep 16 17:04:02.810039 ip-10-0-136-228 kubenswrapper[2554]: E0916 17:04:02.810019    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a92467a915b963e369042cf723630dbfae57dc8a8b44ecc27812fefb650ee20\": container with ID starting with 8a92467a915b963e369042cf723630dbfae57dc8a8b44ecc27812fefb650ee20 not found: ID does not exist" containerID="8a92467a915b963e369042cf723630dbfae57dc8a8b44ecc27812fefb650ee20"
Sep 16 17:04:02.810102 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:02.810050    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a92467a915b963e369042cf723630dbfae57dc8a8b44ecc27812fefb650ee20"} err="failed to get container status \"8a92467a915b963e369042cf723630dbfae57dc8a8b44ecc27812fefb650ee20\": rpc error: code = NotFound desc = could not find container \"8a92467a915b963e369042cf723630dbfae57dc8a8b44ecc27812fefb650ee20\": container with ID starting with 8a92467a915b963e369042cf723630dbfae57dc8a8b44ecc27812fefb650ee20 not found: ID does not exist"
Sep 16 17:04:02.810102 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:02.810068    2554 scope.go:117] "RemoveContainer" containerID="fb649e3b59fd0d275939800ae87d01994831917f79fafbc5673070f6d0dbce3b"
Sep 16 17:04:02.810384 ip-10-0-136-228 kubenswrapper[2554]: E0916 17:04:02.810365    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb649e3b59fd0d275939800ae87d01994831917f79fafbc5673070f6d0dbce3b\": container with ID starting with fb649e3b59fd0d275939800ae87d01994831917f79fafbc5673070f6d0dbce3b not found: ID does not exist" containerID="fb649e3b59fd0d275939800ae87d01994831917f79fafbc5673070f6d0dbce3b"
Sep 16 17:04:02.810441 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:02.810393    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb649e3b59fd0d275939800ae87d01994831917f79fafbc5673070f6d0dbce3b"} err="failed to get container status \"fb649e3b59fd0d275939800ae87d01994831917f79fafbc5673070f6d0dbce3b\": rpc error: code = NotFound desc = could not find container \"fb649e3b59fd0d275939800ae87d01994831917f79fafbc5673070f6d0dbce3b\": container with ID starting with fb649e3b59fd0d275939800ae87d01994831917f79fafbc5673070f6d0dbce3b not found: ID does not exist"
Sep 16 17:04:02.814989 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:02.814966    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq"]
Sep 16 17:04:02.820321 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:02.820298    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-86995c845c-kskhq"]
Sep 16 17:04:04.076607 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:04.076571    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d20b8e0-3a26-41d9-b09c-6e9567d1dc94" path="/var/lib/kubelet/pods/5d20b8e0-3a26-41d9-b09c-6e9567d1dc94/volumes"
Sep 16 17:04:11.482136 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:11.482092    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr" podUID="1972bb3c-af5c-4913-8e0e-eadd9ddceb1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 16 17:04:21.481310 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:21.481217    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr" podUID="1972bb3c-af5c-4913-8e0e-eadd9ddceb1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 16 17:04:23.997298 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:23.997262    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:04:31.481435 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:31.481388    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr" podUID="1972bb3c-af5c-4913-8e0e-eadd9ddceb1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 16 17:04:35.182909 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:35.182878    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:04:36.325558 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:36.325521    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:04:36.427522 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:36.427463    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:04:36.589448 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:36.589360    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:04:36.689015 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:36.688981    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:04:39.703982 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:39.703952    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:04:41.481523 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:41.481481    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr" podUID="1972bb3c-af5c-4913-8e0e-eadd9ddceb1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 16 17:04:43.244941 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:43.244908    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:04:46.117910 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.117875    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj"]
Sep 16 17:04:46.118317 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.118124    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d20b8e0-3a26-41d9-b09c-6e9567d1dc94" containerName="main"
Sep 16 17:04:46.118317 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.118135    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d20b8e0-3a26-41d9-b09c-6e9567d1dc94" containerName="main"
Sep 16 17:04:46.118317 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.118153    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d20b8e0-3a26-41d9-b09c-6e9567d1dc94" containerName="storage-initializer"
Sep 16 17:04:46.118317 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.118158    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d20b8e0-3a26-41d9-b09c-6e9567d1dc94" containerName="storage-initializer"
Sep 16 17:04:46.118317 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.118203    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d20b8e0-3a26-41d9-b09c-6e9567d1dc94" containerName="main"
Sep 16 17:04:46.120278 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.120258    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj"
Sep 16 17:04:46.123148 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.123091    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 17:04:46.123361 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.123330    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 16 17:04:46.123474 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.123450    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-qr8qg\""
Sep 16 17:04:46.123474 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.123462    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-dockercfg-xf8dg\""
Sep 16 17:04:46.123585 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.123489    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 16 17:04:46.124726 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.124695    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 16 17:04:46.124842 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.124758    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 16 17:04:46.137371 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.137343    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj"]
Sep 16 17:04:46.173571 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.173532    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/904ae881-c154-4898-a1c9-12f9284a20a4-tmp-dir\") pod \"router-with-refs-pd-test-kserve-5b6456f669-k4rwj\" (UID: \"904ae881-c154-4898-a1c9-12f9284a20a4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj"
Sep 16 17:04:46.173571 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.173573    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/904ae881-c154-4898-a1c9-12f9284a20a4-home\") pod \"router-with-refs-pd-test-kserve-5b6456f669-k4rwj\" (UID: \"904ae881-c154-4898-a1c9-12f9284a20a4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj"
Sep 16 17:04:46.173862 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.173599    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/904ae881-c154-4898-a1c9-12f9284a20a4-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-5b6456f669-k4rwj\" (UID: \"904ae881-c154-4898-a1c9-12f9284a20a4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj"
Sep 16 17:04:46.173862 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.173642    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/904ae881-c154-4898-a1c9-12f9284a20a4-model-cache\") pod \"router-with-refs-pd-test-kserve-5b6456f669-k4rwj\" (UID: \"904ae881-c154-4898-a1c9-12f9284a20a4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj"
Sep 16 17:04:46.173862 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.173693    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jn59\" (UniqueName: \"kubernetes.io/projected/904ae881-c154-4898-a1c9-12f9284a20a4-kube-api-access-4jn59\") pod \"router-with-refs-pd-test-kserve-5b6456f669-k4rwj\" (UID: \"904ae881-c154-4898-a1c9-12f9284a20a4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj"
Sep 16 17:04:46.173862 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.173718    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/904ae881-c154-4898-a1c9-12f9284a20a4-dshm\") pod \"router-with-refs-pd-test-kserve-5b6456f669-k4rwj\" (UID: \"904ae881-c154-4898-a1c9-12f9284a20a4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj"
Sep 16 17:04:46.173862 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.173741    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/904ae881-c154-4898-a1c9-12f9284a20a4-tls-certs\") pod \"router-with-refs-pd-test-kserve-5b6456f669-k4rwj\" (UID: \"904ae881-c154-4898-a1c9-12f9284a20a4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj"
Sep 16 17:04:46.173862 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.173772    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/904ae881-c154-4898-a1c9-12f9284a20a4-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-5b6456f669-k4rwj\" (UID: \"904ae881-c154-4898-a1c9-12f9284a20a4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj"
Sep 16 17:04:46.253425 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.253388    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:04:46.274130 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.274098    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jn59\" (UniqueName: \"kubernetes.io/projected/904ae881-c154-4898-a1c9-12f9284a20a4-kube-api-access-4jn59\") pod \"router-with-refs-pd-test-kserve-5b6456f669-k4rwj\" (UID: \"904ae881-c154-4898-a1c9-12f9284a20a4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj"
Sep 16 17:04:46.274255 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.274135    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/904ae881-c154-4898-a1c9-12f9284a20a4-dshm\") pod \"router-with-refs-pd-test-kserve-5b6456f669-k4rwj\" (UID: \"904ae881-c154-4898-a1c9-12f9284a20a4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj"
Sep 16 17:04:46.274255 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.274155    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/904ae881-c154-4898-a1c9-12f9284a20a4-tls-certs\") pod \"router-with-refs-pd-test-kserve-5b6456f669-k4rwj\" (UID: \"904ae881-c154-4898-a1c9-12f9284a20a4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj"
Sep 16 17:04:46.274255 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.274177    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/904ae881-c154-4898-a1c9-12f9284a20a4-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-5b6456f669-k4rwj\" (UID: \"904ae881-c154-4898-a1c9-12f9284a20a4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj"
Sep 16 17:04:46.274255 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.274205    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/904ae881-c154-4898-a1c9-12f9284a20a4-tmp-dir\") pod \"router-with-refs-pd-test-kserve-5b6456f669-k4rwj\" (UID: \"904ae881-c154-4898-a1c9-12f9284a20a4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj"
Sep 16 17:04:46.274255 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.274246    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/904ae881-c154-4898-a1c9-12f9284a20a4-home\") pod \"router-with-refs-pd-test-kserve-5b6456f669-k4rwj\" (UID: \"904ae881-c154-4898-a1c9-12f9284a20a4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj"
Sep 16 17:04:46.274522 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.274286    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/904ae881-c154-4898-a1c9-12f9284a20a4-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-5b6456f669-k4rwj\" (UID: \"904ae881-c154-4898-a1c9-12f9284a20a4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj"
Sep 16 17:04:46.274522 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.274314    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/904ae881-c154-4898-a1c9-12f9284a20a4-model-cache\") pod \"router-with-refs-pd-test-kserve-5b6456f669-k4rwj\" (UID: \"904ae881-c154-4898-a1c9-12f9284a20a4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj"
Sep 16 17:04:46.274732 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.274706    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/904ae881-c154-4898-a1c9-12f9284a20a4-model-cache\") pod \"router-with-refs-pd-test-kserve-5b6456f669-k4rwj\" (UID: \"904ae881-c154-4898-a1c9-12f9284a20a4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj"
Sep 16 17:04:46.274847 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.274733    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/904ae881-c154-4898-a1c9-12f9284a20a4-tmp-dir\") pod \"router-with-refs-pd-test-kserve-5b6456f669-k4rwj\" (UID: \"904ae881-c154-4898-a1c9-12f9284a20a4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj"
Sep 16 17:04:46.274984 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.274960    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/904ae881-c154-4898-a1c9-12f9284a20a4-home\") pod \"router-with-refs-pd-test-kserve-5b6456f669-k4rwj\" (UID: \"904ae881-c154-4898-a1c9-12f9284a20a4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj"
Sep 16 17:04:46.275057 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.274987    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/904ae881-c154-4898-a1c9-12f9284a20a4-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-5b6456f669-k4rwj\" (UID: \"904ae881-c154-4898-a1c9-12f9284a20a4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj"
Sep 16 17:04:46.275160 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.275138    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/904ae881-c154-4898-a1c9-12f9284a20a4-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-5b6456f669-k4rwj\" (UID: \"904ae881-c154-4898-a1c9-12f9284a20a4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj"
Sep 16 17:04:46.276533 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.276517    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/904ae881-c154-4898-a1c9-12f9284a20a4-dshm\") pod \"router-with-refs-pd-test-kserve-5b6456f669-k4rwj\" (UID: \"904ae881-c154-4898-a1c9-12f9284a20a4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj"
Sep 16 17:04:46.276948 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.276934    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/904ae881-c154-4898-a1c9-12f9284a20a4-tls-certs\") pod \"router-with-refs-pd-test-kserve-5b6456f669-k4rwj\" (UID: \"904ae881-c154-4898-a1c9-12f9284a20a4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj"
Sep 16 17:04:46.282229 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.282192    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jn59\" (UniqueName: \"kubernetes.io/projected/904ae881-c154-4898-a1c9-12f9284a20a4-kube-api-access-4jn59\") pod \"router-with-refs-pd-test-kserve-5b6456f669-k4rwj\" (UID: \"904ae881-c154-4898-a1c9-12f9284a20a4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj"
Sep 16 17:04:46.431384 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.431283    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj"
Sep 16 17:04:46.668565 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.668526    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:04:46.712711 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.712681    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj"]
Sep 16 17:04:46.714849 ip-10-0-136-228 kubenswrapper[2554]: W0916 17:04:46.714823    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod904ae881_c154_4898_a1c9_12f9284a20a4.slice/crio-bd19028a90cf153782def33d9edbc53b12c88f4906784c9c81a481e20cdcd062 WatchSource:0}: Error finding container bd19028a90cf153782def33d9edbc53b12c88f4906784c9c81a481e20cdcd062: Status 404 returned error can't find the container with id bd19028a90cf153782def33d9edbc53b12c88f4906784c9c81a481e20cdcd062
Sep 16 17:04:46.742912 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.742879    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:04:46.971030 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.970944    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj" event={"ID":"904ae881-c154-4898-a1c9-12f9284a20a4","Type":"ContainerStarted","Data":"55558b1365fd7bfbb252dda4a3203e5dd29c3f918ccee5ad8cd4ca4575c9b894"}
Sep 16 17:04:46.971030 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.970996    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj" event={"ID":"904ae881-c154-4898-a1c9-12f9284a20a4","Type":"ContainerStarted","Data":"bd19028a90cf153782def33d9edbc53b12c88f4906784c9c81a481e20cdcd062"}
Sep 16 17:04:46.971226 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:46.971098    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj"
Sep 16 17:04:47.977535 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:47.977502    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj" event={"ID":"904ae881-c154-4898-a1c9-12f9284a20a4","Type":"ContainerStarted","Data":"83b0867ca9bcbe95d8a238589ec72665e7271d85c5ba7e6b895f233c247551d1"}
Sep 16 17:04:49.314424 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:49.314387    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:04:49.414205 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:49.414171    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:04:49.993902 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:49.993854    2554 generic.go:356] "Generic (PLEG): container finished" podID="904ae881-c154-4898-a1c9-12f9284a20a4" containerID="83b0867ca9bcbe95d8a238589ec72665e7271d85c5ba7e6b895f233c247551d1" exitCode=0
Sep 16 17:04:49.994071 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:49.993931    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj" event={"ID":"904ae881-c154-4898-a1c9-12f9284a20a4","Type":"ContainerDied","Data":"83b0867ca9bcbe95d8a238589ec72665e7271d85c5ba7e6b895f233c247551d1"}
Sep 16 17:04:51.000960 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:51.000923    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj" event={"ID":"904ae881-c154-4898-a1c9-12f9284a20a4","Type":"ContainerStarted","Data":"81fcf56ac24b65b72acd27ef7a068f53a408f8bc84aef8bd94dded09d84a8212"}
Sep 16 17:04:51.027448 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:51.027393    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj" podStartSLOduration=2.7121320840000003 podStartE2EDuration="5.027375893s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.315243809s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:04:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:04:51.024567619 +0000 UTC m=+4491.469259821" watchObservedRunningTime="2026-09-16 17:04:51.027375893 +0000 UTC m=+4491.472068072"
Sep 16 17:04:51.481416 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:51.481354    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr" podUID="1972bb3c-af5c-4913-8e0e-eadd9ddceb1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 16 17:04:56.431890 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:56.431844    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj"
Sep 16 17:04:56.432509 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:56.432006    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj"
Sep 16 17:04:56.432945 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:56.432910    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj" podUID="904ae881-c154-4898-a1c9-12f9284a20a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8001/health\": dial tcp 10.132.0.47:8001: connect: connection refused"
Sep 16 17:04:56.451344 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:04:56.451307    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj"
Sep 16 17:05:01.491551 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:01.491511    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr"
Sep 16 17:05:01.509783 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:01.509744    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr"
Sep 16 17:05:06.208351 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:06.208313    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:05:06.214333 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:06.214301    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr"]
Sep 16 17:05:06.306320 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:06.306291    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:05:06.432652 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:06.432604    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj" podUID="904ae881-c154-4898-a1c9-12f9284a20a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8001/health\": dial tcp 10.132.0.47:8001: connect: connection refused"
Sep 16 17:05:16.431691 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:16.431644    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj" podUID="904ae881-c154-4898-a1c9-12f9284a20a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8001/health\": dial tcp 10.132.0.47:8001: connect: connection refused"
Sep 16 17:05:20.968187 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:20.968153    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:05:21.237701 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:21.237598    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr" podUID="1972bb3c-af5c-4913-8e0e-eadd9ddceb1e" containerName="main" containerID="cri-o://90bf5f131d5896829cc3530fa384cc94b2957cfe7c0b96213266f4b984bdb6bc" gracePeriod=45
Sep 16 17:05:21.493207 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:21.493111    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr" podUID="1972bb3c-af5c-4913-8e0e-eadd9ddceb1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 16 17:05:21.981913 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:21.981884    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr"
Sep 16 17:05:22.077908 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:22.077885    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-model-cache\") pod \"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e\" (UID: \"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e\") "
Sep 16 17:05:22.078081 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:22.077915    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bnmq\" (UniqueName: \"kubernetes.io/projected/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-kube-api-access-6bnmq\") pod \"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e\" (UID: \"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e\") "
Sep 16 17:05:22.078081 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:22.077940    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-tmp-dir\") pod \"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e\" (UID: \"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e\") "
Sep 16 17:05:22.078081 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:22.077962    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-home\") pod \"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e\" (UID: \"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e\") "
Sep 16 17:05:22.078081 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:22.077981    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-kserve-provision-location\") pod \"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e\" (UID: \"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e\") "
Sep 16 17:05:22.078081 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:22.078020    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-tls-certs\") pod \"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e\" (UID: \"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e\") "
Sep 16 17:05:22.078370 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:22.078089    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-dshm\") pod \"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e\" (UID: \"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e\") "
Sep 16 17:05:22.078370 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:22.078117    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-cabundle-cert\") pod \"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e\" (UID: \"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e\") "
Sep 16 17:05:22.078370 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:22.078243    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-model-cache" (OuterVolumeSpecName: "model-cache") pod "1972bb3c-af5c-4913-8e0e-eadd9ddceb1e" (UID: "1972bb3c-af5c-4913-8e0e-eadd9ddceb1e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:05:22.078370 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:22.078322    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-model-cache\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:05:22.078665 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:22.078642    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1972bb3c-af5c-4913-8e0e-eadd9ddceb1e" (UID: "1972bb3c-af5c-4913-8e0e-eadd9ddceb1e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:05:22.078946 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:22.078898    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-home" (OuterVolumeSpecName: "home") pod "1972bb3c-af5c-4913-8e0e-eadd9ddceb1e" (UID: "1972bb3c-af5c-4913-8e0e-eadd9ddceb1e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:05:22.080365 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:22.080339    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-kube-api-access-6bnmq" (OuterVolumeSpecName: "kube-api-access-6bnmq") pod "1972bb3c-af5c-4913-8e0e-eadd9ddceb1e" (UID: "1972bb3c-af5c-4913-8e0e-eadd9ddceb1e"). InnerVolumeSpecName "kube-api-access-6bnmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:05:22.080517 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:22.080493    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1972bb3c-af5c-4913-8e0e-eadd9ddceb1e" (UID: "1972bb3c-af5c-4913-8e0e-eadd9ddceb1e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:05:22.080783 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:22.080761    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-dshm" (OuterVolumeSpecName: "dshm") pod "1972bb3c-af5c-4913-8e0e-eadd9ddceb1e" (UID: "1972bb3c-af5c-4913-8e0e-eadd9ddceb1e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:05:22.097031 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:22.096985    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1972bb3c-af5c-4913-8e0e-eadd9ddceb1e" (UID: "1972bb3c-af5c-4913-8e0e-eadd9ddceb1e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:05:22.105054 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:22.105029    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1972bb3c-af5c-4913-8e0e-eadd9ddceb1e" (UID: "1972bb3c-af5c-4913-8e0e-eadd9ddceb1e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:05:22.137932 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:22.137897    2554 generic.go:356] "Generic (PLEG): container finished" podID="1972bb3c-af5c-4913-8e0e-eadd9ddceb1e" containerID="90bf5f131d5896829cc3530fa384cc94b2957cfe7c0b96213266f4b984bdb6bc" exitCode=0
Sep 16 17:05:22.138090 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:22.137962    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr" event={"ID":"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e","Type":"ContainerDied","Data":"90bf5f131d5896829cc3530fa384cc94b2957cfe7c0b96213266f4b984bdb6bc"}
Sep 16 17:05:22.138090 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:22.138006    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr" event={"ID":"1972bb3c-af5c-4913-8e0e-eadd9ddceb1e","Type":"ContainerDied","Data":"c273d8ea576ee514e03afd0a73757ba7bb987a19d05585a5cd6ee4019e6c06b4"}
Sep 16 17:05:22.138090 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:22.138029    2554 scope.go:117] "RemoveContainer" containerID="90bf5f131d5896829cc3530fa384cc94b2957cfe7c0b96213266f4b984bdb6bc"
Sep 16 17:05:22.138090 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:22.137963    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr"
Sep 16 17:05:22.147770 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:22.147751    2554 scope.go:117] "RemoveContainer" containerID="0afc608aa690204c5f0d2501b979832053aaddb87669f02122b4a9c0e9a62b0e"
Sep 16 17:05:22.157009 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:22.156992    2554 scope.go:117] "RemoveContainer" containerID="90bf5f131d5896829cc3530fa384cc94b2957cfe7c0b96213266f4b984bdb6bc"
Sep 16 17:05:22.157344 ip-10-0-136-228 kubenswrapper[2554]: E0916 17:05:22.157312    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90bf5f131d5896829cc3530fa384cc94b2957cfe7c0b96213266f4b984bdb6bc\": container with ID starting with 90bf5f131d5896829cc3530fa384cc94b2957cfe7c0b96213266f4b984bdb6bc not found: ID does not exist" containerID="90bf5f131d5896829cc3530fa384cc94b2957cfe7c0b96213266f4b984bdb6bc"
Sep 16 17:05:22.157417 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:22.157357    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90bf5f131d5896829cc3530fa384cc94b2957cfe7c0b96213266f4b984bdb6bc"} err="failed to get container status \"90bf5f131d5896829cc3530fa384cc94b2957cfe7c0b96213266f4b984bdb6bc\": rpc error: code = NotFound desc = could not find container \"90bf5f131d5896829cc3530fa384cc94b2957cfe7c0b96213266f4b984bdb6bc\": container with ID starting with 90bf5f131d5896829cc3530fa384cc94b2957cfe7c0b96213266f4b984bdb6bc not found: ID does not exist"
Sep 16 17:05:22.157417 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:22.157381    2554 scope.go:117] "RemoveContainer" containerID="0afc608aa690204c5f0d2501b979832053aaddb87669f02122b4a9c0e9a62b0e"
Sep 16 17:05:22.157694 ip-10-0-136-228 kubenswrapper[2554]: E0916 17:05:22.157675    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0afc608aa690204c5f0d2501b979832053aaddb87669f02122b4a9c0e9a62b0e\": container with ID starting with 0afc608aa690204c5f0d2501b979832053aaddb87669f02122b4a9c0e9a62b0e not found: ID does not exist" containerID="0afc608aa690204c5f0d2501b979832053aaddb87669f02122b4a9c0e9a62b0e"
Sep 16 17:05:22.157770 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:22.157698    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0afc608aa690204c5f0d2501b979832053aaddb87669f02122b4a9c0e9a62b0e"} err="failed to get container status \"0afc608aa690204c5f0d2501b979832053aaddb87669f02122b4a9c0e9a62b0e\": rpc error: code = NotFound desc = could not find container \"0afc608aa690204c5f0d2501b979832053aaddb87669f02122b4a9c0e9a62b0e\": container with ID starting with 0afc608aa690204c5f0d2501b979832053aaddb87669f02122b4a9c0e9a62b0e not found: ID does not exist"
Sep 16 17:05:22.161112 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:22.161090    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr"]
Sep 16 17:05:22.166034 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:22.166009    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5bc4ff64-mnpsr"]
Sep 16 17:05:22.179034 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:22.179012    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-dshm\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:05:22.179034 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:22.179033    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:05:22.179188 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:22.179043    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bnmq\" (UniqueName: \"kubernetes.io/projected/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-kube-api-access-6bnmq\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:05:22.179188 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:22.179073    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-tmp-dir\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:05:22.179188 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:22.179085    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-home\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:05:22.179188 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:22.179098    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:05:22.179188 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:22.179107    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e-tls-certs\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:05:24.077098 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:24.077064    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1972bb3c-af5c-4913-8e0e-eadd9ddceb1e" path="/var/lib/kubelet/pods/1972bb3c-af5c-4913-8e0e-eadd9ddceb1e/volumes"
Sep 16 17:05:26.431933 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:26.431882    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj" podUID="904ae881-c154-4898-a1c9-12f9284a20a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8001/health\": dial tcp 10.132.0.47:8001: connect: connection refused"
Sep 16 17:05:36.432604 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:36.432556    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj" podUID="904ae881-c154-4898-a1c9-12f9284a20a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8001/health\": dial tcp 10.132.0.47:8001: connect: connection refused"
Sep 16 17:05:46.432198 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:46.432145    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj" podUID="904ae881-c154-4898-a1c9-12f9284a20a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8001/health\": dial tcp 10.132.0.47:8001: connect: connection refused"
Sep 16 17:05:56.341017 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:56.340982    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:05:56.432378 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:56.432330    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj" podUID="904ae881-c154-4898-a1c9-12f9284a20a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8001/health\": dial tcp 10.132.0.47:8001: connect: connection refused"
Sep 16 17:05:56.435742 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:05:56.435709    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:06:02.576751 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:02.576711    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:06:06.432034 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:06.431985    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj" podUID="904ae881-c154-4898-a1c9-12f9284a20a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8001/health\": dial tcp 10.132.0.47:8001: connect: connection refused"
Sep 16 17:06:07.980260 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:07.980224    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578"]
Sep 16 17:06:07.980722 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:07.980515    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1972bb3c-af5c-4913-8e0e-eadd9ddceb1e" containerName="storage-initializer"
Sep 16 17:06:07.980722 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:07.980528    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1972bb3c-af5c-4913-8e0e-eadd9ddceb1e" containerName="storage-initializer"
Sep 16 17:06:07.980722 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:07.980542    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1972bb3c-af5c-4913-8e0e-eadd9ddceb1e" containerName="main"
Sep 16 17:06:07.980722 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:07.980548    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1972bb3c-af5c-4913-8e0e-eadd9ddceb1e" containerName="main"
Sep 16 17:06:07.980722 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:07.980600    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="1972bb3c-af5c-4913-8e0e-eadd9ddceb1e" containerName="main"
Sep 16 17:06:07.985006 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:07.984987    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578"
Sep 16 17:06:07.987889 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:07.987853    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 17:06:07.988026 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:07.987961    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 16 17:06:07.988026 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:07.987969    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 16 17:06:07.988165 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:07.988028    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-tznph\""
Sep 16 17:06:07.988165 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:07.988077    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 16 17:06:07.988165 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:07.988102    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 16 17:06:07.995930 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:07.995905    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578"]
Sep 16 17:06:08.002858 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:08.002833    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b314208-e5a0-444b-9ac1-987ba23296ea-dshm\") pod \"prestop-hook-test-kserve-7f5c764488-g7578\" (UID: \"1b314208-e5a0-444b-9ac1-987ba23296ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578"
Sep 16 17:06:08.002976 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:08.002868    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b314208-e5a0-444b-9ac1-987ba23296ea-kserve-provision-location\") pod \"prestop-hook-test-kserve-7f5c764488-g7578\" (UID: \"1b314208-e5a0-444b-9ac1-987ba23296ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578"
Sep 16 17:06:08.002976 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:08.002893    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gc2k\" (UniqueName: \"kubernetes.io/projected/1b314208-e5a0-444b-9ac1-987ba23296ea-kube-api-access-2gc2k\") pod \"prestop-hook-test-kserve-7f5c764488-g7578\" (UID: \"1b314208-e5a0-444b-9ac1-987ba23296ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578"
Sep 16 17:06:08.002976 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:08.002964    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b314208-e5a0-444b-9ac1-987ba23296ea-model-cache\") pod \"prestop-hook-test-kserve-7f5c764488-g7578\" (UID: \"1b314208-e5a0-444b-9ac1-987ba23296ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578"
Sep 16 17:06:08.003153 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:08.002993    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b314208-e5a0-444b-9ac1-987ba23296ea-home\") pod \"prestop-hook-test-kserve-7f5c764488-g7578\" (UID: \"1b314208-e5a0-444b-9ac1-987ba23296ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578"
Sep 16 17:06:08.003153 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:08.003012    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b314208-e5a0-444b-9ac1-987ba23296ea-cabundle-cert\") pod \"prestop-hook-test-kserve-7f5c764488-g7578\" (UID: \"1b314208-e5a0-444b-9ac1-987ba23296ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578"
Sep 16 17:06:08.003153 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:08.003040    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b314208-e5a0-444b-9ac1-987ba23296ea-tls-certs\") pod \"prestop-hook-test-kserve-7f5c764488-g7578\" (UID: \"1b314208-e5a0-444b-9ac1-987ba23296ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578"
Sep 16 17:06:08.003153 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:08.003064    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b314208-e5a0-444b-9ac1-987ba23296ea-tmp-dir\") pod \"prestop-hook-test-kserve-7f5c764488-g7578\" (UID: \"1b314208-e5a0-444b-9ac1-987ba23296ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578"
Sep 16 17:06:08.016727 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:08.016702    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:06:08.103872 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:08.103847    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b314208-e5a0-444b-9ac1-987ba23296ea-dshm\") pod \"prestop-hook-test-kserve-7f5c764488-g7578\" (UID: \"1b314208-e5a0-444b-9ac1-987ba23296ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578"
Sep 16 17:06:08.104062 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:08.103881    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b314208-e5a0-444b-9ac1-987ba23296ea-kserve-provision-location\") pod \"prestop-hook-test-kserve-7f5c764488-g7578\" (UID: \"1b314208-e5a0-444b-9ac1-987ba23296ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578"
Sep 16 17:06:08.104062 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:08.103904    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gc2k\" (UniqueName: \"kubernetes.io/projected/1b314208-e5a0-444b-9ac1-987ba23296ea-kube-api-access-2gc2k\") pod \"prestop-hook-test-kserve-7f5c764488-g7578\" (UID: \"1b314208-e5a0-444b-9ac1-987ba23296ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578"
Sep 16 17:06:08.104272 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:08.104243    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b314208-e5a0-444b-9ac1-987ba23296ea-kserve-provision-location\") pod \"prestop-hook-test-kserve-7f5c764488-g7578\" (UID: \"1b314208-e5a0-444b-9ac1-987ba23296ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578"
Sep 16 17:06:08.104411 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:08.104309    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b314208-e5a0-444b-9ac1-987ba23296ea-model-cache\") pod \"prestop-hook-test-kserve-7f5c764488-g7578\" (UID: \"1b314208-e5a0-444b-9ac1-987ba23296ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578"
Sep 16 17:06:08.104411 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:08.104362    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b314208-e5a0-444b-9ac1-987ba23296ea-home\") pod \"prestop-hook-test-kserve-7f5c764488-g7578\" (UID: \"1b314208-e5a0-444b-9ac1-987ba23296ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578"
Sep 16 17:06:08.104526 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:08.104440    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b314208-e5a0-444b-9ac1-987ba23296ea-cabundle-cert\") pod \"prestop-hook-test-kserve-7f5c764488-g7578\" (UID: \"1b314208-e5a0-444b-9ac1-987ba23296ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578"
Sep 16 17:06:08.104526 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:08.104491    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b314208-e5a0-444b-9ac1-987ba23296ea-tls-certs\") pod \"prestop-hook-test-kserve-7f5c764488-g7578\" (UID: \"1b314208-e5a0-444b-9ac1-987ba23296ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578"
Sep 16 17:06:08.104632 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:08.104525    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b314208-e5a0-444b-9ac1-987ba23296ea-tmp-dir\") pod \"prestop-hook-test-kserve-7f5c764488-g7578\" (UID: \"1b314208-e5a0-444b-9ac1-987ba23296ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578"
Sep 16 17:06:08.104632 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:08.104614    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b314208-e5a0-444b-9ac1-987ba23296ea-model-cache\") pod \"prestop-hook-test-kserve-7f5c764488-g7578\" (UID: \"1b314208-e5a0-444b-9ac1-987ba23296ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578"
Sep 16 17:06:08.104749 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:08.104655    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b314208-e5a0-444b-9ac1-987ba23296ea-home\") pod \"prestop-hook-test-kserve-7f5c764488-g7578\" (UID: \"1b314208-e5a0-444b-9ac1-987ba23296ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578"
Sep 16 17:06:08.104839 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:08.104822    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b314208-e5a0-444b-9ac1-987ba23296ea-tmp-dir\") pod \"prestop-hook-test-kserve-7f5c764488-g7578\" (UID: \"1b314208-e5a0-444b-9ac1-987ba23296ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578"
Sep 16 17:06:08.105142 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:08.105122    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b314208-e5a0-444b-9ac1-987ba23296ea-cabundle-cert\") pod \"prestop-hook-test-kserve-7f5c764488-g7578\" (UID: \"1b314208-e5a0-444b-9ac1-987ba23296ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578"
Sep 16 17:06:08.106415 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:08.106388    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b314208-e5a0-444b-9ac1-987ba23296ea-dshm\") pod \"prestop-hook-test-kserve-7f5c764488-g7578\" (UID: \"1b314208-e5a0-444b-9ac1-987ba23296ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578"
Sep 16 17:06:08.106953 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:08.106934    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b314208-e5a0-444b-9ac1-987ba23296ea-tls-certs\") pod \"prestop-hook-test-kserve-7f5c764488-g7578\" (UID: \"1b314208-e5a0-444b-9ac1-987ba23296ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578"
Sep 16 17:06:08.111760 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:08.111738    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gc2k\" (UniqueName: \"kubernetes.io/projected/1b314208-e5a0-444b-9ac1-987ba23296ea-kube-api-access-2gc2k\") pod \"prestop-hook-test-kserve-7f5c764488-g7578\" (UID: \"1b314208-e5a0-444b-9ac1-987ba23296ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578"
Sep 16 17:06:08.240154 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:08.240075    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:06:08.298692 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:08.298665    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578"
Sep 16 17:06:08.577787 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:08.577756    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578"]
Sep 16 17:06:08.579941 ip-10-0-136-228 kubenswrapper[2554]: W0916 17:06:08.579912    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b314208_e5a0_444b_9ac1_987ba23296ea.slice/crio-1330b18277a4fdf70d2d18cca8c280409bbb34968ff52d662323743310696563 WatchSource:0}: Error finding container 1330b18277a4fdf70d2d18cca8c280409bbb34968ff52d662323743310696563: Status 404 returned error can't find the container with id 1330b18277a4fdf70d2d18cca8c280409bbb34968ff52d662323743310696563
Sep 16 17:06:08.842651 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:08.842539    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:06:08.947827 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:08.947782    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:06:09.335269 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:09.335238    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578" event={"ID":"1b314208-e5a0-444b-9ac1-987ba23296ea","Type":"ContainerStarted","Data":"d1933441ad4479d06bfbcd261c39d0d7fd185554cfc404117399a69cedb0cc80"}
Sep 16 17:06:09.335687 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:09.335288    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578" event={"ID":"1b314208-e5a0-444b-9ac1-987ba23296ea","Type":"ContainerStarted","Data":"1330b18277a4fdf70d2d18cca8c280409bbb34968ff52d662323743310696563"}
Sep 16 17:06:11.344332 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:11.344302    2554 generic.go:356] "Generic (PLEG): container finished" podID="1b314208-e5a0-444b-9ac1-987ba23296ea" containerID="d1933441ad4479d06bfbcd261c39d0d7fd185554cfc404117399a69cedb0cc80" exitCode=0
Sep 16 17:06:11.344740 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:11.344378    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578" event={"ID":"1b314208-e5a0-444b-9ac1-987ba23296ea","Type":"ContainerDied","Data":"d1933441ad4479d06bfbcd261c39d0d7fd185554cfc404117399a69cedb0cc80"}
Sep 16 17:06:12.351863 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:12.351824    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578" event={"ID":"1b314208-e5a0-444b-9ac1-987ba23296ea","Type":"ContainerStarted","Data":"082bc2b7a3dc9191b645f4c41af80b7240e5aef4d2e07632818288cd0467304f"}
Sep 16 17:06:12.375986 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:12.375932    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578" podStartSLOduration=3.133400476 podStartE2EDuration="5.375914671s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.242514195s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:06:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:06:12.373735683 +0000 UTC m=+4572.818427884" watchObservedRunningTime="2026-09-16 17:06:12.375914671 +0000 UTC m=+4572.820606849"
Sep 16 17:06:13.299517 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:13.299477    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578"
Sep 16 17:06:16.432527 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:16.432396    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj" podUID="904ae881-c154-4898-a1c9-12f9284a20a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8001/health\": dial tcp 10.132.0.47:8001: connect: connection refused"
Sep 16 17:06:18.299688 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:18.299640    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578"
Sep 16 17:06:18.301211 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:18.301177    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578" podUID="1b314208-e5a0-444b-9ac1-987ba23296ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 16 17:06:26.432410 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:26.432301    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj" podUID="904ae881-c154-4898-a1c9-12f9284a20a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8001/health\": dial tcp 10.132.0.47:8001: connect: connection refused"
Sep 16 17:06:28.299447 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:28.299397    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578" podUID="1b314208-e5a0-444b-9ac1-987ba23296ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 16 17:06:36.432285 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:36.432240    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj" podUID="904ae881-c154-4898-a1c9-12f9284a20a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8001/health\": dial tcp 10.132.0.47:8001: connect: connection refused"
Sep 16 17:06:38.299661 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:38.299611    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578" podUID="1b314208-e5a0-444b-9ac1-987ba23296ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 16 17:06:39.293687 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:39.293650    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:06:46.441301 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:46.441268    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj"
Sep 16 17:06:46.453002 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:46.452978    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj"
Sep 16 17:06:48.299200 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:48.299148    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578" podUID="1b314208-e5a0-444b-9ac1-987ba23296ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 16 17:06:55.738692 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:55.738655    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:06:55.746822 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:55.746777    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj"]
Sep 16 17:06:55.799951 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:55.799914    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:06:58.299814 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:06:58.299770    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578" podUID="1b314208-e5a0-444b-9ac1-987ba23296ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 16 17:07:08.299447 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:08.299405    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578" podUID="1b314208-e5a0-444b-9ac1-987ba23296ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 16 17:07:10.768986 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:10.768937    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj" podUID="904ae881-c154-4898-a1c9-12f9284a20a4" containerName="main" containerID="cri-o://81fcf56ac24b65b72acd27ef7a068f53a408f8bc84aef8bd94dded09d84a8212" gracePeriod=45
Sep 16 17:07:11.381581 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.381543    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj" podUID="904ae881-c154-4898-a1c9-12f9284a20a4" containerName="llm-d-routing-sidecar" containerID="cri-o://55558b1365fd7bfbb252dda4a3203e5dd29c3f918ccee5ad8cd4ca4575c9b894" gracePeriod=45
Sep 16 17:07:11.618339 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.618302    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj"
Sep 16 17:07:11.620460 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.620431    2554 generic.go:356] "Generic (PLEG): container finished" podID="904ae881-c154-4898-a1c9-12f9284a20a4" containerID="81fcf56ac24b65b72acd27ef7a068f53a408f8bc84aef8bd94dded09d84a8212" exitCode=0
Sep 16 17:07:11.620460 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.620458    2554 generic.go:356] "Generic (PLEG): container finished" podID="904ae881-c154-4898-a1c9-12f9284a20a4" containerID="55558b1365fd7bfbb252dda4a3203e5dd29c3f918ccee5ad8cd4ca4575c9b894" exitCode=0
Sep 16 17:07:11.620625 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.620505    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj" event={"ID":"904ae881-c154-4898-a1c9-12f9284a20a4","Type":"ContainerDied","Data":"81fcf56ac24b65b72acd27ef7a068f53a408f8bc84aef8bd94dded09d84a8212"}
Sep 16 17:07:11.620625 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.620564    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj" event={"ID":"904ae881-c154-4898-a1c9-12f9284a20a4","Type":"ContainerDied","Data":"55558b1365fd7bfbb252dda4a3203e5dd29c3f918ccee5ad8cd4ca4575c9b894"}
Sep 16 17:07:11.620625 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.620585    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj" event={"ID":"904ae881-c154-4898-a1c9-12f9284a20a4","Type":"ContainerDied","Data":"bd19028a90cf153782def33d9edbc53b12c88f4906784c9c81a481e20cdcd062"}
Sep 16 17:07:11.620625 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.620606    2554 scope.go:117] "RemoveContainer" containerID="81fcf56ac24b65b72acd27ef7a068f53a408f8bc84aef8bd94dded09d84a8212"
Sep 16 17:07:11.629258 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.629233    2554 scope.go:117] "RemoveContainer" containerID="83b0867ca9bcbe95d8a238589ec72665e7271d85c5ba7e6b895f233c247551d1"
Sep 16 17:07:11.638320 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.638301    2554 scope.go:117] "RemoveContainer" containerID="55558b1365fd7bfbb252dda4a3203e5dd29c3f918ccee5ad8cd4ca4575c9b894"
Sep 16 17:07:11.646550 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.646508    2554 scope.go:117] "RemoveContainer" containerID="81fcf56ac24b65b72acd27ef7a068f53a408f8bc84aef8bd94dded09d84a8212"
Sep 16 17:07:11.646956 ip-10-0-136-228 kubenswrapper[2554]: E0916 17:07:11.646931    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81fcf56ac24b65b72acd27ef7a068f53a408f8bc84aef8bd94dded09d84a8212\": container with ID starting with 81fcf56ac24b65b72acd27ef7a068f53a408f8bc84aef8bd94dded09d84a8212 not found: ID does not exist" containerID="81fcf56ac24b65b72acd27ef7a068f53a408f8bc84aef8bd94dded09d84a8212"
Sep 16 17:07:11.647070 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.646987    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81fcf56ac24b65b72acd27ef7a068f53a408f8bc84aef8bd94dded09d84a8212"} err="failed to get container status \"81fcf56ac24b65b72acd27ef7a068f53a408f8bc84aef8bd94dded09d84a8212\": rpc error: code = NotFound desc = could not find container \"81fcf56ac24b65b72acd27ef7a068f53a408f8bc84aef8bd94dded09d84a8212\": container with ID starting with 81fcf56ac24b65b72acd27ef7a068f53a408f8bc84aef8bd94dded09d84a8212 not found: ID does not exist"
Sep 16 17:07:11.647070 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.647014    2554 scope.go:117] "RemoveContainer" containerID="83b0867ca9bcbe95d8a238589ec72665e7271d85c5ba7e6b895f233c247551d1"
Sep 16 17:07:11.647384 ip-10-0-136-228 kubenswrapper[2554]: E0916 17:07:11.647353    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83b0867ca9bcbe95d8a238589ec72665e7271d85c5ba7e6b895f233c247551d1\": container with ID starting with 83b0867ca9bcbe95d8a238589ec72665e7271d85c5ba7e6b895f233c247551d1 not found: ID does not exist" containerID="83b0867ca9bcbe95d8a238589ec72665e7271d85c5ba7e6b895f233c247551d1"
Sep 16 17:07:11.647465 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.647391    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83b0867ca9bcbe95d8a238589ec72665e7271d85c5ba7e6b895f233c247551d1"} err="failed to get container status \"83b0867ca9bcbe95d8a238589ec72665e7271d85c5ba7e6b895f233c247551d1\": rpc error: code = NotFound desc = could not find container \"83b0867ca9bcbe95d8a238589ec72665e7271d85c5ba7e6b895f233c247551d1\": container with ID starting with 83b0867ca9bcbe95d8a238589ec72665e7271d85c5ba7e6b895f233c247551d1 not found: ID does not exist"
Sep 16 17:07:11.647465 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.647415    2554 scope.go:117] "RemoveContainer" containerID="55558b1365fd7bfbb252dda4a3203e5dd29c3f918ccee5ad8cd4ca4575c9b894"
Sep 16 17:07:11.647778 ip-10-0-136-228 kubenswrapper[2554]: E0916 17:07:11.647745    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55558b1365fd7bfbb252dda4a3203e5dd29c3f918ccee5ad8cd4ca4575c9b894\": container with ID starting with 55558b1365fd7bfbb252dda4a3203e5dd29c3f918ccee5ad8cd4ca4575c9b894 not found: ID does not exist" containerID="55558b1365fd7bfbb252dda4a3203e5dd29c3f918ccee5ad8cd4ca4575c9b894"
Sep 16 17:07:11.647862 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.647788    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55558b1365fd7bfbb252dda4a3203e5dd29c3f918ccee5ad8cd4ca4575c9b894"} err="failed to get container status \"55558b1365fd7bfbb252dda4a3203e5dd29c3f918ccee5ad8cd4ca4575c9b894\": rpc error: code = NotFound desc = could not find container \"55558b1365fd7bfbb252dda4a3203e5dd29c3f918ccee5ad8cd4ca4575c9b894\": container with ID starting with 55558b1365fd7bfbb252dda4a3203e5dd29c3f918ccee5ad8cd4ca4575c9b894 not found: ID does not exist"
Sep 16 17:07:11.647862 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.647823    2554 scope.go:117] "RemoveContainer" containerID="81fcf56ac24b65b72acd27ef7a068f53a408f8bc84aef8bd94dded09d84a8212"
Sep 16 17:07:11.648226 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.648194    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81fcf56ac24b65b72acd27ef7a068f53a408f8bc84aef8bd94dded09d84a8212"} err="failed to get container status \"81fcf56ac24b65b72acd27ef7a068f53a408f8bc84aef8bd94dded09d84a8212\": rpc error: code = NotFound desc = could not find container \"81fcf56ac24b65b72acd27ef7a068f53a408f8bc84aef8bd94dded09d84a8212\": container with ID starting with 81fcf56ac24b65b72acd27ef7a068f53a408f8bc84aef8bd94dded09d84a8212 not found: ID does not exist"
Sep 16 17:07:11.648321 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.648228    2554 scope.go:117] "RemoveContainer" containerID="83b0867ca9bcbe95d8a238589ec72665e7271d85c5ba7e6b895f233c247551d1"
Sep 16 17:07:11.648546 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.648518    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83b0867ca9bcbe95d8a238589ec72665e7271d85c5ba7e6b895f233c247551d1"} err="failed to get container status \"83b0867ca9bcbe95d8a238589ec72665e7271d85c5ba7e6b895f233c247551d1\": rpc error: code = NotFound desc = could not find container \"83b0867ca9bcbe95d8a238589ec72665e7271d85c5ba7e6b895f233c247551d1\": container with ID starting with 83b0867ca9bcbe95d8a238589ec72665e7271d85c5ba7e6b895f233c247551d1 not found: ID does not exist"
Sep 16 17:07:11.648546 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.648547    2554 scope.go:117] "RemoveContainer" containerID="55558b1365fd7bfbb252dda4a3203e5dd29c3f918ccee5ad8cd4ca4575c9b894"
Sep 16 17:07:11.648890 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.648870    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55558b1365fd7bfbb252dda4a3203e5dd29c3f918ccee5ad8cd4ca4575c9b894"} err="failed to get container status \"55558b1365fd7bfbb252dda4a3203e5dd29c3f918ccee5ad8cd4ca4575c9b894\": rpc error: code = NotFound desc = could not find container \"55558b1365fd7bfbb252dda4a3203e5dd29c3f918ccee5ad8cd4ca4575c9b894\": container with ID starting with 55558b1365fd7bfbb252dda4a3203e5dd29c3f918ccee5ad8cd4ca4575c9b894 not found: ID does not exist"
Sep 16 17:07:11.719143 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.719110    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/904ae881-c154-4898-a1c9-12f9284a20a4-home\") pod \"904ae881-c154-4898-a1c9-12f9284a20a4\" (UID: \"904ae881-c154-4898-a1c9-12f9284a20a4\") "
Sep 16 17:07:11.719321 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.719152    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/904ae881-c154-4898-a1c9-12f9284a20a4-cabundle-cert\") pod \"904ae881-c154-4898-a1c9-12f9284a20a4\" (UID: \"904ae881-c154-4898-a1c9-12f9284a20a4\") "
Sep 16 17:07:11.719321 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.719179    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jn59\" (UniqueName: \"kubernetes.io/projected/904ae881-c154-4898-a1c9-12f9284a20a4-kube-api-access-4jn59\") pod \"904ae881-c154-4898-a1c9-12f9284a20a4\" (UID: \"904ae881-c154-4898-a1c9-12f9284a20a4\") "
Sep 16 17:07:11.719321 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.719200    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/904ae881-c154-4898-a1c9-12f9284a20a4-tmp-dir\") pod \"904ae881-c154-4898-a1c9-12f9284a20a4\" (UID: \"904ae881-c154-4898-a1c9-12f9284a20a4\") "
Sep 16 17:07:11.719488 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.719345    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/904ae881-c154-4898-a1c9-12f9284a20a4-tls-certs\") pod \"904ae881-c154-4898-a1c9-12f9284a20a4\" (UID: \"904ae881-c154-4898-a1c9-12f9284a20a4\") "
Sep 16 17:07:11.719488 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.719435    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/904ae881-c154-4898-a1c9-12f9284a20a4-dshm\") pod \"904ae881-c154-4898-a1c9-12f9284a20a4\" (UID: \"904ae881-c154-4898-a1c9-12f9284a20a4\") "
Sep 16 17:07:11.719582 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.719472    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/904ae881-c154-4898-a1c9-12f9284a20a4-kserve-provision-location\") pod \"904ae881-c154-4898-a1c9-12f9284a20a4\" (UID: \"904ae881-c154-4898-a1c9-12f9284a20a4\") "
Sep 16 17:07:11.719582 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.719520    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/904ae881-c154-4898-a1c9-12f9284a20a4-model-cache\") pod \"904ae881-c154-4898-a1c9-12f9284a20a4\" (UID: \"904ae881-c154-4898-a1c9-12f9284a20a4\") "
Sep 16 17:07:11.719582 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.719531    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/904ae881-c154-4898-a1c9-12f9284a20a4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "904ae881-c154-4898-a1c9-12f9284a20a4" (UID: "904ae881-c154-4898-a1c9-12f9284a20a4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:07:11.720091 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.719885    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904ae881-c154-4898-a1c9-12f9284a20a4-home" (OuterVolumeSpecName: "home") pod "904ae881-c154-4898-a1c9-12f9284a20a4" (UID: "904ae881-c154-4898-a1c9-12f9284a20a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:07:11.720091 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.719893    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/904ae881-c154-4898-a1c9-12f9284a20a4-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:07:11.720091 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.720035    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904ae881-c154-4898-a1c9-12f9284a20a4-model-cache" (OuterVolumeSpecName: "model-cache") pod "904ae881-c154-4898-a1c9-12f9284a20a4" (UID: "904ae881-c154-4898-a1c9-12f9284a20a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:07:11.722206 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.722183    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904ae881-c154-4898-a1c9-12f9284a20a4-dshm" (OuterVolumeSpecName: "dshm") pod "904ae881-c154-4898-a1c9-12f9284a20a4" (UID: "904ae881-c154-4898-a1c9-12f9284a20a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:07:11.722206 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.722188    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/904ae881-c154-4898-a1c9-12f9284a20a4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "904ae881-c154-4898-a1c9-12f9284a20a4" (UID: "904ae881-c154-4898-a1c9-12f9284a20a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:07:11.722351 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.722276    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/904ae881-c154-4898-a1c9-12f9284a20a4-kube-api-access-4jn59" (OuterVolumeSpecName: "kube-api-access-4jn59") pod "904ae881-c154-4898-a1c9-12f9284a20a4" (UID: "904ae881-c154-4898-a1c9-12f9284a20a4"). InnerVolumeSpecName "kube-api-access-4jn59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:07:11.734687 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.734661    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904ae881-c154-4898-a1c9-12f9284a20a4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "904ae881-c154-4898-a1c9-12f9284a20a4" (UID: "904ae881-c154-4898-a1c9-12f9284a20a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:07:11.739990 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.739966    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904ae881-c154-4898-a1c9-12f9284a20a4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "904ae881-c154-4898-a1c9-12f9284a20a4" (UID: "904ae881-c154-4898-a1c9-12f9284a20a4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:07:11.820914 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.820835    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/904ae881-c154-4898-a1c9-12f9284a20a4-tls-certs\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:07:11.820914 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.820863    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/904ae881-c154-4898-a1c9-12f9284a20a4-dshm\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:07:11.820914 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.820874    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/904ae881-c154-4898-a1c9-12f9284a20a4-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:07:11.820914 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.820886    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/904ae881-c154-4898-a1c9-12f9284a20a4-model-cache\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:07:11.820914 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.820894    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/904ae881-c154-4898-a1c9-12f9284a20a4-home\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:07:11.820914 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.820902    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jn59\" (UniqueName: \"kubernetes.io/projected/904ae881-c154-4898-a1c9-12f9284a20a4-kube-api-access-4jn59\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:07:11.820914 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:11.820909    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/904ae881-c154-4898-a1c9-12f9284a20a4-tmp-dir\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:07:12.628632 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:12.628601    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj"
Sep 16 17:07:12.648580 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:12.648545    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj"]
Sep 16 17:07:12.654584 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:12.654552    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5b6456f669-k4rwj"]
Sep 16 17:07:14.075933 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:14.075898    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="904ae881-c154-4898-a1c9-12f9284a20a4" path="/var/lib/kubelet/pods/904ae881-c154-4898-a1c9-12f9284a20a4/volumes"
Sep 16 17:07:18.299519 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:18.299478    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578" podUID="1b314208-e5a0-444b-9ac1-987ba23296ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 16 17:07:28.299648 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:28.299610    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578" podUID="1b314208-e5a0-444b-9ac1-987ba23296ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 16 17:07:38.299983 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:38.299934    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578" podUID="1b314208-e5a0-444b-9ac1-987ba23296ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 16 17:07:45.901859 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:45.901824    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:07:46.004655 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:46.004622    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:07:48.299090 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:48.299051    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578" podUID="1b314208-e5a0-444b-9ac1-987ba23296ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 16 17:07:50.952589 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:50.952557    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:07:51.474983 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:51.474948    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:07:58.299496 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:07:58.299453    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578" podUID="1b314208-e5a0-444b-9ac1-987ba23296ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 16 17:08:08.310459 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:08.310427    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578"
Sep 16 17:08:08.318237 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:08.318214    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578"
Sep 16 17:08:12.027869 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:12.027829    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:08:12.084960 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:12.084929    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578"]
Sep 16 17:08:12.154904 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:12.154867    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf"]
Sep 16 17:08:12.155141 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:12.155127    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="904ae881-c154-4898-a1c9-12f9284a20a4" containerName="storage-initializer"
Sep 16 17:08:12.155191 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:12.155144    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="904ae881-c154-4898-a1c9-12f9284a20a4" containerName="storage-initializer"
Sep 16 17:08:12.155191 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:12.155165    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="904ae881-c154-4898-a1c9-12f9284a20a4" containerName="main"
Sep 16 17:08:12.155191 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:12.155172    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="904ae881-c154-4898-a1c9-12f9284a20a4" containerName="main"
Sep 16 17:08:12.155191 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:12.155180    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="904ae881-c154-4898-a1c9-12f9284a20a4" containerName="llm-d-routing-sidecar"
Sep 16 17:08:12.155191 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:12.155186    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="904ae881-c154-4898-a1c9-12f9284a20a4" containerName="llm-d-routing-sidecar"
Sep 16 17:08:12.155360 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:12.155235    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="904ae881-c154-4898-a1c9-12f9284a20a4" containerName="llm-d-routing-sidecar"
Sep 16 17:08:12.155360 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:12.155244    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="904ae881-c154-4898-a1c9-12f9284a20a4" containerName="main"
Sep 16 17:08:12.160952 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:12.160931    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf"
Sep 16 17:08:12.177921 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:12.177897    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf"]
Sep 16 17:08:12.282935 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:12.282907    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c60aecef-b09c-4447-9baf-fd4d5b90614a-tmp-dir\") pod \"prestop-hook-test-kserve-7f5c764488-95dmf\" (UID: \"c60aecef-b09c-4447-9baf-fd4d5b90614a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf"
Sep 16 17:08:12.283090 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:12.282946    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pllpd\" (UniqueName: \"kubernetes.io/projected/c60aecef-b09c-4447-9baf-fd4d5b90614a-kube-api-access-pllpd\") pod \"prestop-hook-test-kserve-7f5c764488-95dmf\" (UID: \"c60aecef-b09c-4447-9baf-fd4d5b90614a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf"
Sep 16 17:08:12.283090 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:12.282972    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c60aecef-b09c-4447-9baf-fd4d5b90614a-dshm\") pod \"prestop-hook-test-kserve-7f5c764488-95dmf\" (UID: \"c60aecef-b09c-4447-9baf-fd4d5b90614a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf"
Sep 16 17:08:12.283090 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:12.283037    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c60aecef-b09c-4447-9baf-fd4d5b90614a-model-cache\") pod \"prestop-hook-test-kserve-7f5c764488-95dmf\" (UID: \"c60aecef-b09c-4447-9baf-fd4d5b90614a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf"
Sep 16 17:08:12.283090 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:12.283069    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c60aecef-b09c-4447-9baf-fd4d5b90614a-kserve-provision-location\") pod \"prestop-hook-test-kserve-7f5c764488-95dmf\" (UID: \"c60aecef-b09c-4447-9baf-fd4d5b90614a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf"
Sep 16 17:08:12.283090 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:12.283091    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c60aecef-b09c-4447-9baf-fd4d5b90614a-home\") pod \"prestop-hook-test-kserve-7f5c764488-95dmf\" (UID: \"c60aecef-b09c-4447-9baf-fd4d5b90614a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf"
Sep 16 17:08:12.283292 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:12.283108    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c60aecef-b09c-4447-9baf-fd4d5b90614a-tls-certs\") pod \"prestop-hook-test-kserve-7f5c764488-95dmf\" (UID: \"c60aecef-b09c-4447-9baf-fd4d5b90614a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf"
Sep 16 17:08:12.283292 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:12.283139    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c60aecef-b09c-4447-9baf-fd4d5b90614a-cabundle-cert\") pod \"prestop-hook-test-kserve-7f5c764488-95dmf\" (UID: \"c60aecef-b09c-4447-9baf-fd4d5b90614a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf"
Sep 16 17:08:12.384348 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:12.384316    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c60aecef-b09c-4447-9baf-fd4d5b90614a-dshm\") pod \"prestop-hook-test-kserve-7f5c764488-95dmf\" (UID: \"c60aecef-b09c-4447-9baf-fd4d5b90614a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf"
Sep 16 17:08:12.384348 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:12.384351    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c60aecef-b09c-4447-9baf-fd4d5b90614a-model-cache\") pod \"prestop-hook-test-kserve-7f5c764488-95dmf\" (UID: \"c60aecef-b09c-4447-9baf-fd4d5b90614a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf"
Sep 16 17:08:12.384545 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:12.384369    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c60aecef-b09c-4447-9baf-fd4d5b90614a-kserve-provision-location\") pod \"prestop-hook-test-kserve-7f5c764488-95dmf\" (UID: \"c60aecef-b09c-4447-9baf-fd4d5b90614a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf"
Sep 16 17:08:12.384545 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:12.384480    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c60aecef-b09c-4447-9baf-fd4d5b90614a-home\") pod \"prestop-hook-test-kserve-7f5c764488-95dmf\" (UID: \"c60aecef-b09c-4447-9baf-fd4d5b90614a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf"
Sep 16 17:08:12.384545 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:12.384517    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c60aecef-b09c-4447-9baf-fd4d5b90614a-tls-certs\") pod \"prestop-hook-test-kserve-7f5c764488-95dmf\" (UID: \"c60aecef-b09c-4447-9baf-fd4d5b90614a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf"
Sep 16 17:08:12.384674 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:12.384565    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c60aecef-b09c-4447-9baf-fd4d5b90614a-cabundle-cert\") pod \"prestop-hook-test-kserve-7f5c764488-95dmf\" (UID: \"c60aecef-b09c-4447-9baf-fd4d5b90614a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf"
Sep 16 17:08:12.384674 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:12.384611    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c60aecef-b09c-4447-9baf-fd4d5b90614a-tmp-dir\") pod \"prestop-hook-test-kserve-7f5c764488-95dmf\" (UID: \"c60aecef-b09c-4447-9baf-fd4d5b90614a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf"
Sep 16 17:08:12.384782 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:12.384713    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c60aecef-b09c-4447-9baf-fd4d5b90614a-model-cache\") pod \"prestop-hook-test-kserve-7f5c764488-95dmf\" (UID: \"c60aecef-b09c-4447-9baf-fd4d5b90614a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf"
Sep 16 17:08:12.384782 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:12.384762    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pllpd\" (UniqueName: \"kubernetes.io/projected/c60aecef-b09c-4447-9baf-fd4d5b90614a-kube-api-access-pllpd\") pod \"prestop-hook-test-kserve-7f5c764488-95dmf\" (UID: \"c60aecef-b09c-4447-9baf-fd4d5b90614a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf"
Sep 16 17:08:12.384999 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:12.384954    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c60aecef-b09c-4447-9baf-fd4d5b90614a-home\") pod \"prestop-hook-test-kserve-7f5c764488-95dmf\" (UID: \"c60aecef-b09c-4447-9baf-fd4d5b90614a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf"
Sep 16 17:08:12.385186 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:12.385001    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c60aecef-b09c-4447-9baf-fd4d5b90614a-tmp-dir\") pod \"prestop-hook-test-kserve-7f5c764488-95dmf\" (UID: \"c60aecef-b09c-4447-9baf-fd4d5b90614a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf"
Sep 16 17:08:12.385186 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:12.385161    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c60aecef-b09c-4447-9baf-fd4d5b90614a-kserve-provision-location\") pod \"prestop-hook-test-kserve-7f5c764488-95dmf\" (UID: \"c60aecef-b09c-4447-9baf-fd4d5b90614a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf"
Sep 16 17:08:12.385332 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:12.385312    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c60aecef-b09c-4447-9baf-fd4d5b90614a-cabundle-cert\") pod \"prestop-hook-test-kserve-7f5c764488-95dmf\" (UID: \"c60aecef-b09c-4447-9baf-fd4d5b90614a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf"
Sep 16 17:08:12.386724 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:12.386705    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c60aecef-b09c-4447-9baf-fd4d5b90614a-dshm\") pod \"prestop-hook-test-kserve-7f5c764488-95dmf\" (UID: \"c60aecef-b09c-4447-9baf-fd4d5b90614a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf"
Sep 16 17:08:12.387018 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:12.387001    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c60aecef-b09c-4447-9baf-fd4d5b90614a-tls-certs\") pod \"prestop-hook-test-kserve-7f5c764488-95dmf\" (UID: \"c60aecef-b09c-4447-9baf-fd4d5b90614a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf"
Sep 16 17:08:12.393627 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:12.393605    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pllpd\" (UniqueName: \"kubernetes.io/projected/c60aecef-b09c-4447-9baf-fd4d5b90614a-kube-api-access-pllpd\") pod \"prestop-hook-test-kserve-7f5c764488-95dmf\" (UID: \"c60aecef-b09c-4447-9baf-fd4d5b90614a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf"
Sep 16 17:08:12.470806 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:12.470765    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf"
Sep 16 17:08:12.745820 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:12.745718    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf"]
Sep 16 17:08:12.748523 ip-10-0-136-228 kubenswrapper[2554]: W0916 17:08:12.748492    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc60aecef_b09c_4447_9baf_fd4d5b90614a.slice/crio-a46c6f2164b458ee54118b40f93e8110f7559c8306bba3d42cd3926a5b89c0f5 WatchSource:0}: Error finding container a46c6f2164b458ee54118b40f93e8110f7559c8306bba3d42cd3926a5b89c0f5: Status 404 returned error can't find the container with id a46c6f2164b458ee54118b40f93e8110f7559c8306bba3d42cd3926a5b89c0f5
Sep 16 17:08:12.866274 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:12.866229    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf" event={"ID":"c60aecef-b09c-4447-9baf-fd4d5b90614a","Type":"ContainerStarted","Data":"5538d5219c7f7cd6488840bf467bde5ef128db24c5241031de3226a2a6534e8c"}
Sep 16 17:08:12.866411 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:12.866295    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf" event={"ID":"c60aecef-b09c-4447-9baf-fd4d5b90614a","Type":"ContainerStarted","Data":"a46c6f2164b458ee54118b40f93e8110f7559c8306bba3d42cd3926a5b89c0f5"}
Sep 16 17:08:14.016840 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:14.016780    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:08:15.881008 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:15.880975    2554 generic.go:356] "Generic (PLEG): container finished" podID="c60aecef-b09c-4447-9baf-fd4d5b90614a" containerID="5538d5219c7f7cd6488840bf467bde5ef128db24c5241031de3226a2a6534e8c" exitCode=0
Sep 16 17:08:15.881443 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:15.881026    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf" event={"ID":"c60aecef-b09c-4447-9baf-fd4d5b90614a","Type":"ContainerDied","Data":"5538d5219c7f7cd6488840bf467bde5ef128db24c5241031de3226a2a6534e8c"}
Sep 16 17:08:16.886942 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:16.886907    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf" event={"ID":"c60aecef-b09c-4447-9baf-fd4d5b90614a","Type":"ContainerStarted","Data":"8b382867d6254b024a594e6801fd46333968a07dcc2eadaee4521dd9b5de7944"}
Sep 16 17:08:16.911524 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:16.911482    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf" podStartSLOduration=2.8137765359999998 podStartE2EDuration="4.911468131s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.097691595s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:08:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:08:16.909244691 +0000 UTC m=+4697.353936889" watchObservedRunningTime="2026-09-16 17:08:16.911468131 +0000 UTC m=+4697.356160349"
Sep 16 17:08:17.471411 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:17.471370    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf"
Sep 16 17:08:19.960101 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:19.960069    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:08:22.471091 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:22.471052    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf"
Sep 16 17:08:22.472716 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:22.472685    2554 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf" podUID="c60aecef-b09c-4447-9baf-fd4d5b90614a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Sep 16 17:08:27.106120 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:27.106043    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578" podUID="1b314208-e5a0-444b-9ac1-987ba23296ea" containerName="main" containerID="cri-o://082bc2b7a3dc9191b645f4c41af80b7240e5aef4d2e07632818288cd0467304f" gracePeriod=45
Sep 16 17:08:27.310939 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:27.310893    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578" podUID="1b314208-e5a0-444b-9ac1-987ba23296ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 16 17:08:27.850896 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:27.850869    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578"
Sep 16 17:08:27.898177 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:27.898106    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b314208-e5a0-444b-9ac1-987ba23296ea-kserve-provision-location\") pod \"1b314208-e5a0-444b-9ac1-987ba23296ea\" (UID: \"1b314208-e5a0-444b-9ac1-987ba23296ea\") "
Sep 16 17:08:27.898177 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:27.898168    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gc2k\" (UniqueName: \"kubernetes.io/projected/1b314208-e5a0-444b-9ac1-987ba23296ea-kube-api-access-2gc2k\") pod \"1b314208-e5a0-444b-9ac1-987ba23296ea\" (UID: \"1b314208-e5a0-444b-9ac1-987ba23296ea\") "
Sep 16 17:08:27.898382 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:27.898199    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b314208-e5a0-444b-9ac1-987ba23296ea-dshm\") pod \"1b314208-e5a0-444b-9ac1-987ba23296ea\" (UID: \"1b314208-e5a0-444b-9ac1-987ba23296ea\") "
Sep 16 17:08:27.898382 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:27.898226    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b314208-e5a0-444b-9ac1-987ba23296ea-tls-certs\") pod \"1b314208-e5a0-444b-9ac1-987ba23296ea\" (UID: \"1b314208-e5a0-444b-9ac1-987ba23296ea\") "
Sep 16 17:08:27.898382 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:27.898253    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b314208-e5a0-444b-9ac1-987ba23296ea-tmp-dir\") pod \"1b314208-e5a0-444b-9ac1-987ba23296ea\" (UID: \"1b314208-e5a0-444b-9ac1-987ba23296ea\") "
Sep 16 17:08:27.898382 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:27.898280    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b314208-e5a0-444b-9ac1-987ba23296ea-cabundle-cert\") pod \"1b314208-e5a0-444b-9ac1-987ba23296ea\" (UID: \"1b314208-e5a0-444b-9ac1-987ba23296ea\") "
Sep 16 17:08:27.898382 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:27.898338    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b314208-e5a0-444b-9ac1-987ba23296ea-home\") pod \"1b314208-e5a0-444b-9ac1-987ba23296ea\" (UID: \"1b314208-e5a0-444b-9ac1-987ba23296ea\") "
Sep 16 17:08:27.898382 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:27.898367    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b314208-e5a0-444b-9ac1-987ba23296ea-model-cache\") pod \"1b314208-e5a0-444b-9ac1-987ba23296ea\" (UID: \"1b314208-e5a0-444b-9ac1-987ba23296ea\") "
Sep 16 17:08:27.898865 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:27.898805    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b314208-e5a0-444b-9ac1-987ba23296ea-model-cache" (OuterVolumeSpecName: "model-cache") pod "1b314208-e5a0-444b-9ac1-987ba23296ea" (UID: "1b314208-e5a0-444b-9ac1-987ba23296ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:08:27.898997 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:27.898968    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b314208-e5a0-444b-9ac1-987ba23296ea-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1b314208-e5a0-444b-9ac1-987ba23296ea" (UID: "1b314208-e5a0-444b-9ac1-987ba23296ea"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:08:27.899686 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:27.899645    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b314208-e5a0-444b-9ac1-987ba23296ea-home" (OuterVolumeSpecName: "home") pod "1b314208-e5a0-444b-9ac1-987ba23296ea" (UID: "1b314208-e5a0-444b-9ac1-987ba23296ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:08:27.900932 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:27.900904    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b314208-e5a0-444b-9ac1-987ba23296ea-kube-api-access-2gc2k" (OuterVolumeSpecName: "kube-api-access-2gc2k") pod "1b314208-e5a0-444b-9ac1-987ba23296ea" (UID: "1b314208-e5a0-444b-9ac1-987ba23296ea"). InnerVolumeSpecName "kube-api-access-2gc2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:08:27.901373 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:27.901350    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b314208-e5a0-444b-9ac1-987ba23296ea-dshm" (OuterVolumeSpecName: "dshm") pod "1b314208-e5a0-444b-9ac1-987ba23296ea" (UID: "1b314208-e5a0-444b-9ac1-987ba23296ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:08:27.901563 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:27.901534    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b314208-e5a0-444b-9ac1-987ba23296ea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1b314208-e5a0-444b-9ac1-987ba23296ea" (UID: "1b314208-e5a0-444b-9ac1-987ba23296ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:08:27.912730 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:27.912696    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b314208-e5a0-444b-9ac1-987ba23296ea-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1b314208-e5a0-444b-9ac1-987ba23296ea" (UID: "1b314208-e5a0-444b-9ac1-987ba23296ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:08:27.924222 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:27.924194    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b314208-e5a0-444b-9ac1-987ba23296ea-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1b314208-e5a0-444b-9ac1-987ba23296ea" (UID: "1b314208-e5a0-444b-9ac1-987ba23296ea"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:08:27.937736 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:27.937711    2554 generic.go:356] "Generic (PLEG): container finished" podID="1b314208-e5a0-444b-9ac1-987ba23296ea" containerID="082bc2b7a3dc9191b645f4c41af80b7240e5aef4d2e07632818288cd0467304f" exitCode=0
Sep 16 17:08:27.937877 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:27.937775    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578"
Sep 16 17:08:27.937877 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:27.937811    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578" event={"ID":"1b314208-e5a0-444b-9ac1-987ba23296ea","Type":"ContainerDied","Data":"082bc2b7a3dc9191b645f4c41af80b7240e5aef4d2e07632818288cd0467304f"}
Sep 16 17:08:27.937877 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:27.937867    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578" event={"ID":"1b314208-e5a0-444b-9ac1-987ba23296ea","Type":"ContainerDied","Data":"1330b18277a4fdf70d2d18cca8c280409bbb34968ff52d662323743310696563"}
Sep 16 17:08:27.938007 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:27.937891    2554 scope.go:117] "RemoveContainer" containerID="082bc2b7a3dc9191b645f4c41af80b7240e5aef4d2e07632818288cd0467304f"
Sep 16 17:08:27.946011 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:27.945992    2554 scope.go:117] "RemoveContainer" containerID="d1933441ad4479d06bfbcd261c39d0d7fd185554cfc404117399a69cedb0cc80"
Sep 16 17:08:27.953788 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:27.953762    2554 scope.go:117] "RemoveContainer" containerID="082bc2b7a3dc9191b645f4c41af80b7240e5aef4d2e07632818288cd0467304f"
Sep 16 17:08:27.954108 ip-10-0-136-228 kubenswrapper[2554]: E0916 17:08:27.954090    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"082bc2b7a3dc9191b645f4c41af80b7240e5aef4d2e07632818288cd0467304f\": container with ID starting with 082bc2b7a3dc9191b645f4c41af80b7240e5aef4d2e07632818288cd0467304f not found: ID does not exist" containerID="082bc2b7a3dc9191b645f4c41af80b7240e5aef4d2e07632818288cd0467304f"
Sep 16 17:08:27.954163 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:27.954118    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"082bc2b7a3dc9191b645f4c41af80b7240e5aef4d2e07632818288cd0467304f"} err="failed to get container status \"082bc2b7a3dc9191b645f4c41af80b7240e5aef4d2e07632818288cd0467304f\": rpc error: code = NotFound desc = could not find container \"082bc2b7a3dc9191b645f4c41af80b7240e5aef4d2e07632818288cd0467304f\": container with ID starting with 082bc2b7a3dc9191b645f4c41af80b7240e5aef4d2e07632818288cd0467304f not found: ID does not exist"
Sep 16 17:08:27.954163 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:27.954136    2554 scope.go:117] "RemoveContainer" containerID="d1933441ad4479d06bfbcd261c39d0d7fd185554cfc404117399a69cedb0cc80"
Sep 16 17:08:27.954377 ip-10-0-136-228 kubenswrapper[2554]: E0916 17:08:27.954361    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1933441ad4479d06bfbcd261c39d0d7fd185554cfc404117399a69cedb0cc80\": container with ID starting with d1933441ad4479d06bfbcd261c39d0d7fd185554cfc404117399a69cedb0cc80 not found: ID does not exist" containerID="d1933441ad4479d06bfbcd261c39d0d7fd185554cfc404117399a69cedb0cc80"
Sep 16 17:08:27.954422 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:27.954383    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1933441ad4479d06bfbcd261c39d0d7fd185554cfc404117399a69cedb0cc80"} err="failed to get container status \"d1933441ad4479d06bfbcd261c39d0d7fd185554cfc404117399a69cedb0cc80\": rpc error: code = NotFound desc = could not find container \"d1933441ad4479d06bfbcd261c39d0d7fd185554cfc404117399a69cedb0cc80\": container with ID starting with d1933441ad4479d06bfbcd261c39d0d7fd185554cfc404117399a69cedb0cc80 not found: ID does not exist"
Sep 16 17:08:27.964890 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:27.964865    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578"]
Sep 16 17:08:27.967772 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:27.967750    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-g7578"]
Sep 16 17:08:27.998828 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:27.998807    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b314208-e5a0-444b-9ac1-987ba23296ea-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:08:27.998828 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:27.998826    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gc2k\" (UniqueName: \"kubernetes.io/projected/1b314208-e5a0-444b-9ac1-987ba23296ea-kube-api-access-2gc2k\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:08:27.998946 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:27.998837    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b314208-e5a0-444b-9ac1-987ba23296ea-dshm\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:08:27.998946 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:27.998847    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b314208-e5a0-444b-9ac1-987ba23296ea-tls-certs\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:08:27.998946 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:27.998855    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b314208-e5a0-444b-9ac1-987ba23296ea-tmp-dir\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:08:27.998946 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:27.998863    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b314208-e5a0-444b-9ac1-987ba23296ea-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:08:27.998946 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:27.998870    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b314208-e5a0-444b-9ac1-987ba23296ea-home\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:08:27.998946 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:27.998877    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b314208-e5a0-444b-9ac1-987ba23296ea-model-cache\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:08:28.076573 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:28.076539    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b314208-e5a0-444b-9ac1-987ba23296ea" path="/var/lib/kubelet/pods/1b314208-e5a0-444b-9ac1-987ba23296ea/volumes"
Sep 16 17:08:28.622676 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:28.622619    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:08:28.637458 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:28.637428    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf"]
Sep 16 17:08:28.721837 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:28.721789    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:08:33.258304 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:33.258271    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:08:34.842317 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:34.842278    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:08:34.943675 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:34.943635    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:08:43.659669 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:08:43.659624    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf" podUID="c60aecef-b09c-4447-9baf-fd4d5b90614a" containerName="main" containerID="cri-o://8b382867d6254b024a594e6801fd46333968a07dcc2eadaee4521dd9b5de7944" gracePeriod=45
Sep 16 17:09:29.191856 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:29.191831    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-7f5c764488-95dmf_c60aecef-b09c-4447-9baf-fd4d5b90614a/main/0.log"
Sep 16 17:09:29.192328 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:29.192305    2554 generic.go:356] "Generic (PLEG): container finished" podID="c60aecef-b09c-4447-9baf-fd4d5b90614a" containerID="8b382867d6254b024a594e6801fd46333968a07dcc2eadaee4521dd9b5de7944" exitCode=137
Sep 16 17:09:29.192383 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:29.192355    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf" event={"ID":"c60aecef-b09c-4447-9baf-fd4d5b90614a","Type":"ContainerDied","Data":"8b382867d6254b024a594e6801fd46333968a07dcc2eadaee4521dd9b5de7944"}
Sep 16 17:09:29.308725 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:29.308701    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-7f5c764488-95dmf_c60aecef-b09c-4447-9baf-fd4d5b90614a/main/0.log"
Sep 16 17:09:29.310036 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:29.310016    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf"
Sep 16 17:09:29.375125 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:29.375093    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c60aecef-b09c-4447-9baf-fd4d5b90614a-tls-certs\") pod \"c60aecef-b09c-4447-9baf-fd4d5b90614a\" (UID: \"c60aecef-b09c-4447-9baf-fd4d5b90614a\") "
Sep 16 17:09:29.375276 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:29.375136    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c60aecef-b09c-4447-9baf-fd4d5b90614a-tmp-dir\") pod \"c60aecef-b09c-4447-9baf-fd4d5b90614a\" (UID: \"c60aecef-b09c-4447-9baf-fd4d5b90614a\") "
Sep 16 17:09:29.375276 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:29.375160    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c60aecef-b09c-4447-9baf-fd4d5b90614a-home\") pod \"c60aecef-b09c-4447-9baf-fd4d5b90614a\" (UID: \"c60aecef-b09c-4447-9baf-fd4d5b90614a\") "
Sep 16 17:09:29.375276 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:29.375187    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c60aecef-b09c-4447-9baf-fd4d5b90614a-kserve-provision-location\") pod \"c60aecef-b09c-4447-9baf-fd4d5b90614a\" (UID: \"c60aecef-b09c-4447-9baf-fd4d5b90614a\") "
Sep 16 17:09:29.375276 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:29.375210    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c60aecef-b09c-4447-9baf-fd4d5b90614a-model-cache\") pod \"c60aecef-b09c-4447-9baf-fd4d5b90614a\" (UID: \"c60aecef-b09c-4447-9baf-fd4d5b90614a\") "
Sep 16 17:09:29.375276 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:29.375238    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c60aecef-b09c-4447-9baf-fd4d5b90614a-cabundle-cert\") pod \"c60aecef-b09c-4447-9baf-fd4d5b90614a\" (UID: \"c60aecef-b09c-4447-9baf-fd4d5b90614a\") "
Sep 16 17:09:29.375276 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:29.375265    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pllpd\" (UniqueName: \"kubernetes.io/projected/c60aecef-b09c-4447-9baf-fd4d5b90614a-kube-api-access-pllpd\") pod \"c60aecef-b09c-4447-9baf-fd4d5b90614a\" (UID: \"c60aecef-b09c-4447-9baf-fd4d5b90614a\") "
Sep 16 17:09:29.375617 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:29.375307    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c60aecef-b09c-4447-9baf-fd4d5b90614a-dshm\") pod \"c60aecef-b09c-4447-9baf-fd4d5b90614a\" (UID: \"c60aecef-b09c-4447-9baf-fd4d5b90614a\") "
Sep 16 17:09:29.375617 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:29.375523    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c60aecef-b09c-4447-9baf-fd4d5b90614a-model-cache" (OuterVolumeSpecName: "model-cache") pod "c60aecef-b09c-4447-9baf-fd4d5b90614a" (UID: "c60aecef-b09c-4447-9baf-fd4d5b90614a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:09:29.375617 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:29.375601    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c60aecef-b09c-4447-9baf-fd4d5b90614a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c60aecef-b09c-4447-9baf-fd4d5b90614a" (UID: "c60aecef-b09c-4447-9baf-fd4d5b90614a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:09:29.375773 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:29.375614    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c60aecef-b09c-4447-9baf-fd4d5b90614a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c60aecef-b09c-4447-9baf-fd4d5b90614a" (UID: "c60aecef-b09c-4447-9baf-fd4d5b90614a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:09:29.375885 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:29.375857    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c60aecef-b09c-4447-9baf-fd4d5b90614a-home" (OuterVolumeSpecName: "home") pod "c60aecef-b09c-4447-9baf-fd4d5b90614a" (UID: "c60aecef-b09c-4447-9baf-fd4d5b90614a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:09:29.377531 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:29.377507    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c60aecef-b09c-4447-9baf-fd4d5b90614a-dshm" (OuterVolumeSpecName: "dshm") pod "c60aecef-b09c-4447-9baf-fd4d5b90614a" (UID: "c60aecef-b09c-4447-9baf-fd4d5b90614a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:09:29.377531 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:29.377506    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c60aecef-b09c-4447-9baf-fd4d5b90614a-kube-api-access-pllpd" (OuterVolumeSpecName: "kube-api-access-pllpd") pod "c60aecef-b09c-4447-9baf-fd4d5b90614a" (UID: "c60aecef-b09c-4447-9baf-fd4d5b90614a"). InnerVolumeSpecName "kube-api-access-pllpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:09:29.377669 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:29.377540    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c60aecef-b09c-4447-9baf-fd4d5b90614a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c60aecef-b09c-4447-9baf-fd4d5b90614a" (UID: "c60aecef-b09c-4447-9baf-fd4d5b90614a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:09:29.395847 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:29.395822    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c60aecef-b09c-4447-9baf-fd4d5b90614a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c60aecef-b09c-4447-9baf-fd4d5b90614a" (UID: "c60aecef-b09c-4447-9baf-fd4d5b90614a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:09:29.476657 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:29.476578    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c60aecef-b09c-4447-9baf-fd4d5b90614a-tls-certs\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:09:29.476657 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:29.476603    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c60aecef-b09c-4447-9baf-fd4d5b90614a-tmp-dir\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:09:29.476657 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:29.476614    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c60aecef-b09c-4447-9baf-fd4d5b90614a-home\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:09:29.476657 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:29.476626    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c60aecef-b09c-4447-9baf-fd4d5b90614a-kserve-provision-location\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:09:29.476657 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:29.476638    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c60aecef-b09c-4447-9baf-fd4d5b90614a-model-cache\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:09:29.476657 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:29.476651    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c60aecef-b09c-4447-9baf-fd4d5b90614a-cabundle-cert\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:09:29.476957 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:29.476664    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pllpd\" (UniqueName: \"kubernetes.io/projected/c60aecef-b09c-4447-9baf-fd4d5b90614a-kube-api-access-pllpd\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:09:29.476957 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:29.476675    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c60aecef-b09c-4447-9baf-fd4d5b90614a-dshm\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:09:30.197271 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:30.197244    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-7f5c764488-95dmf_c60aecef-b09c-4447-9baf-fd4d5b90614a/main/0.log"
Sep 16 17:09:30.197670 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:30.197645    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf" event={"ID":"c60aecef-b09c-4447-9baf-fd4d5b90614a","Type":"ContainerDied","Data":"a46c6f2164b458ee54118b40f93e8110f7559c8306bba3d42cd3926a5b89c0f5"}
Sep 16 17:09:30.197740 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:30.197701    2554 scope.go:117] "RemoveContainer" containerID="8b382867d6254b024a594e6801fd46333968a07dcc2eadaee4521dd9b5de7944"
Sep 16 17:09:30.197740 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:30.197707    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf"
Sep 16 17:09:30.205915 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:30.205900    2554 scope.go:117] "RemoveContainer" containerID="5538d5219c7f7cd6488840bf467bde5ef128db24c5241031de3226a2a6534e8c"
Sep 16 17:09:30.218521 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:30.218496    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf"]
Sep 16 17:09:30.221883 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:30.221861    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f5c764488-95dmf"]
Sep 16 17:09:32.075887 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:32.075854    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c60aecef-b09c-4447-9baf-fd4d5b90614a" path="/var/lib/kubelet/pods/c60aecef-b09c-4447-9baf-fd4d5b90614a/volumes"
Sep 16 17:09:33.593780 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:33.593746    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:09:33.690345 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:33.690294    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:09:39.904646 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:39.904615    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:09:43.871255 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:43.871217    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-55bdc"]
Sep 16 17:09:43.871617 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:43.871473    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b314208-e5a0-444b-9ac1-987ba23296ea" containerName="storage-initializer"
Sep 16 17:09:43.871617 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:43.871485    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b314208-e5a0-444b-9ac1-987ba23296ea" containerName="storage-initializer"
Sep 16 17:09:43.871617 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:43.871495    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c60aecef-b09c-4447-9baf-fd4d5b90614a" containerName="main"
Sep 16 17:09:43.871617 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:43.871500    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c60aecef-b09c-4447-9baf-fd4d5b90614a" containerName="main"
Sep 16 17:09:43.871617 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:43.871515    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b314208-e5a0-444b-9ac1-987ba23296ea" containerName="main"
Sep 16 17:09:43.871617 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:43.871521    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b314208-e5a0-444b-9ac1-987ba23296ea" containerName="main"
Sep 16 17:09:43.871617 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:43.871527    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c60aecef-b09c-4447-9baf-fd4d5b90614a" containerName="storage-initializer"
Sep 16 17:09:43.871617 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:43.871533    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c60aecef-b09c-4447-9baf-fd4d5b90614a" containerName="storage-initializer"
Sep 16 17:09:43.871617 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:43.871572    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c60aecef-b09c-4447-9baf-fd4d5b90614a" containerName="main"
Sep 16 17:09:43.871617 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:43.871579    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b314208-e5a0-444b-9ac1-987ba23296ea" containerName="main"
Sep 16 17:09:43.874290 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:43.874268    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-55bdc"
Sep 16 17:09:43.878365 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:43.878339    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 16 17:09:43.878481 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:43.878368    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 16 17:09:43.878481 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:43.878340    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-jbn9c\""
Sep 16 17:09:43.878481 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:43.878417    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 16 17:09:43.885568 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:43.885541    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-55bdc"]
Sep 16 17:09:43.905371 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:43.905344    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:09:43.973148 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:43.973117    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c833be02-a6dd-4e99-b7ac-d0e6cfc232c0-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-55bdc\" (UID: \"c833be02-a6dd-4e99-b7ac-d0e6cfc232c0\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-55bdc"
Sep 16 17:09:43.973315 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:43.973166    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c833be02-a6dd-4e99-b7ac-d0e6cfc232c0-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-55bdc\" (UID: \"c833be02-a6dd-4e99-b7ac-d0e6cfc232c0\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-55bdc"
Sep 16 17:09:43.973315 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:43.973230    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wk7p\" (UniqueName: \"kubernetes.io/projected/c833be02-a6dd-4e99-b7ac-d0e6cfc232c0-kube-api-access-5wk7p\") pod \"rolling-upgrade-test-kserve-74b79d7f77-55bdc\" (UID: \"c833be02-a6dd-4e99-b7ac-d0e6cfc232c0\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-55bdc"
Sep 16 17:09:43.973315 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:43.973282    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c833be02-a6dd-4e99-b7ac-d0e6cfc232c0-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-55bdc\" (UID: \"c833be02-a6dd-4e99-b7ac-d0e6cfc232c0\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-55bdc"
Sep 16 17:09:43.973315 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:43.973306    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c833be02-a6dd-4e99-b7ac-d0e6cfc232c0-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-55bdc\" (UID: \"c833be02-a6dd-4e99-b7ac-d0e6cfc232c0\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-55bdc"
Sep 16 17:09:43.973453 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:43.973328    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c833be02-a6dd-4e99-b7ac-d0e6cfc232c0-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-55bdc\" (UID: \"c833be02-a6dd-4e99-b7ac-d0e6cfc232c0\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-55bdc"
Sep 16 17:09:44.073724 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:44.073687    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c833be02-a6dd-4e99-b7ac-d0e6cfc232c0-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-55bdc\" (UID: \"c833be02-a6dd-4e99-b7ac-d0e6cfc232c0\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-55bdc"
Sep 16 17:09:44.073939 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:44.073731    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wk7p\" (UniqueName: \"kubernetes.io/projected/c833be02-a6dd-4e99-b7ac-d0e6cfc232c0-kube-api-access-5wk7p\") pod \"rolling-upgrade-test-kserve-74b79d7f77-55bdc\" (UID: \"c833be02-a6dd-4e99-b7ac-d0e6cfc232c0\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-55bdc"
Sep 16 17:09:44.073939 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:44.073754    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c833be02-a6dd-4e99-b7ac-d0e6cfc232c0-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-55bdc\" (UID: \"c833be02-a6dd-4e99-b7ac-d0e6cfc232c0\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-55bdc"
Sep 16 17:09:44.073939 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:44.073775    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c833be02-a6dd-4e99-b7ac-d0e6cfc232c0-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-55bdc\" (UID: \"c833be02-a6dd-4e99-b7ac-d0e6cfc232c0\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-55bdc"
Sep 16 17:09:44.074132 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:44.073933    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c833be02-a6dd-4e99-b7ac-d0e6cfc232c0-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-55bdc\" (UID: \"c833be02-a6dd-4e99-b7ac-d0e6cfc232c0\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-55bdc"
Sep 16 17:09:44.074132 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:44.074024    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c833be02-a6dd-4e99-b7ac-d0e6cfc232c0-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-55bdc\" (UID: \"c833be02-a6dd-4e99-b7ac-d0e6cfc232c0\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-55bdc"
Sep 16 17:09:44.074245 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:44.074171    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c833be02-a6dd-4e99-b7ac-d0e6cfc232c0-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-55bdc\" (UID: \"c833be02-a6dd-4e99-b7ac-d0e6cfc232c0\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-55bdc"
Sep 16 17:09:44.074304 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:44.074289    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c833be02-a6dd-4e99-b7ac-d0e6cfc232c0-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-55bdc\" (UID: \"c833be02-a6dd-4e99-b7ac-d0e6cfc232c0\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-55bdc"
Sep 16 17:09:44.074402 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:44.074354    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c833be02-a6dd-4e99-b7ac-d0e6cfc232c0-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-55bdc\" (UID: \"c833be02-a6dd-4e99-b7ac-d0e6cfc232c0\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-55bdc"
Sep 16 17:09:44.076257 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:44.076238    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c833be02-a6dd-4e99-b7ac-d0e6cfc232c0-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-55bdc\" (UID: \"c833be02-a6dd-4e99-b7ac-d0e6cfc232c0\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-55bdc"
Sep 16 17:09:44.076619 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:44.076597    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c833be02-a6dd-4e99-b7ac-d0e6cfc232c0-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-55bdc\" (UID: \"c833be02-a6dd-4e99-b7ac-d0e6cfc232c0\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-55bdc"
Sep 16 17:09:44.082041 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:44.082018    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wk7p\" (UniqueName: \"kubernetes.io/projected/c833be02-a6dd-4e99-b7ac-d0e6cfc232c0-kube-api-access-5wk7p\") pod \"rolling-upgrade-test-kserve-74b79d7f77-55bdc\" (UID: \"c833be02-a6dd-4e99-b7ac-d0e6cfc232c0\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-55bdc"
Sep 16 17:09:44.154429 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:44.154355    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:09:44.184405 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:44.184380    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-55bdc"
Sep 16 17:09:44.451691 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:44.451614    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-55bdc"]
Sep 16 17:09:44.456296 ip-10-0-136-228 kubenswrapper[2554]: W0916 17:09:44.456228    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc833be02_a6dd_4e99_b7ac_d0e6cfc232c0.slice/crio-5d8082019cf267732882d0e477e3a6e4a67c1a791765214be0cc17417b25b05c WatchSource:0}: Error finding container 5d8082019cf267732882d0e477e3a6e4a67c1a791765214be0cc17417b25b05c: Status 404 returned error can't find the container with id 5d8082019cf267732882d0e477e3a6e4a67c1a791765214be0cc17417b25b05c
Sep 16 17:09:45.252231 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:45.252200    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-55bdc" event={"ID":"c833be02-a6dd-4e99-b7ac-d0e6cfc232c0","Type":"ContainerStarted","Data":"c01137e4d9741439b8e658c0a446144fa51ed297183de40989211145d3d46826"}
Sep 16 17:09:45.252595 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:45.252248    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-55bdc" event={"ID":"c833be02-a6dd-4e99-b7ac-d0e6cfc232c0","Type":"ContainerStarted","Data":"5d8082019cf267732882d0e477e3a6e4a67c1a791765214be0cc17417b25b05c"}
Sep 16 17:09:45.268561 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:45.268514    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-55bdc" podStartSLOduration=2.268499789 podStartE2EDuration="2.268499789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:09:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:09:45.267759989 +0000 UTC m=+4785.712452184" watchObservedRunningTime="2026-09-16 17:09:45.268499789 +0000 UTC m=+4785.713191966"
Sep 16 17:09:46.184902 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:46.184861    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-55bdc"
Sep 16 17:09:46.558077 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:46.558048    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:09:46.657099 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:46.657071    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:09:53.686406 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:53.686366    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:09:54.185082 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:54.185028    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-55bdc"
Sep 16 17:09:54.197317 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:54.197289    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-55bdc"
Sep 16 17:09:54.298933 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:54.298906    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-55bdc"
Sep 16 17:09:58.632424 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:09:58.632390    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:10:18.684075 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:10:18.684042    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:10:24.561982 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:10:24.561949    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:10:26.903441 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:10:26.903408    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-55bdc"]
Sep 16 17:10:26.913954 ip-10-0-136-228 kubenswrapper[2554]: E0916 17:10:26.913919    2554 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 17:10:26.913954 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:10:26.913954 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 17:10:26.913954 ip-10-0-136-228 kubenswrapper[2554]:  > containerID="c01137e4d9741439b8e658c0a446144fa51ed297183de40989211145d3d46826" cmd=["/bin/sleep","15"]
Sep 16 17:10:26.914262 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:10:26.913975    2554 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-55bdc" message="" err=<
Sep 16 17:10:26.914262 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:10:26.914262 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 17:10:26.914262 ip-10-0-136-228 kubenswrapper[2554]:  >
Sep 16 17:10:26.914262 ip-10-0-136-228 kubenswrapper[2554]: E0916 17:10:26.913992    2554 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 17:10:26.914262 ip-10-0-136-228 kubenswrapper[2554]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:10:26.914262 ip-10-0-136-228 kubenswrapper[2554]:         , stdout: , stderr: , exit code -1
Sep 16 17:10:26.914262 ip-10-0-136-228 kubenswrapper[2554]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-55bdc" podUID="c833be02-a6dd-4e99-b7ac-d0e6cfc232c0" containerName="main" containerID="cri-o://c01137e4d9741439b8e658c0a446144fa51ed297183de40989211145d3d46826"
Sep 16 17:10:26.914262 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:10:26.914043    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-55bdc" podUID="c833be02-a6dd-4e99-b7ac-d0e6cfc232c0" containerName="main" containerID="cri-o://c01137e4d9741439b8e658c0a446144fa51ed297183de40989211145d3d46826" gracePeriod=60
Sep 16 17:10:26.916435 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:10:26.916396    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:10:26.981325 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:10:26.981291    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:10:27.288889 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:10:27.288852    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-55bdc" podUID="c833be02-a6dd-4e99-b7ac-d0e6cfc232c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Sep 16 17:10:27.403411 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:10:27.403380    2554 generic.go:356] "Generic (PLEG): container finished" podID="c833be02-a6dd-4e99-b7ac-d0e6cfc232c0" containerID="c01137e4d9741439b8e658c0a446144fa51ed297183de40989211145d3d46826" exitCode=0
Sep 16 17:10:27.403569 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:10:27.403421    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-55bdc" event={"ID":"c833be02-a6dd-4e99-b7ac-d0e6cfc232c0","Type":"ContainerDied","Data":"c01137e4d9741439b8e658c0a446144fa51ed297183de40989211145d3d46826"}
Sep 16 17:10:27.652544 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:10:27.652520    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-55bdc"
Sep 16 17:10:27.792330 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:10:27.792303    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c833be02-a6dd-4e99-b7ac-d0e6cfc232c0-home\") pod \"c833be02-a6dd-4e99-b7ac-d0e6cfc232c0\" (UID: \"c833be02-a6dd-4e99-b7ac-d0e6cfc232c0\") "
Sep 16 17:10:27.792522 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:10:27.792345    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wk7p\" (UniqueName: \"kubernetes.io/projected/c833be02-a6dd-4e99-b7ac-d0e6cfc232c0-kube-api-access-5wk7p\") pod \"c833be02-a6dd-4e99-b7ac-d0e6cfc232c0\" (UID: \"c833be02-a6dd-4e99-b7ac-d0e6cfc232c0\") "
Sep 16 17:10:27.792522 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:10:27.792373    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c833be02-a6dd-4e99-b7ac-d0e6cfc232c0-model-cache\") pod \"c833be02-a6dd-4e99-b7ac-d0e6cfc232c0\" (UID: \"c833be02-a6dd-4e99-b7ac-d0e6cfc232c0\") "
Sep 16 17:10:27.792522 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:10:27.792397    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c833be02-a6dd-4e99-b7ac-d0e6cfc232c0-tmp-dir\") pod \"c833be02-a6dd-4e99-b7ac-d0e6cfc232c0\" (UID: \"c833be02-a6dd-4e99-b7ac-d0e6cfc232c0\") "
Sep 16 17:10:27.792522 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:10:27.792429    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c833be02-a6dd-4e99-b7ac-d0e6cfc232c0-dshm\") pod \"c833be02-a6dd-4e99-b7ac-d0e6cfc232c0\" (UID: \"c833be02-a6dd-4e99-b7ac-d0e6cfc232c0\") "
Sep 16 17:10:27.792522 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:10:27.792494    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c833be02-a6dd-4e99-b7ac-d0e6cfc232c0-tls-certs\") pod \"c833be02-a6dd-4e99-b7ac-d0e6cfc232c0\" (UID: \"c833be02-a6dd-4e99-b7ac-d0e6cfc232c0\") "
Sep 16 17:10:27.792790 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:10:27.792569    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c833be02-a6dd-4e99-b7ac-d0e6cfc232c0-home" (OuterVolumeSpecName: "home") pod "c833be02-a6dd-4e99-b7ac-d0e6cfc232c0" (UID: "c833be02-a6dd-4e99-b7ac-d0e6cfc232c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:10:27.792790 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:10:27.792682    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c833be02-a6dd-4e99-b7ac-d0e6cfc232c0-model-cache" (OuterVolumeSpecName: "model-cache") pod "c833be02-a6dd-4e99-b7ac-d0e6cfc232c0" (UID: "c833be02-a6dd-4e99-b7ac-d0e6cfc232c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:10:27.792790 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:10:27.792678    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c833be02-a6dd-4e99-b7ac-d0e6cfc232c0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c833be02-a6dd-4e99-b7ac-d0e6cfc232c0" (UID: "c833be02-a6dd-4e99-b7ac-d0e6cfc232c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:10:27.792790 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:10:27.792713    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c833be02-a6dd-4e99-b7ac-d0e6cfc232c0-home\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:10:27.794553 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:10:27.794526    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c833be02-a6dd-4e99-b7ac-d0e6cfc232c0-kube-api-access-5wk7p" (OuterVolumeSpecName: "kube-api-access-5wk7p") pod "c833be02-a6dd-4e99-b7ac-d0e6cfc232c0" (UID: "c833be02-a6dd-4e99-b7ac-d0e6cfc232c0"). InnerVolumeSpecName "kube-api-access-5wk7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:10:27.794671 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:10:27.794568    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c833be02-a6dd-4e99-b7ac-d0e6cfc232c0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c833be02-a6dd-4e99-b7ac-d0e6cfc232c0" (UID: "c833be02-a6dd-4e99-b7ac-d0e6cfc232c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:10:27.794671 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:10:27.794553    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c833be02-a6dd-4e99-b7ac-d0e6cfc232c0-dshm" (OuterVolumeSpecName: "dshm") pod "c833be02-a6dd-4e99-b7ac-d0e6cfc232c0" (UID: "c833be02-a6dd-4e99-b7ac-d0e6cfc232c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:10:27.893233 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:10:27.893161    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c833be02-a6dd-4e99-b7ac-d0e6cfc232c0-tls-certs\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:10:27.893233 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:10:27.893189    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wk7p\" (UniqueName: \"kubernetes.io/projected/c833be02-a6dd-4e99-b7ac-d0e6cfc232c0-kube-api-access-5wk7p\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:10:27.893233 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:10:27.893200    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c833be02-a6dd-4e99-b7ac-d0e6cfc232c0-model-cache\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:10:27.893233 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:10:27.893209    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c833be02-a6dd-4e99-b7ac-d0e6cfc232c0-tmp-dir\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:10:27.893233 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:10:27.893229    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c833be02-a6dd-4e99-b7ac-d0e6cfc232c0-dshm\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:10:28.408518 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:10:28.408488    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-55bdc"
Sep 16 17:10:28.408931 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:10:28.408487    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-55bdc" event={"ID":"c833be02-a6dd-4e99-b7ac-d0e6cfc232c0","Type":"ContainerDied","Data":"5d8082019cf267732882d0e477e3a6e4a67c1a791765214be0cc17417b25b05c"}
Sep 16 17:10:28.408931 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:10:28.408633    2554 scope.go:117] "RemoveContainer" containerID="c01137e4d9741439b8e658c0a446144fa51ed297183de40989211145d3d46826"
Sep 16 17:10:28.425170 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:10:28.425147    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-55bdc"]
Sep 16 17:10:28.429517 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:10:28.429491    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-55bdc"]
Sep 16 17:10:30.076022 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:10:30.075990    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c833be02-a6dd-4e99-b7ac-d0e6cfc232c0" path="/var/lib/kubelet/pods/c833be02-a6dd-4e99-b7ac-d0e6cfc232c0/volumes"
Sep 16 17:10:30.359839 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:10:30.359741    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:10:30.422498 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:10:30.422451    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:11:15.363495 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:15.363453    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:11:17.063489 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:17.063449    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kgtsj/must-gather-sn4fs"]
Sep 16 17:11:17.063877 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:17.063719    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c833be02-a6dd-4e99-b7ac-d0e6cfc232c0" containerName="main"
Sep 16 17:11:17.063877 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:17.063730    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c833be02-a6dd-4e99-b7ac-d0e6cfc232c0" containerName="main"
Sep 16 17:11:17.063877 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:17.063779    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c833be02-a6dd-4e99-b7ac-d0e6cfc232c0" containerName="main"
Sep 16 17:11:17.067752 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:17.067734    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kgtsj/must-gather-sn4fs"
Sep 16 17:11:17.072579 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:17.072556    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kgtsj\"/\"kube-root-ca.crt\""
Sep 16 17:11:17.072769 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:17.072556    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kgtsj\"/\"default-dockercfg-s9lxh\""
Sep 16 17:11:17.072916 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:17.072898    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kgtsj\"/\"openshift-service-ca.crt\""
Sep 16 17:11:17.080487 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:17.080466    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kgtsj/must-gather-sn4fs"]
Sep 16 17:11:17.097522 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:17.097492    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:11:17.142385 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:17.142356    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwc4v\" (UniqueName: \"kubernetes.io/projected/6b6cdb29-f428-40e6-ac02-73ece22cf9c3-kube-api-access-qwc4v\") pod \"must-gather-sn4fs\" (UID: \"6b6cdb29-f428-40e6-ac02-73ece22cf9c3\") " pod="openshift-must-gather-kgtsj/must-gather-sn4fs"
Sep 16 17:11:17.142522 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:17.142410    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6b6cdb29-f428-40e6-ac02-73ece22cf9c3-must-gather-output\") pod \"must-gather-sn4fs\" (UID: \"6b6cdb29-f428-40e6-ac02-73ece22cf9c3\") " pod="openshift-must-gather-kgtsj/must-gather-sn4fs"
Sep 16 17:11:17.236207 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:17.236175    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:11:17.243195 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:17.243173    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6b6cdb29-f428-40e6-ac02-73ece22cf9c3-must-gather-output\") pod \"must-gather-sn4fs\" (UID: \"6b6cdb29-f428-40e6-ac02-73ece22cf9c3\") " pod="openshift-must-gather-kgtsj/must-gather-sn4fs"
Sep 16 17:11:17.243290 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:17.243222    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwc4v\" (UniqueName: \"kubernetes.io/projected/6b6cdb29-f428-40e6-ac02-73ece22cf9c3-kube-api-access-qwc4v\") pod \"must-gather-sn4fs\" (UID: \"6b6cdb29-f428-40e6-ac02-73ece22cf9c3\") " pod="openshift-must-gather-kgtsj/must-gather-sn4fs"
Sep 16 17:11:17.243535 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:17.243516    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6b6cdb29-f428-40e6-ac02-73ece22cf9c3-must-gather-output\") pod \"must-gather-sn4fs\" (UID: \"6b6cdb29-f428-40e6-ac02-73ece22cf9c3\") " pod="openshift-must-gather-kgtsj/must-gather-sn4fs"
Sep 16 17:11:17.261039 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:17.261017    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwc4v\" (UniqueName: \"kubernetes.io/projected/6b6cdb29-f428-40e6-ac02-73ece22cf9c3-kube-api-access-qwc4v\") pod \"must-gather-sn4fs\" (UID: \"6b6cdb29-f428-40e6-ac02-73ece22cf9c3\") " pod="openshift-must-gather-kgtsj/must-gather-sn4fs"
Sep 16 17:11:17.375751 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:17.375679    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kgtsj/must-gather-sn4fs"
Sep 16 17:11:17.650280 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:17.650253    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kgtsj/must-gather-sn4fs"]
Sep 16 17:11:17.652230 ip-10-0-136-228 kubenswrapper[2554]: W0916 17:11:17.652205    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b6cdb29_f428_40e6_ac02_73ece22cf9c3.slice/crio-2eba3ebfa2e7f1dd7c0725674bb7827817c7fdc4f3ca7841d4e59327df3cda3b WatchSource:0}: Error finding container 2eba3ebfa2e7f1dd7c0725674bb7827817c7fdc4f3ca7841d4e59327df3cda3b: Status 404 returned error can't find the container with id 2eba3ebfa2e7f1dd7c0725674bb7827817c7fdc4f3ca7841d4e59327df3cda3b
Sep 16 17:11:17.654029 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:17.654011    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 17:11:18.594506 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:18.594466    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kgtsj/must-gather-sn4fs" event={"ID":"6b6cdb29-f428-40e6-ac02-73ece22cf9c3","Type":"ContainerStarted","Data":"2eba3ebfa2e7f1dd7c0725674bb7827817c7fdc4f3ca7841d4e59327df3cda3b"}
Sep 16 17:11:21.398786 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:21.398711    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:11:24.623003 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:24.622970    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kgtsj/must-gather-sn4fs" event={"ID":"6b6cdb29-f428-40e6-ac02-73ece22cf9c3","Type":"ContainerStarted","Data":"16c7e2ed46970cead0767904c9f76875c4ae6d7a0ce56fbf0bc107a94347fc96"}
Sep 16 17:11:24.623399 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:24.623015    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kgtsj/must-gather-sn4fs" event={"ID":"6b6cdb29-f428-40e6-ac02-73ece22cf9c3","Type":"ContainerStarted","Data":"cd64dd8c2d7926f3f1cab5afb383af52b5535e7099ecb6d2afe4941ba5eecfb0"}
Sep 16 17:11:24.639911 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:24.639864    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kgtsj/must-gather-sn4fs" podStartSLOduration=1.7171088129999998 podStartE2EDuration="7.639847684s" totalImagesPullingTime="5.922738871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:11:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:11:24.638349056 +0000 UTC m=+4885.083041233" watchObservedRunningTime="2026-09-16 17:11:24.639847684 +0000 UTC m=+4885.084539859"
Sep 16 17:11:27.204270 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:27.204237    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:11:27.263756 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:27.263721    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:11:35.631273 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:35.631235    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:11:41.113441 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:41.113410    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-2qjkj_93232e18-073e-43c6-b4ce-0b88d0a675d5/manager/0.log"
Sep 16 17:11:41.136895 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:41.136869    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-2cglz_5b483164-2cb4-419d-a6ff-e726301444ed/kuadrant-console-plugin/0.log"
Sep 16 17:11:41.260406 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:41.260381    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-q7kx5_fc5c5232-5455-488f-9b79-ed0048e637c0/limitador/0.log"
Sep 16 17:11:45.708937 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:45.708899    2554 generic.go:356] "Generic (PLEG): container finished" podID="6b6cdb29-f428-40e6-ac02-73ece22cf9c3" containerID="cd64dd8c2d7926f3f1cab5afb383af52b5535e7099ecb6d2afe4941ba5eecfb0" exitCode=0
Sep 16 17:11:45.709357 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:45.708968    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kgtsj/must-gather-sn4fs" event={"ID":"6b6cdb29-f428-40e6-ac02-73ece22cf9c3","Type":"ContainerDied","Data":"cd64dd8c2d7926f3f1cab5afb383af52b5535e7099ecb6d2afe4941ba5eecfb0"}
Sep 16 17:11:45.709357 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:45.709281    2554 scope.go:117] "RemoveContainer" containerID="cd64dd8c2d7926f3f1cab5afb383af52b5535e7099ecb6d2afe4941ba5eecfb0"
Sep 16 17:11:46.616156 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:46.616125    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-kgtsj_must-gather-sn4fs_6b6cdb29-f428-40e6-ac02-73ece22cf9c3/gather/0.log"
Sep 16 17:11:47.074744 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:47.074706    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:11:47.235542 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:47.235501    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:11:47.308702 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:47.308669    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:11:50.292930 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:50.292900    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wcrfk_c0d31180-c1e0-48f6-a5a2-d1eb9ae02886/global-pull-secret-syncer/0.log"
Sep 16 17:11:50.333935 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:50.333903    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log"
Sep 16 17:11:50.449142 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:50.449109    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-228.ec2.internal_c8010e8ff400f9e15027bac559fe6673/haproxy/0.log"
Sep 16 17:11:52.051420 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:52.051384    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-kgtsj/must-gather-sn4fs"]
Sep 16 17:11:52.051852 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:52.051557    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-kgtsj/must-gather-sn4fs" podUID="6b6cdb29-f428-40e6-ac02-73ece22cf9c3" containerName="copy" containerID="cri-o://16c7e2ed46970cead0767904c9f76875c4ae6d7a0ce56fbf0bc107a94347fc96" gracePeriod=2
Sep 16 17:11:52.056480 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:52.056117    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-kgtsj/must-gather-sn4fs"]
Sep 16 17:11:52.279568 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:52.279547    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-kgtsj_must-gather-sn4fs_6b6cdb29-f428-40e6-ac02-73ece22cf9c3/copy/0.log"
Sep 16 17:11:52.280941 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:52.280924    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kgtsj/must-gather-sn4fs"
Sep 16 17:11:52.431590 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:52.431498    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwc4v\" (UniqueName: \"kubernetes.io/projected/6b6cdb29-f428-40e6-ac02-73ece22cf9c3-kube-api-access-qwc4v\") pod \"6b6cdb29-f428-40e6-ac02-73ece22cf9c3\" (UID: \"6b6cdb29-f428-40e6-ac02-73ece22cf9c3\") "
Sep 16 17:11:52.431590 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:52.431554    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6b6cdb29-f428-40e6-ac02-73ece22cf9c3-must-gather-output\") pod \"6b6cdb29-f428-40e6-ac02-73ece22cf9c3\" (UID: \"6b6cdb29-f428-40e6-ac02-73ece22cf9c3\") "
Sep 16 17:11:52.433716 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:52.433682    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b6cdb29-f428-40e6-ac02-73ece22cf9c3-kube-api-access-qwc4v" (OuterVolumeSpecName: "kube-api-access-qwc4v") pod "6b6cdb29-f428-40e6-ac02-73ece22cf9c3" (UID: "6b6cdb29-f428-40e6-ac02-73ece22cf9c3"). InnerVolumeSpecName "kube-api-access-qwc4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:11:52.438192 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:52.438164    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b6cdb29-f428-40e6-ac02-73ece22cf9c3-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "6b6cdb29-f428-40e6-ac02-73ece22cf9c3" (UID: "6b6cdb29-f428-40e6-ac02-73ece22cf9c3"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:11:52.532767 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:52.532718    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwc4v\" (UniqueName: \"kubernetes.io/projected/6b6cdb29-f428-40e6-ac02-73ece22cf9c3-kube-api-access-qwc4v\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:11:52.532767 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:52.532761    2554 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6b6cdb29-f428-40e6-ac02-73ece22cf9c3-must-gather-output\") on node \"ip-10-0-136-228.ec2.internal\" DevicePath \"\""
Sep 16 17:11:52.739219 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:52.739146    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-kgtsj_must-gather-sn4fs_6b6cdb29-f428-40e6-ac02-73ece22cf9c3/copy/0.log"
Sep 16 17:11:52.739572 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:52.739543    2554 generic.go:356] "Generic (PLEG): container finished" podID="6b6cdb29-f428-40e6-ac02-73ece22cf9c3" containerID="16c7e2ed46970cead0767904c9f76875c4ae6d7a0ce56fbf0bc107a94347fc96" exitCode=143
Sep 16 17:11:52.739681 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:52.739587    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kgtsj/must-gather-sn4fs"
Sep 16 17:11:52.739681 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:52.739617    2554 scope.go:117] "RemoveContainer" containerID="16c7e2ed46970cead0767904c9f76875c4ae6d7a0ce56fbf0bc107a94347fc96"
Sep 16 17:11:52.748996 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:52.748679    2554 scope.go:117] "RemoveContainer" containerID="cd64dd8c2d7926f3f1cab5afb383af52b5535e7099ecb6d2afe4941ba5eecfb0"
Sep 16 17:11:52.760332 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:52.760315    2554 scope.go:117] "RemoveContainer" containerID="16c7e2ed46970cead0767904c9f76875c4ae6d7a0ce56fbf0bc107a94347fc96"
Sep 16 17:11:52.760582 ip-10-0-136-228 kubenswrapper[2554]: E0916 17:11:52.760563    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16c7e2ed46970cead0767904c9f76875c4ae6d7a0ce56fbf0bc107a94347fc96\": container with ID starting with 16c7e2ed46970cead0767904c9f76875c4ae6d7a0ce56fbf0bc107a94347fc96 not found: ID does not exist" containerID="16c7e2ed46970cead0767904c9f76875c4ae6d7a0ce56fbf0bc107a94347fc96"
Sep 16 17:11:52.760629 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:52.760589    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16c7e2ed46970cead0767904c9f76875c4ae6d7a0ce56fbf0bc107a94347fc96"} err="failed to get container status \"16c7e2ed46970cead0767904c9f76875c4ae6d7a0ce56fbf0bc107a94347fc96\": rpc error: code = NotFound desc = could not find container \"16c7e2ed46970cead0767904c9f76875c4ae6d7a0ce56fbf0bc107a94347fc96\": container with ID starting with 16c7e2ed46970cead0767904c9f76875c4ae6d7a0ce56fbf0bc107a94347fc96 not found: ID does not exist"
Sep 16 17:11:52.760629 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:52.760607    2554 scope.go:117] "RemoveContainer" containerID="cd64dd8c2d7926f3f1cab5afb383af52b5535e7099ecb6d2afe4941ba5eecfb0"
Sep 16 17:11:52.760866 ip-10-0-136-228 kubenswrapper[2554]: E0916 17:11:52.760848    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd64dd8c2d7926f3f1cab5afb383af52b5535e7099ecb6d2afe4941ba5eecfb0\": container with ID starting with cd64dd8c2d7926f3f1cab5afb383af52b5535e7099ecb6d2afe4941ba5eecfb0 not found: ID does not exist" containerID="cd64dd8c2d7926f3f1cab5afb383af52b5535e7099ecb6d2afe4941ba5eecfb0"
Sep 16 17:11:52.760911 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:52.760870    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd64dd8c2d7926f3f1cab5afb383af52b5535e7099ecb6d2afe4941ba5eecfb0"} err="failed to get container status \"cd64dd8c2d7926f3f1cab5afb383af52b5535e7099ecb6d2afe4941ba5eecfb0\": rpc error: code = NotFound desc = could not find container \"cd64dd8c2d7926f3f1cab5afb383af52b5535e7099ecb6d2afe4941ba5eecfb0\": container with ID starting with cd64dd8c2d7926f3f1cab5afb383af52b5535e7099ecb6d2afe4941ba5eecfb0 not found: ID does not exist"
Sep 16 17:11:53.384091 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:53.384057    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:11:53.454125 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:53.454095    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gc8gs_9e1f1b7b-ac55-41fd-b6d5-1c6257652517/konnectivity-agent/0.log" limit=1024
Sep 16 17:11:54.076810 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:54.076767    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b6cdb29-f428-40e6-ac02-73ece22cf9c3" path="/var/lib/kubelet/pods/6b6cdb29-f428-40e6-ac02-73ece22cf9c3/volumes"
Sep 16 17:11:55.834516 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:55.834486    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-2qjkj_93232e18-073e-43c6-b4ce-0b88d0a675d5/manager/0.log"
Sep 16 17:11:55.887858 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:55.887829    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-2cglz_5b483164-2cb4-419d-a6ff-e726301444ed/kuadrant-console-plugin/0.log"
Sep 16 17:11:56.068112 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:56.068081    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-q7kx5_fc5c5232-5455-488f-9b79-ed0048e637c0/limitador/0.log"
Sep 16 17:11:57.700009 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:57.699923    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fsvlc_23df07fb-e55c-416b-a43e-adb9b81fe3b3/node-exporter/0.log"
Sep 16 17:11:57.730139 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:57.730116    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fsvlc_23df07fb-e55c-416b-a43e-adb9b81fe3b3/kube-rbac-proxy/0.log"
Sep 16 17:11:57.754676 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:11:57.754645    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fsvlc_23df07fb-e55c-416b-a43e-adb9b81fe3b3/init-textfile/0.log"
Sep 16 17:12:00.055505 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:00.055471    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zgcmz/perf-node-gather-daemonset-s2n8n"]
Sep 16 17:12:00.055919 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:00.055744    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b6cdb29-f428-40e6-ac02-73ece22cf9c3" containerName="copy"
Sep 16 17:12:00.055919 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:00.055754    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b6cdb29-f428-40e6-ac02-73ece22cf9c3" containerName="copy"
Sep 16 17:12:00.055919 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:00.055766    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b6cdb29-f428-40e6-ac02-73ece22cf9c3" containerName="gather"
Sep 16 17:12:00.055919 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:00.055772    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b6cdb29-f428-40e6-ac02-73ece22cf9c3" containerName="gather"
Sep 16 17:12:00.055919 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:00.055830    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b6cdb29-f428-40e6-ac02-73ece22cf9c3" containerName="gather"
Sep 16 17:12:00.055919 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:00.055839    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b6cdb29-f428-40e6-ac02-73ece22cf9c3" containerName="copy"
Sep 16 17:12:00.057583 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:00.057561    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zgcmz/perf-node-gather-daemonset-s2n8n"
Sep 16 17:12:00.060299 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:00.060281    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zgcmz\"/\"default-dockercfg-lmlsv\""
Sep 16 17:12:00.060416 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:00.060321    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zgcmz\"/\"openshift-service-ca.crt\""
Sep 16 17:12:00.060416 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:00.060335    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zgcmz\"/\"kube-root-ca.crt\""
Sep 16 17:12:00.066460 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:00.066432    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zgcmz/perf-node-gather-daemonset-s2n8n"]
Sep 16 17:12:00.190669 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:00.190633    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aeeaa809-665d-4b2e-868f-dcb40198f556-podres\") pod \"perf-node-gather-daemonset-s2n8n\" (UID: \"aeeaa809-665d-4b2e-868f-dcb40198f556\") " pod="openshift-must-gather-zgcmz/perf-node-gather-daemonset-s2n8n"
Sep 16 17:12:00.190669 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:00.190672    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aeeaa809-665d-4b2e-868f-dcb40198f556-proc\") pod \"perf-node-gather-daemonset-s2n8n\" (UID: \"aeeaa809-665d-4b2e-868f-dcb40198f556\") " pod="openshift-must-gather-zgcmz/perf-node-gather-daemonset-s2n8n"
Sep 16 17:12:00.190960 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:00.190701    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j228s\" (UniqueName: \"kubernetes.io/projected/aeeaa809-665d-4b2e-868f-dcb40198f556-kube-api-access-j228s\") pod \"perf-node-gather-daemonset-s2n8n\" (UID: \"aeeaa809-665d-4b2e-868f-dcb40198f556\") " pod="openshift-must-gather-zgcmz/perf-node-gather-daemonset-s2n8n"
Sep 16 17:12:00.190960 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:00.190787    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aeeaa809-665d-4b2e-868f-dcb40198f556-sys\") pod \"perf-node-gather-daemonset-s2n8n\" (UID: \"aeeaa809-665d-4b2e-868f-dcb40198f556\") " pod="openshift-must-gather-zgcmz/perf-node-gather-daemonset-s2n8n"
Sep 16 17:12:00.190960 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:00.190851    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aeeaa809-665d-4b2e-868f-dcb40198f556-lib-modules\") pod \"perf-node-gather-daemonset-s2n8n\" (UID: \"aeeaa809-665d-4b2e-868f-dcb40198f556\") " pod="openshift-must-gather-zgcmz/perf-node-gather-daemonset-s2n8n"
Sep 16 17:12:00.291901 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:00.291872    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aeeaa809-665d-4b2e-868f-dcb40198f556-podres\") pod \"perf-node-gather-daemonset-s2n8n\" (UID: \"aeeaa809-665d-4b2e-868f-dcb40198f556\") " pod="openshift-must-gather-zgcmz/perf-node-gather-daemonset-s2n8n"
Sep 16 17:12:00.291901 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:00.291904    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aeeaa809-665d-4b2e-868f-dcb40198f556-proc\") pod \"perf-node-gather-daemonset-s2n8n\" (UID: \"aeeaa809-665d-4b2e-868f-dcb40198f556\") " pod="openshift-must-gather-zgcmz/perf-node-gather-daemonset-s2n8n"
Sep 16 17:12:00.292116 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:00.291929    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j228s\" (UniqueName: \"kubernetes.io/projected/aeeaa809-665d-4b2e-868f-dcb40198f556-kube-api-access-j228s\") pod \"perf-node-gather-daemonset-s2n8n\" (UID: \"aeeaa809-665d-4b2e-868f-dcb40198f556\") " pod="openshift-must-gather-zgcmz/perf-node-gather-daemonset-s2n8n"
Sep 16 17:12:00.292116 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:00.291988    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aeeaa809-665d-4b2e-868f-dcb40198f556-proc\") pod \"perf-node-gather-daemonset-s2n8n\" (UID: \"aeeaa809-665d-4b2e-868f-dcb40198f556\") " pod="openshift-must-gather-zgcmz/perf-node-gather-daemonset-s2n8n"
Sep 16 17:12:00.292116 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:00.292012    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aeeaa809-665d-4b2e-868f-dcb40198f556-sys\") pod \"perf-node-gather-daemonset-s2n8n\" (UID: \"aeeaa809-665d-4b2e-868f-dcb40198f556\") " pod="openshift-must-gather-zgcmz/perf-node-gather-daemonset-s2n8n"
Sep 16 17:12:00.292116 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:00.292029    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aeeaa809-665d-4b2e-868f-dcb40198f556-lib-modules\") pod \"perf-node-gather-daemonset-s2n8n\" (UID: \"aeeaa809-665d-4b2e-868f-dcb40198f556\") " pod="openshift-must-gather-zgcmz/perf-node-gather-daemonset-s2n8n"
Sep 16 17:12:00.292116 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:00.292066    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aeeaa809-665d-4b2e-868f-dcb40198f556-podres\") pod \"perf-node-gather-daemonset-s2n8n\" (UID: \"aeeaa809-665d-4b2e-868f-dcb40198f556\") " pod="openshift-must-gather-zgcmz/perf-node-gather-daemonset-s2n8n"
Sep 16 17:12:00.292284 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:00.292117    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aeeaa809-665d-4b2e-868f-dcb40198f556-sys\") pod \"perf-node-gather-daemonset-s2n8n\" (UID: \"aeeaa809-665d-4b2e-868f-dcb40198f556\") " pod="openshift-must-gather-zgcmz/perf-node-gather-daemonset-s2n8n"
Sep 16 17:12:00.292284 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:00.292146    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aeeaa809-665d-4b2e-868f-dcb40198f556-lib-modules\") pod \"perf-node-gather-daemonset-s2n8n\" (UID: \"aeeaa809-665d-4b2e-868f-dcb40198f556\") " pod="openshift-must-gather-zgcmz/perf-node-gather-daemonset-s2n8n"
Sep 16 17:12:00.300329 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:00.300297    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j228s\" (UniqueName: \"kubernetes.io/projected/aeeaa809-665d-4b2e-868f-dcb40198f556-kube-api-access-j228s\") pod \"perf-node-gather-daemonset-s2n8n\" (UID: \"aeeaa809-665d-4b2e-868f-dcb40198f556\") " pod="openshift-must-gather-zgcmz/perf-node-gather-daemonset-s2n8n"
Sep 16 17:12:00.367294 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:00.367229    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zgcmz/perf-node-gather-daemonset-s2n8n"
Sep 16 17:12:00.634313 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:00.634269    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zgcmz/perf-node-gather-daemonset-s2n8n"]
Sep 16 17:12:00.636956 ip-10-0-136-228 kubenswrapper[2554]: W0916 17:12:00.636920    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podaeeaa809_665d_4b2e_868f_dcb40198f556.slice/crio-dec0e17ac6119481c7ce61891b63486b78d7924dd91b68484b58cece64d3c942 WatchSource:0}: Error finding container dec0e17ac6119481c7ce61891b63486b78d7924dd91b68484b58cece64d3c942: Status 404 returned error can't find the container with id dec0e17ac6119481c7ce61891b63486b78d7924dd91b68484b58cece64d3c942
Sep 16 17:12:00.768660 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:00.768628    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zgcmz/perf-node-gather-daemonset-s2n8n" event={"ID":"aeeaa809-665d-4b2e-868f-dcb40198f556","Type":"ContainerStarted","Data":"f7982eb717666c7dd39fbed1f88c84990113a2924c66e46e1a261451363e2ffd"}
Sep 16 17:12:00.768786 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:00.768686    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zgcmz/perf-node-gather-daemonset-s2n8n" event={"ID":"aeeaa809-665d-4b2e-868f-dcb40198f556","Type":"ContainerStarted","Data":"dec0e17ac6119481c7ce61891b63486b78d7924dd91b68484b58cece64d3c942"}
Sep 16 17:12:00.768786 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:00.768741    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zgcmz/perf-node-gather-daemonset-s2n8n"
Sep 16 17:12:00.786748 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:00.786703    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zgcmz/perf-node-gather-daemonset-s2n8n" podStartSLOduration=0.786690584 podStartE2EDuration="786.690584ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:12:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:12:00.785787897 +0000 UTC m=+4921.230480091" watchObservedRunningTime="2026-09-16 17:12:00.786690584 +0000 UTC m=+4921.231382761"
Sep 16 17:12:02.221659 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:02.221633    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gb9f6_8f22950d-c9c3-44bd-bd0c-56ae5fb22821/dns/0.log"
Sep 16 17:12:02.261175 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:02.261148    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gb9f6_8f22950d-c9c3-44bd-bd0c-56ae5fb22821/kube-rbac-proxy/0.log"
Sep 16 17:12:02.500774 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:02.500700    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ffp62_ef023c4c-0a32-416e-8a44-f8e8e4302e5f/dns-node-resolver/0.log"
Sep 16 17:12:03.083938 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:03.083910    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fzl79_7b3420ee-3d02-4cbf-a6b2-fc58c2da0298/node-ca/0.log"
Sep 16 17:12:04.825505 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:04.825478    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-g826v_83d036c1-52ce-4513-8ffa-d2e74def24c6/serve-healthcheck-canary/0.log"
Sep 16 17:12:05.355695 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:05.355661    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-62g56_bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a/kube-rbac-proxy/0.log"
Sep 16 17:12:05.377494 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:05.377474    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-62g56_bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a/exporter/0.log"
Sep 16 17:12:05.401892 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:05.401858    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-62g56_bc0e5d11-ed3b-4c32-996a-35bbc8ffc88a/extractor/0.log"
Sep 16 17:12:06.781713 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:06.781683    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zgcmz/perf-node-gather-daemonset-s2n8n"
Sep 16 17:12:08.478473 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:08.478443    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-75c8bf98c8-k8hnj_1cca4de1-23f3-4a52-8439-6d5ab51428de/manager/0.log"
Sep 16 17:12:11.041645 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:11.041616    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-87g7k_295f4d72-af70-4a3b-ad51-79874ee0345b/s3-init/0.log"
Sep 16 17:12:11.072859 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:11.072837    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-87g7k_295f4d72-af70-4a3b-ad51-79874ee0345b/download-hf-model/0.log"
Sep 16 17:12:17.342515 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:17.342479    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-7gbkh_c30e3d8c-1676-44b8-9a25-fce32e24c7bc/kube-multus/0.log"
Sep 16 17:12:17.604726 ip-10-0-136-228 kubenswrapper[2554]: I0916 17:12:17.604652    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-65c8p_d8e64660-b4ae-45ae-b31f-1faee114f212/kube-multus-additional-cni-plugins/0.log"