Sep 17 21:56:24.517812 ip-10-0-140-157 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 21:56:24.517825 ip-10-0-140-157 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 21:56:24.517835 ip-10-0-140-157 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 21:56:24.518034 ip-10-0-140-157 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 21:56:34.594307 ip-10-0-140-157 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 21:56:34.594331 ip-10-0-140-157 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 702f6581c8c644399bcdfe67915dd803 --
Sep 17 21:58:17.097465 ip-10-0-140-157 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 21:58:20.481859 ip-10-0-140-157 kubenswrapper[2555]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 21:58:20.481859 ip-10-0-140-157 kubenswrapper[2555]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 21:58:20.481859 ip-10-0-140-157 kubenswrapper[2555]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 21:58:20.481859 ip-10-0-140-157 kubenswrapper[2555]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 21:58:20.481859 ip-10-0-140-157 kubenswrapper[2555]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 21:58:20.482604 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.481927    2555 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 21:58:20.487103 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487086    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:58:20.487103 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487102    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:58:20.487103 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487105    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:58:20.487212 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487109    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:58:20.487212 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487112    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:58:20.487212 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487120    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:58:20.487212 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487125    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:58:20.487212 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487129    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:58:20.487212 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487132    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:58:20.487212 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487135    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:58:20.487212 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487138    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:58:20.487212 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487141    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:58:20.487212 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487144    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:58:20.487212 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487146    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:58:20.487212 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487149    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:58:20.487212 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487151    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:58:20.487212 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487154    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:58:20.487212 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487157    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:58:20.487212 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487160    2555 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:58:20.487212 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487162    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:58:20.487212 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487165    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:58:20.487212 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487168    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:58:20.487662 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487170    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:58:20.487662 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487173    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:58:20.487662 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487175    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:58:20.487662 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487190    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:58:20.487662 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487193    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:58:20.487662 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487195    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:58:20.487662 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487198    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:58:20.487662 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487200    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:58:20.487662 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487203    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:58:20.487662 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487206    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:58:20.487662 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487209    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:58:20.487662 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487212    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:58:20.487662 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487215    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:58:20.487662 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487218    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:58:20.487662 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487222    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:58:20.487662 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487224    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:58:20.487662 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487227    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:58:20.487662 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487229    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:58:20.487662 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487232    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:58:20.487662 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487234    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:58:20.488174 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487237    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:58:20.488174 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487239    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:58:20.488174 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487242    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:58:20.488174 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487244    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:58:20.488174 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487247    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:58:20.488174 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487250    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:58:20.488174 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487252    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:58:20.488174 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487255    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:58:20.488174 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487257    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:58:20.488174 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487260    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:58:20.488174 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487265    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:58:20.488174 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487268    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:58:20.488174 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487271    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:58:20.488174 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487275    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:58:20.488174 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487278    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:58:20.488174 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487280    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:58:20.488174 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487283    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:58:20.488174 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487285    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:58:20.488174 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487288    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:58:20.488643 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487291    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:58:20.488643 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487293    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:58:20.488643 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487296    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:58:20.488643 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487298    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:58:20.488643 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487301    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:58:20.488643 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487303    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:58:20.488643 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487306    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:58:20.488643 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487309    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:58:20.488643 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487311    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:58:20.488643 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487316    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:58:20.488643 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487318    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:58:20.488643 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487321    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:58:20.488643 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487323    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:58:20.488643 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487326    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:58:20.488643 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487329    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:58:20.488643 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487331    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:58:20.488643 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487334    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:58:20.488643 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487336    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:58:20.488643 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487339    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:58:20.488643 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487341    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:58:20.489158 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487343    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:58:20.489158 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487346    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:58:20.489158 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487348    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:58:20.489158 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487351    2555 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:58:20.489158 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487353    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:58:20.489158 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487356    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:58:20.489158 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487360    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:58:20.489158 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487363    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:58:20.489158 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487365    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:58:20.489158 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487368    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:58:20.489158 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487370    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:58:20.489158 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487374    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:58:20.489158 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487378    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:58:20.489158 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487380    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:58:20.489158 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487384    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:58:20.489158 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487386    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:58:20.489158 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487389    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:58:20.489158 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.487392    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:58:20.489158 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488061    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:58:20.489158 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488068    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:58:20.489651 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488072    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:58:20.489651 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488075    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:58:20.489651 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488078    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:58:20.489651 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488081    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:58:20.489651 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488083    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:58:20.489651 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488086    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:58:20.489651 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488089    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:58:20.489651 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488091    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:58:20.489651 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488094    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:58:20.489651 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488096    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:58:20.489651 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488099    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:58:20.489651 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488101    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:58:20.489651 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488104    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:58:20.489651 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488106    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:58:20.489651 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488109    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:58:20.489651 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488112    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:58:20.489651 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488114    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:58:20.489651 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488117    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:58:20.489651 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488120    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:58:20.489651 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488123    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:58:20.490172 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488126    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:58:20.490172 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488128    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:58:20.490172 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488131    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:58:20.490172 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488134    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:58:20.490172 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488136    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:58:20.490172 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488139    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:58:20.490172 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488141    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:58:20.490172 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488144    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:58:20.490172 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488146    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:58:20.490172 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488149    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:58:20.490172 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488151    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:58:20.490172 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488155    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:58:20.490172 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488158    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:58:20.490172 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488161    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:58:20.490172 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488163    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:58:20.490172 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488166    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:58:20.490172 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488168    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:58:20.490172 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488170    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:58:20.490172 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488175    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:58:20.490655 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488193    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:58:20.490655 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488196    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:58:20.490655 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488199    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:58:20.490655 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488202    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:58:20.490655 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488206    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:58:20.490655 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488209    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:58:20.490655 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488211    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:58:20.490655 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488214    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:58:20.490655 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488216    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:58:20.490655 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488219    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:58:20.490655 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488222    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:58:20.490655 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488226    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:58:20.490655 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488228    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:58:20.490655 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488231    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:58:20.490655 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488234    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:58:20.490655 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488237    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:58:20.490655 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488239    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:58:20.490655 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488242    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:58:20.490655 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488245    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:58:20.490655 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488247    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:58:20.491137 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488250    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:58:20.491137 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488253    2555 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:58:20.491137 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488255    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:58:20.491137 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488259    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:58:20.491137 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488262    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:58:20.491137 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488266    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:58:20.491137 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488269    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:58:20.491137 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488273    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:58:20.491137 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488275    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:58:20.491137 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488278    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:58:20.491137 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488281    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:58:20.491137 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488283    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:58:20.491137 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488286    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:58:20.491137 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488289    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:58:20.491137 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488291    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:58:20.491137 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488294    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:58:20.491137 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488296    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:58:20.491137 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488299    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:58:20.491137 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488301    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:58:20.491607 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488304    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:58:20.491607 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488307    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:58:20.491607 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488310    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:58:20.491607 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488313    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:58:20.491607 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488316    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:58:20.491607 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488319    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:58:20.491607 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488321    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:58:20.491607 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488324    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:58:20.491607 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488326    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:58:20.491607 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488329    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:58:20.491607 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488331    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:58:20.491607 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488334    2555 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:58:20.491607 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488336    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:58:20.491607 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488339    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:58:20.491607 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488341    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:58:20.491607 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488344    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:58:20.491607 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488346    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:58:20.491607 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488349    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:58:20.491607 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488351    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:58:20.491607 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488728    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:58:20.492089 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488733    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:58:20.492089 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488736    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:58:20.492089 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488739    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:58:20.492089 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488742    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:58:20.492089 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488744    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:58:20.492089 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488747    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:58:20.492089 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488749    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:58:20.492089 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488752    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:58:20.492089 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488754    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:58:20.492089 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488757    2555 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:58:20.492089 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488761    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:58:20.492089 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488764    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:58:20.492089 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488767    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:58:20.492089 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488770    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:58:20.492089 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488772    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:58:20.492089 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488778    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:58:20.492089 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488781    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:58:20.492089 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488785    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:58:20.492089 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488788    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:58:20.492608 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488791    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:58:20.492608 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488793    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:58:20.492608 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488796    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:58:20.492608 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488799    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:58:20.492608 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488801    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:58:20.492608 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488804    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:58:20.492608 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488807    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:58:20.492608 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488809    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:58:20.492608 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488812    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:58:20.492608 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488814    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:58:20.492608 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488817    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:58:20.492608 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488820    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:58:20.492608 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488823    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:58:20.492608 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488826    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:58:20.492608 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488828    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:58:20.492608 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488831    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:58:20.492608 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488833    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:58:20.492608 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488836    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:58:20.492608 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488838    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:58:20.493082 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488841    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:58:20.493082 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488843    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:58:20.493082 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488847    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:58:20.493082 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488849    2555 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:58:20.493082 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488852    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:58:20.493082 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488854    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:58:20.493082 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488856    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:58:20.493082 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488859    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:58:20.493082 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488862    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:58:20.493082 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488865    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:58:20.493082 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488867    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:58:20.493082 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488870    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:58:20.493082 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488873    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:58:20.493082 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488875    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:58:20.493082 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488878    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:58:20.493082 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488880    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:58:20.493082 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488883    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:58:20.493082 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488885    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:58:20.493082 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488888    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:58:20.493082 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488890    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:58:20.493583 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488893    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:58:20.493583 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488895    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:58:20.493583 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488898    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:58:20.493583 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488900    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:58:20.493583 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488903    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:58:20.493583 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488906    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:58:20.493583 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488908    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:58:20.493583 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488911    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:58:20.493583 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488913    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:58:20.493583 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488916    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:58:20.493583 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488921    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:58:20.493583 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488924    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:58:20.493583 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488927    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:58:20.493583 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488930    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:58:20.493583 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488933    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:58:20.493583 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488935    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:58:20.493583 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488938    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:58:20.493583 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488940    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:58:20.493583 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488943    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:58:20.493583 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488949    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:58:20.494098 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488952    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:58:20.494098 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488954    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:58:20.494098 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488957    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:58:20.494098 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488960    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:58:20.494098 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488962    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:58:20.494098 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488965    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:58:20.494098 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488967    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:58:20.494098 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488970    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:58:20.494098 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488973    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:58:20.494098 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488975    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:58:20.494098 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488978    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:58:20.494098 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488980    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:58:20.494098 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488983    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:58:20.494098 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488985    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:58:20.494098 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488988    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:58:20.494098 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488991    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:58:20.494098 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488993    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:58:20.494098 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488996    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:58:20.494098 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.488998    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:58:20.494098 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489001    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:58:20.494599 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489073    2555 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 21:58:20.494599 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489108    2555 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 21:58:20.494599 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489114    2555 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 21:58:20.494599 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489118    2555 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 21:58:20.494599 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489122    2555 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 21:58:20.494599 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489125    2555 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 21:58:20.494599 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489129    2555 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 21:58:20.494599 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489133    2555 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 21:58:20.494599 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489136    2555 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 21:58:20.494599 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489139    2555 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 21:58:20.494599 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489143    2555 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 21:58:20.494599 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489148    2555 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 21:58:20.494599 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489151    2555 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 21:58:20.494599 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489154    2555 flags.go:64] FLAG: --cgroup-root=""
Sep 17 21:58:20.494599 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489157    2555 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 21:58:20.494599 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489160    2555 flags.go:64] FLAG: --client-ca-file=""
Sep 17 21:58:20.494599 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489163    2555 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 21:58:20.494599 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489166    2555 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 21:58:20.494599 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489171    2555 flags.go:64] FLAG: --cluster-domain=""
Sep 17 21:58:20.494599 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489174    2555 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 21:58:20.494599 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489190    2555 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 21:58:20.494599 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489194    2555 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 21:58:20.494599 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489197    2555 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 21:58:20.494599 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489201    2555 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 21:58:20.495224 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489204    2555 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 21:58:20.495224 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489207    2555 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 21:58:20.495224 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489210    2555 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 21:58:20.495224 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489213    2555 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 21:58:20.495224 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489216    2555 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 21:58:20.495224 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489219    2555 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 21:58:20.495224 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489222    2555 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 21:58:20.495224 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489225    2555 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 21:58:20.495224 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489230    2555 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 21:58:20.495224 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489233    2555 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 21:58:20.495224 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489236    2555 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 21:58:20.495224 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489239    2555 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 21:58:20.495224 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489242    2555 flags.go:64] FLAG: --enable-server="true"
Sep 17 21:58:20.495224 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489244    2555 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 21:58:20.495224 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489249    2555 flags.go:64] FLAG: --event-burst="100"
Sep 17 21:58:20.495224 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489252    2555 flags.go:64] FLAG: --event-qps="50"
Sep 17 21:58:20.495224 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489255    2555 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 21:58:20.495224 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489258    2555 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 21:58:20.495224 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489261    2555 flags.go:64] FLAG: --eviction-hard=""
Sep 17 21:58:20.495224 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489266    2555 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 21:58:20.495224 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489270    2555 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 21:58:20.495224 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489273    2555 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 21:58:20.495224 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489276    2555 flags.go:64] FLAG: --eviction-soft=""
Sep 17 21:58:20.495224 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489279    2555 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 21:58:20.495224 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489282    2555 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 21:58:20.495845 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489285    2555 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 21:58:20.495845 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489288    2555 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 21:58:20.495845 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489291    2555 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 21:58:20.495845 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489294    2555 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 21:58:20.495845 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489297    2555 flags.go:64] FLAG: --feature-gates=""
Sep 17 21:58:20.495845 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489301    2555 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 21:58:20.495845 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489304    2555 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 21:58:20.495845 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489307    2555 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 21:58:20.495845 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489310    2555 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 21:58:20.495845 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489312    2555 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 21:58:20.495845 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489315    2555 flags.go:64] FLAG: --help="false"
Sep 17 21:58:20.495845 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489318    2555 flags.go:64] FLAG: --hostname-override="ip-10-0-140-157.ec2.internal"
Sep 17 21:58:20.495845 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489321    2555 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 21:58:20.495845 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489324    2555 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 21:58:20.495845 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489327    2555 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 21:58:20.495845 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489330    2555 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 21:58:20.495845 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489334    2555 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 21:58:20.495845 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489337    2555 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 21:58:20.495845 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489340    2555 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 21:58:20.495845 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489343    2555 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 21:58:20.495845 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489346    2555 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 21:58:20.495845 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489348    2555 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 21:58:20.495845 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489351    2555 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 21:58:20.495845 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489354    2555 flags.go:64] FLAG: --kube-reserved=""
Sep 17 21:58:20.496439 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489357    2555 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 21:58:20.496439 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489360    2555 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 21:58:20.496439 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489364    2555 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 21:58:20.496439 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489368    2555 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 21:58:20.496439 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489371    2555 flags.go:64] FLAG: --lock-file=""
Sep 17 21:58:20.496439 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489373    2555 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 21:58:20.496439 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489376    2555 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 21:58:20.496439 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489379    2555 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 21:58:20.496439 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489385    2555 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 21:58:20.496439 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489388    2555 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 21:58:20.496439 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489390    2555 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 21:58:20.496439 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489393    2555 flags.go:64] FLAG: --logging-format="text"
Sep 17 21:58:20.496439 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489396    2555 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 21:58:20.496439 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489399    2555 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 21:58:20.496439 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489403    2555 flags.go:64] FLAG: --manifest-url=""
Sep 17 21:58:20.496439 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489406    2555 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 21:58:20.496439 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489409    2555 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 21:58:20.496439 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489412    2555 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 21:58:20.496439 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489416    2555 flags.go:64] FLAG: --max-pods="110"
Sep 17 21:58:20.496439 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489419    2555 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 21:58:20.496439 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489422    2555 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 21:58:20.496439 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489425    2555 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 21:58:20.496439 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489428    2555 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 21:58:20.496439 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489431    2555 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 21:58:20.496439 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489433    2555 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 21:58:20.497267 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489437    2555 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 21:58:20.497267 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489444    2555 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 21:58:20.497267 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489447    2555 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 21:58:20.497267 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489450    2555 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 21:58:20.497267 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489452    2555 flags.go:64] FLAG: --pod-cidr=""
Sep 17 21:58:20.497267 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489455    2555 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 21:58:20.497267 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489460    2555 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 21:58:20.497267 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489463    2555 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 21:58:20.497267 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489466    2555 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 21:58:20.497267 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489468    2555 flags.go:64] FLAG: --port="10250"
Sep 17 21:58:20.497267 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489474    2555 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 21:58:20.497267 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489477    2555 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f84c75455d5534bf"
Sep 17 21:58:20.497267 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489480    2555 flags.go:64] FLAG: --qos-reserved=""
Sep 17 21:58:20.497267 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489483    2555 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 21:58:20.497267 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489486    2555 flags.go:64] FLAG: --register-node="true"
Sep 17 21:58:20.497267 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489488    2555 flags.go:64] FLAG: --register-with-taints=""
Sep 17 21:58:20.497267 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489492    2555 flags.go:64] FLAG: --registry-burst="10"
Sep 17 21:58:20.497267 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489495    2555 flags.go:64] FLAG: --registry-qps="5"
Sep 17 21:58:20.497267 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489498    2555 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 21:58:20.497267 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489500    2555 flags.go:64] FLAG: --reserved-memory=""
Sep 17 21:58:20.497267 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489504    2555 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 21:58:20.497267 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489507    2555 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 21:58:20.497267 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489510    2555 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 21:58:20.497267 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489512    2555 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 21:58:20.497267 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489515    2555 flags.go:64] FLAG: --runonce="false"
Sep 17 21:58:20.498198 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489518    2555 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 21:58:20.498198 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489521    2555 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 21:58:20.498198 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489524    2555 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 21:58:20.498198 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489527    2555 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 21:58:20.498198 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489530    2555 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 21:58:20.498198 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489532    2555 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 21:58:20.498198 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489535    2555 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 21:58:20.498198 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489538    2555 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 21:58:20.498198 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489542    2555 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 21:58:20.498198 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489544    2555 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 21:58:20.498198 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489547    2555 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 21:58:20.498198 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489550    2555 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 21:58:20.498198 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489553    2555 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 21:58:20.498198 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489556    2555 flags.go:64] FLAG: --system-cgroups=""
Sep 17 21:58:20.498198 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489559    2555 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 21:58:20.498198 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489564    2555 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 21:58:20.498198 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489567    2555 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 21:58:20.498198 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489573    2555 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 21:58:20.498198 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489577    2555 flags.go:64] FLAG: --tls-min-version=""
Sep 17 21:58:20.498198 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489580    2555 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 21:58:20.498198 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489582    2555 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 21:58:20.498198 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489585    2555 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 21:58:20.498198 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489588    2555 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 21:58:20.498198 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489591    2555 flags.go:64] FLAG: --v="2"
Sep 17 21:58:20.498198 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489595    2555 flags.go:64] FLAG: --version="false"
Sep 17 21:58:20.498822 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489599    2555 flags.go:64] FLAG: --vmodule=""
Sep 17 21:58:20.498822 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489602    2555 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 21:58:20.498822 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.489605    2555 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 21:58:20.498822 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489715    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:58:20.498822 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489719    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:58:20.498822 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489722    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:58:20.498822 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489725    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:58:20.498822 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489728    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:58:20.498822 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489731    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:58:20.498822 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489733    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:58:20.498822 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489736    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:58:20.498822 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489739    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:58:20.498822 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489741    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:58:20.498822 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489744    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:58:20.498822 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489746    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:58:20.498822 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489749    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:58:20.498822 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489752    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:58:20.498822 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489754    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:58:20.498822 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489757    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:58:20.498822 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489759    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:58:20.498822 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489762    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:58:20.499394 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489765    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:58:20.499394 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489769    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:58:20.499394 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489773    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:58:20.499394 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489781    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:58:20.499394 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489784    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:58:20.499394 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489787    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:58:20.499394 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489789    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:58:20.499394 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489792    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:58:20.499394 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489795    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:58:20.499394 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489797    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:58:20.499394 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489800    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:58:20.499394 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489802    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:58:20.499394 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489805    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:58:20.499394 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489807    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:58:20.499394 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489810    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:58:20.499394 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489813    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:58:20.499394 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489815    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:58:20.499394 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489818    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:58:20.499394 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489822    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:58:20.499394 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489825    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:58:20.499894 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489829    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:58:20.499894 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489831    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:58:20.499894 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489834    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:58:20.499894 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489837    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:58:20.499894 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489840    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:58:20.499894 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489843    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:58:20.499894 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489846    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:58:20.499894 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489849    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:58:20.499894 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489855    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:58:20.499894 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489858    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:58:20.499894 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489861    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:58:20.499894 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489863    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:58:20.499894 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489866    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:58:20.499894 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489869    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:58:20.499894 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489871    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:58:20.499894 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489876    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:58:20.499894 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489879    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:58:20.499894 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489881    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:58:20.499894 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489884    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:58:20.500479 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489887    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:58:20.500479 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489890    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:58:20.500479 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489892    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:58:20.500479 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489895    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:58:20.500479 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489898    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:58:20.500479 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489900    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:58:20.500479 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489904    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:58:20.500479 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489908    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:58:20.500479 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489911    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:58:20.500479 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489914    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:58:20.500479 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489917    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:58:20.500479 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489919    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:58:20.500479 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489922    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:58:20.500479 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489924    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:58:20.500479 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489927    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:58:20.500479 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489930    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:58:20.500479 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489932    2555 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:58:20.500479 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489935    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:58:20.500479 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489938    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:58:20.500941 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489940    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:58:20.500941 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489943    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:58:20.500941 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489945    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:58:20.500941 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489948    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:58:20.500941 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489951    2555 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:58:20.500941 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489953    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:58:20.500941 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489956    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:58:20.500941 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489959    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:58:20.500941 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489963    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:58:20.500941 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489968    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:58:20.500941 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489971    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:58:20.500941 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489974    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:58:20.500941 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489976    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:58:20.500941 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489979    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:58:20.500941 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489981    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:58:20.500941 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489984    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:58:20.500941 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489987    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:58:20.500941 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489990    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:58:20.500941 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489992    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:58:20.501422 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489995    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:58:20.501422 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.489997    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:58:20.501422 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.490000    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:58:20.501422 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.490002    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:58:20.501422 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.490537    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 21:58:20.501422 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.498829    2555 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 21:58:20.501422 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.498844    2555 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 21:58:20.501422 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.498864    2555 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 21:58:20.501422 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.498870    2555 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 21:58:20.501422 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.498929    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:58:20.501422 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.498935    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:58:20.501422 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.498938    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:58:20.501422 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.498941    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:58:20.501422 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.498944    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:58:20.501778 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.498946    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:58:20.501778 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.498949    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:58:20.501778 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.498952    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:58:20.501778 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.498955    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:58:20.501778 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.498958    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:58:20.501778 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.498960    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:58:20.501778 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.498963    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:58:20.501778 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.498967    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:58:20.501778 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.498971    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:58:20.501778 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.498974    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:58:20.501778 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.498977    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:58:20.501778 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.498980    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:58:20.501778 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.498982    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:58:20.501778 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.498985    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:58:20.501778 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.498988    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:58:20.501778 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.498990    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:58:20.501778 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.498993    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:58:20.501778 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.498995    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:58:20.501778 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.498998    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:58:20.501778 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499001    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:58:20.502307 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499004    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:58:20.502307 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499006    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:58:20.502307 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499010    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:58:20.502307 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499012    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:58:20.502307 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499015    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:58:20.502307 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499018    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:58:20.502307 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499021    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:58:20.502307 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499023    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:58:20.502307 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499026    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:58:20.502307 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499029    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:58:20.502307 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499032    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:58:20.502307 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499034    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:58:20.502307 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499037    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:58:20.502307 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499039    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:58:20.502307 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499042    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:58:20.502307 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499045    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:58:20.502307 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499048    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:58:20.502307 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499051    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:58:20.502307 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499053    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:58:20.502307 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499056    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:58:20.502810 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499059    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:58:20.502810 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499062    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:58:20.502810 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499064    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:58:20.502810 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499067    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:58:20.502810 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499070    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:58:20.502810 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499072    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:58:20.502810 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499075    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:58:20.502810 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499078    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:58:20.502810 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499081    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:58:20.502810 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499084    2555 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:58:20.502810 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499086    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:58:20.502810 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499089    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:58:20.502810 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499092    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:58:20.502810 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499095    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:58:20.502810 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499097    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:58:20.502810 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499100    2555 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:58:20.502810 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499102    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:58:20.502810 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499105    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:58:20.502810 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499107    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:58:20.502810 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499110    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:58:20.503303 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499113    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:58:20.503303 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499115    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:58:20.503303 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499118    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:58:20.503303 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499120    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:58:20.503303 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499123    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:58:20.503303 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499125    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:58:20.503303 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499128    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:58:20.503303 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499130    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:58:20.503303 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499133    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:58:20.503303 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499135    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:58:20.503303 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499138    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:58:20.503303 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499142    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:58:20.503303 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499145    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:58:20.503303 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499148    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:58:20.503303 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499152    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:58:20.503303 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499156    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:58:20.503303 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499159    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:58:20.503303 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499162    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:58:20.503303 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499165    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:58:20.503776 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499168    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:58:20.503776 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499171    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:58:20.503776 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499174    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:58:20.503776 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499177    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:58:20.503776 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499198    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:58:20.503776 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499201    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:58:20.503776 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499204    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:58:20.503776 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499207    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:58:20.503776 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499209    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:58:20.503776 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499212    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:58:20.503776 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499214    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:58:20.503776 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499217    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:58:20.503776 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499220    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:58:20.503776 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499222    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:58:20.503776 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499225    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:58:20.504144 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.499230    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 21:58:20.504144 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499344    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:58:20.504144 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499349    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:58:20.504144 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499353    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:58:20.504144 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499356    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:58:20.504144 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499359    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:58:20.504144 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499361    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:58:20.504144 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499364    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:58:20.504144 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499367    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:58:20.504144 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499370    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:58:20.504144 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499373    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:58:20.504144 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499375    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:58:20.504144 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499378    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:58:20.504144 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499382    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:58:20.504564 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499385    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:58:20.504564 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499388    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:58:20.504564 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499390    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:58:20.504564 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499393    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:58:20.504564 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499396    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:58:20.504564 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499399    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:58:20.504564 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499402    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:58:20.504564 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499405    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:58:20.504564 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499408    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:58:20.504564 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499411    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:58:20.504564 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499413    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:58:20.504564 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499416    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:58:20.504564 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499419    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:58:20.504564 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499422    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:58:20.504564 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499424    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:58:20.504564 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499427    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:58:20.504564 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499429    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:58:20.504564 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499433    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:58:20.504564 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499437    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:58:20.505092 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499440    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:58:20.505092 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499443    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:58:20.505092 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499446    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:58:20.505092 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499449    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:58:20.505092 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499451    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:58:20.505092 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499454    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:58:20.505092 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499457    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:58:20.505092 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499459    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:58:20.505092 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499462    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:58:20.505092 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499464    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:58:20.505092 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499467    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:58:20.505092 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499469    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:58:20.505092 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499472    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:58:20.505092 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499475    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:58:20.505092 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499478    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:58:20.505092 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499481    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:58:20.505092 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499483    2555 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:58:20.505092 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499486    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:58:20.505092 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499489    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:58:20.505092 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499491    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:58:20.505092 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499494    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:58:20.505622 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499497    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:58:20.505622 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499499    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:58:20.505622 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499503    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:58:20.505622 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499505    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:58:20.505622 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499508    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:58:20.505622 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499510    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:58:20.505622 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499513    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:58:20.505622 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499516    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:58:20.505622 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499518    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:58:20.505622 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499521    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:58:20.505622 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499523    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:58:20.505622 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499526    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:58:20.505622 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499528    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:58:20.505622 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499532    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:58:20.505622 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499535    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:58:20.505622 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499539    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:58:20.505622 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499542    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:58:20.505622 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499544    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:58:20.505622 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499547    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:58:20.506096 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499550    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:58:20.506096 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499552    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:58:20.506096 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499555    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:58:20.506096 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499557    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:58:20.506096 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499560    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:58:20.506096 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499562    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:58:20.506096 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499566    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:58:20.506096 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499568    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:58:20.506096 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499571    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:58:20.506096 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499573    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:58:20.506096 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499576    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:58:20.506096 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499578    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:58:20.506096 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499581    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:58:20.506096 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499584    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:58:20.506096 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499587    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:58:20.506096 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499589    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:58:20.506096 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499592    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:58:20.506096 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499594    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:58:20.506096 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499597    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:58:20.506096 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499599    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:58:20.506690 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499602    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:58:20.506690 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499604    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:58:20.506690 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499607    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:58:20.506690 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499610    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:58:20.506690 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499614    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:58:20.506690 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499616    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:58:20.506690 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:20.499619    2555 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:58:20.506690 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.499623    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 21:58:20.506690 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.499730    2555 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 21:58:20.506690 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.501884    2555 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 21:58:20.506690 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.502690    2555 server.go:1019] "Starting client certificate rotation"
Sep 17 21:58:20.506690 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.502786    2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 21:58:20.506690 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.502826    2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 21:58:20.507053 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.505008    2555 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 21:58:20.507053 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.506066    2555 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 21:58:20.511237 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.511221    2555 log.go:25] "Validated CRI v1 runtime API"
Sep 17 21:58:20.512840 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.512828    2555 log.go:25] "Validated CRI v1 image API"
Sep 17 21:58:20.513643 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.513627    2555 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 21:58:20.515696 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.515677    2555 fs.go:135] Filesystem UUIDs: map[726859c7-dff3-4e28-9fc3-150600c57aac:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 fdf75b44-90c9-4a7c-912d-47e70e2f4a69:/dev/nvme0n1p3]
Sep 17 21:58:20.515757 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.515694    2555 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 21:58:20.522052 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.521935    2555 manager.go:216] Machine: {Timestamp:2026-09-17 21:58:20.520994951 +0000 UTC m=+1.967040215 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101902 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2181fe905ad99d8272bc1915b8a243 SystemUUID:ec2181fe-905a-d99d-8272-bc1915b8a243 BootID:702f6581-c8c6-4439-9bcd-fe67915dd803 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e5:b3:3f:ef Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e5:b3:3f:ef Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7e:b8:0b:1a:96:3b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 21:58:20.522052 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.522042    2555 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 21:58:20.522212 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.522114    2555 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 21:58:20.523725 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.523700    2555 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 21:58:20.523884 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.523727    2555 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-157.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 21:58:20.524963 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.524952    2555 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 21:58:20.524999 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.524967    2555 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 21:58:20.524999 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.524975    2555 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 21:58:20.524999 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.524986    2555 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 21:58:20.525073 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.525058    2555 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 21:58:20.528939 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.528925    2555 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 21:58:20.529096 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.529084    2555 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 21:58:20.529137 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.529128    2555 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 21:58:20.529170 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.529144    2555 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 21:58:20.529170 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.529165    2555 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 21:58:20.529252 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.529177    2555 kubelet.go:398] "Adding apiserver pod source"
Sep 17 21:58:20.529252 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.529207    2555 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 21:58:20.529738 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.529720    2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 21:58:20.529819 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.529747    2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 21:58:20.534674 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.534653    2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 21:58:20.534841 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.534810    2555 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 21:58:20.537778 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.537764    2555 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 21:58:20.537843 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.537794    2555 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 21:58:20.538869 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.538858    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 21:58:20.538914 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.538876    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 21:58:20.538914 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.538882    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 21:58:20.538914 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.538888    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 21:58:20.538914 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.538897    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 21:58:20.538914 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.538902    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 21:58:20.538914 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.538915    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 21:58:20.539131 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.538921    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 21:58:20.539131 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.538927    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 21:58:20.539131 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.538933    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 21:58:20.539131 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.538946    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 21:58:20.539131 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.538952    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 21:58:20.539131 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.539000    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 21:58:20.539131 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.539005    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 21:58:20.543140 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.543126    2555 server.go:1268] "Started kubelet"
Sep 17 21:58:20.543250 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.543221    2555 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 21:58:20.543380 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.543327    2555 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 21:58:20.543433 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.543389    2555 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 21:58:20.543734 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:20.543709    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-157.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 21:58:20.543784 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.543777    2555 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-157.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 21:58:20.544256 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:20.544211    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 21:58:20.544516 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.544501    2555 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 21:58:20.544748 ip-10-0-140-157 systemd[1]: Started Kubernetes Kubelet.
Sep 17 21:58:20.545259 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.545228    2555 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 21:58:20.547703 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:20.547685    2555 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 21:58:20.550724 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.550712    2555 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 21:58:20.550792 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.550723    2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 21:58:20.550844 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:20.550801    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-157.ec2.internal\" not found"
Sep 17 21:58:20.551725 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.551707    2555 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 21:58:20.551725 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.551721    2555 factory.go:55] Registering systemd factory
Sep 17 21:58:20.551725 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.551728    2555 factory.go:223] Registration of the systemd container factory successfully
Sep 17 21:58:20.551981 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.551965    2555 factory.go:153] Registering CRI-O factory
Sep 17 21:58:20.552043 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.551987    2555 factory.go:223] Registration of the crio container factory successfully
Sep 17 21:58:20.552043 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.552006    2555 factory.go:103] Registering Raw factory
Sep 17 21:58:20.552043 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.552018    2555 manager.go:1196] Started watching for new ooms in manager
Sep 17 21:58:20.552043 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.552030    2555 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 21:58:20.552223 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.552066    2555 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 21:58:20.552223 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.552077    2555 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 21:58:20.552324 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.552231    2555 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 21:58:20.552324 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.552239    2555 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 21:58:20.552557 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.552541    2555 manager.go:349] Starting recovery of all containers
Sep 17 21:58:20.553781 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:20.553752    2555 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-157.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 21:58:20.553893 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:20.553871    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 21:58:20.573425 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.573386    2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 21:58:20.578129 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.578093    2555 manager.go:354] Recovery completed
Sep 17 21:58:20.581989 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.581974    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:20.583508 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.583485    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-157.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:20.583570 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.583521    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-157.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:20.583570 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.583531    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-157.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:20.584021 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.584003    2555 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 21:58:20.584021 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.584020    2555 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 21:58:20.584171 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.584038    2555 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 21:58:20.584171 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.584010    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:20.584263 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:20.583368    2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-157.ec2.internal.18d63a91f8aa3e15  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-157.ec2.internal,UID:ip-10-0-140-157.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-157.ec2.internal,},FirstTimestamp:2026-09-17 21:58:20.542926357 +0000 UTC m=+1.988971623,LastTimestamp:2026-09-17 21:58:20.542926357 +0000 UTC m=+1.988971623,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-157.ec2.internal,}"
Sep 17 21:58:20.584811 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.584796    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-157.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:20.584880 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.584825    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-157.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:20.584880 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.584836    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-157.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:20.586360 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.586341    2555 policy_none.go:49] "None policy: Start"
Sep 17 21:58:20.586420 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.586364    2555 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 21:58:20.586420 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.586374    2555 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 21:58:20.588354 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.588343    2555 policy_none.go:47] "Start"
Sep 17 21:58:20.596775 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:20.596716    2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-157.ec2.internal.18d63a91fb1578cd  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-157.ec2.internal,UID:ip-10-0-140-157.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-157.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-157.ec2.internal,},FirstTimestamp:2026-09-17 21:58:20.583508173 +0000 UTC m=+2.029553436,LastTimestamp:2026-09-17 21:58:20.583508173 +0000 UTC m=+2.029553436,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-157.ec2.internal,}"
Sep 17 21:58:20.610027 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:20.609961    2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-157.ec2.internal.18d63a91fb15bd91  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-157.ec2.internal,UID:ip-10-0-140-157.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-157.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-157.ec2.internal,},FirstTimestamp:2026-09-17 21:58:20.583525777 +0000 UTC m=+2.029571041,LastTimestamp:2026-09-17 21:58:20.583525777 +0000 UTC m=+2.029571041,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-157.ec2.internal,}"
Sep 17 21:58:20.622566 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:20.622507    2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-157.ec2.internal.18d63a91fb15e2ed  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-157.ec2.internal,UID:ip-10-0-140-157.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-157.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-157.ec2.internal,},FirstTimestamp:2026-09-17 21:58:20.583535341 +0000 UTC m=+2.029580604,LastTimestamp:2026-09-17 21:58:20.583535341 +0000 UTC m=+2.029580604,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-157.ec2.internal,}"
Sep 17 21:58:20.628760 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.628740    2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9xp4f"
Sep 17 21:58:20.631812 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.631799    2555 manager.go:337] "Starting Device Plugin manager"
Sep 17 21:58:20.649594 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:20.631838    2555 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 21:58:20.649594 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.631847    2555 server.go:87] "Starting device plugin registration server"
Sep 17 21:58:20.649594 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.632077    2555 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 21:58:20.649594 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.632098    2555 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 21:58:20.649594 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.632166    2555 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 21:58:20.649594 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.632251    2555 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 21:58:20.649594 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.632260    2555 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 21:58:20.649594 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:20.632817    2555 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 21:58:20.649594 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:20.632871    2555 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-157.ec2.internal\" not found"
Sep 17 21:58:20.649594 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:20.635002    2555 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-157.ec2.internal.18d63a91fb1578cd\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-157.ec2.internal.18d63a91fb1578cd  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-157.ec2.internal,UID:ip-10-0-140-157.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-157.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-157.ec2.internal,},FirstTimestamp:2026-09-17 21:58:20.583508173 +0000 UTC m=+2.029553436,LastTimestamp:2026-09-17 21:58:20.584809312 +0000 UTC m=+2.030854579,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-157.ec2.internal,}"
Sep 17 21:58:20.649594 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.636906    2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9xp4f"
Sep 17 21:58:20.649594 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:20.644204    2555 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-157.ec2.internal.18d63a91fb15bd91\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-157.ec2.internal.18d63a91fb15bd91  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-157.ec2.internal,UID:ip-10-0-140-157.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-157.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-157.ec2.internal,},FirstTimestamp:2026-09-17 21:58:20.583525777 +0000 UTC m=+2.029571041,LastTimestamp:2026-09-17 21:58:20.584831225 +0000 UTC m=+2.030876488,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-157.ec2.internal,}"
Sep 17 21:58:20.672285 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.672266    2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 21:58:20.672365 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.672312    2555 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 21:58:20.672365 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.672331    2555 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 21:58:20.672464 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:20.672366    2555 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 21:58:20.675481 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.675465    2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:58:20.732408 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.732363    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:20.733134 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.733118    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-157.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:20.733208 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.733152    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-157.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:20.733208 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.733166    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-157.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:20.733208 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.733204    2555 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-157.ec2.internal"
Sep 17 21:58:20.740477 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.740464    2555 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-157.ec2.internal"
Sep 17 21:58:20.740517 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:20.740484    2555 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-157.ec2.internal\": node \"ip-10-0-140-157.ec2.internal\" not found"
Sep 17 21:58:20.772775 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.772754    2555 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-157.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-157.ec2.internal"]
Sep 17 21:58:20.772854 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.772817    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:20.774190 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.774161    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-157.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:20.774258 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.774205    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-157.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:20.774258 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.774221    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-157.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:20.776484 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.776473    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:20.776590 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.776578    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-157.ec2.internal"
Sep 17 21:58:20.776626 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.776605    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:20.777287 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.777269    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-157.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:20.777355 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.777301    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-157.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:20.777355 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.777312    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-157.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:20.777355 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.777275    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-157.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:20.777451 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.777368    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-157.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:20.777451 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.777382    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-157.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:20.779659 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.779644    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-157.ec2.internal"
Sep 17 21:58:20.779740 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.779668    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:20.780693 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.780675    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-157.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:20.780765 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.780702    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-157.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:20.780765 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.780712    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-157.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:20.783096 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:20.783053    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-157.ec2.internal\" not found"
Sep 17 21:58:20.804007 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:20.803794    2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-157.ec2.internal\" not found" node="ip-10-0-140-157.ec2.internal"
Sep 17 21:58:20.808131 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:20.808115    2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-157.ec2.internal\" not found" node="ip-10-0-140-157.ec2.internal"
Sep 17 21:58:20.853742 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.853719    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d837547dfeda455efa71ea03001f07b0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-157.ec2.internal\" (UID: \"d837547dfeda455efa71ea03001f07b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-157.ec2.internal"
Sep 17 21:58:20.853822 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.853744    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d837547dfeda455efa71ea03001f07b0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-157.ec2.internal\" (UID: \"d837547dfeda455efa71ea03001f07b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-157.ec2.internal"
Sep 17 21:58:20.853822 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.853762    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/745ac86812f97f62360d667f4353a619-config\") pod \"kube-apiserver-proxy-ip-10-0-140-157.ec2.internal\" (UID: \"745ac86812f97f62360d667f4353a619\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-157.ec2.internal"
Sep 17 21:58:20.883833 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:20.883818    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-157.ec2.internal\" not found"
Sep 17 21:58:20.954277 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.954260    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/745ac86812f97f62360d667f4353a619-config\") pod \"kube-apiserver-proxy-ip-10-0-140-157.ec2.internal\" (UID: \"745ac86812f97f62360d667f4353a619\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-157.ec2.internal"
Sep 17 21:58:20.954356 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.954284    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d837547dfeda455efa71ea03001f07b0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-157.ec2.internal\" (UID: \"d837547dfeda455efa71ea03001f07b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-157.ec2.internal"
Sep 17 21:58:20.954356 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.954305    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d837547dfeda455efa71ea03001f07b0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-157.ec2.internal\" (UID: \"d837547dfeda455efa71ea03001f07b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-157.ec2.internal"
Sep 17 21:58:20.954356 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.954334    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d837547dfeda455efa71ea03001f07b0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-157.ec2.internal\" (UID: \"d837547dfeda455efa71ea03001f07b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-157.ec2.internal"
Sep 17 21:58:20.954471 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.954364    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d837547dfeda455efa71ea03001f07b0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-157.ec2.internal\" (UID: \"d837547dfeda455efa71ea03001f07b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-157.ec2.internal"
Sep 17 21:58:20.954471 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:20.954368    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/745ac86812f97f62360d667f4353a619-config\") pod \"kube-apiserver-proxy-ip-10-0-140-157.ec2.internal\" (UID: \"745ac86812f97f62360d667f4353a619\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-157.ec2.internal"
Sep 17 21:58:20.984363 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:20.984307    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-157.ec2.internal\" not found"
Sep 17 21:58:21.085085 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:21.085060    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-157.ec2.internal\" not found"
Sep 17 21:58:21.108227 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.108212    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-157.ec2.internal"
Sep 17 21:58:21.112024 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.112002    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-157.ec2.internal"
Sep 17 21:58:21.185702 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:21.185666    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-157.ec2.internal\" not found"
Sep 17 21:58:21.286282 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:21.286208    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-157.ec2.internal\" not found"
Sep 17 21:58:21.386770 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:21.386741    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-157.ec2.internal\" not found"
Sep 17 21:58:21.399374 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.399356    2555 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:58:21.459771 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.459749    2555 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:58:21.503173 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.503155    2555 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 21:58:21.503554 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.503292    2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 21:58:21.503554 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.503316    2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 21:58:21.503554 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.503325    2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 21:58:21.529400 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.529376    2555 apiserver.go:52] "Watching apiserver"
Sep 17 21:58:21.544220 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.544160    2555 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 21:58:21.544507 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.544487    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-sggv9","openshift-image-registry/node-ca-rdqkk","openshift-multus/multus-additional-cni-plugins-9sgr6","openshift-network-operator/iptables-alerter-2n8ms","openshift-ovn-kubernetes/ovnkube-node-5s7f5","kube-system/konnectivity-agent-8sfbd","openshift-cluster-node-tuning-operator/tuned-cr4rb","openshift-multus/multus-cn4mv","openshift-multus/network-metrics-daemon-n7n4d","openshift-network-diagnostics/network-check-target-shx44","kube-system/global-pull-secret-syncer-vl469","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zgpjm"]
Sep 17 21:58:21.547059 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.547040    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rdqkk"
Sep 17 21:58:21.549032 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.549014    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 21:58:21.549347 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.549332    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 21:58:21.549455 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.549439    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 21:58:21.549660 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.549625    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rbvld\""
Sep 17 21:58:21.549900 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.549883    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9sgr6"
Sep 17 21:58:21.550803 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.550785    2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 21:58:21.550939 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.550914    2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-157.ec2.internal"
Sep 17 21:58:21.551779 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.551761    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 21:58:21.551883 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.551824    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 21:58:21.551934 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.551911    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 21:58:21.552135 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.552116    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 21:58:21.552135 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.552126    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zxx5t\""
Sep 17 21:58:21.552266 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.552164    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 21:58:21.552476 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.552459    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2n8ms"
Sep 17 21:58:21.554417 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.554390    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 21:58:21.554537 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.554450    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 21:58:21.554537 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.554487    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 21:58:21.554782 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.554767    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5r7hz\""
Sep 17 21:58:21.555436 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.555419    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cr4rb"
Sep 17 21:58:21.557304 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.557281    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/838fe5eb-ddc1-4cea-8a3c-0c5f0e203174-host\") pod \"node-ca-rdqkk\" (UID: \"838fe5eb-ddc1-4cea-8a3c-0c5f0e203174\") " pod="openshift-image-registry/node-ca-rdqkk"
Sep 17 21:58:21.557394 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.557314    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f2ab99ad-b85d-494c-967c-c4e75d08553e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9sgr6\" (UID: \"f2ab99ad-b85d-494c-967c-c4e75d08553e\") " pod="openshift-multus/multus-additional-cni-plugins-9sgr6"
Sep 17 21:58:21.557394 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.557330    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f2ab99ad-b85d-494c-967c-c4e75d08553e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9sgr6\" (UID: \"f2ab99ad-b85d-494c-967c-c4e75d08553e\") " pod="openshift-multus/multus-additional-cni-plugins-9sgr6"
Sep 17 21:58:21.557394 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.557348    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7zh2\" (UniqueName: \"kubernetes.io/projected/f2ab99ad-b85d-494c-967c-c4e75d08553e-kube-api-access-q7zh2\") pod \"multus-additional-cni-plugins-9sgr6\" (UID: \"f2ab99ad-b85d-494c-967c-c4e75d08553e\") " pod="openshift-multus/multus-additional-cni-plugins-9sgr6"
Sep 17 21:58:21.557581 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.557398    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/838fe5eb-ddc1-4cea-8a3c-0c5f0e203174-serviceca\") pod \"node-ca-rdqkk\" (UID: \"838fe5eb-ddc1-4cea-8a3c-0c5f0e203174\") " pod="openshift-image-registry/node-ca-rdqkk"
Sep 17 21:58:21.557581 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.557429    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f2ab99ad-b85d-494c-967c-c4e75d08553e-system-cni-dir\") pod \"multus-additional-cni-plugins-9sgr6\" (UID: \"f2ab99ad-b85d-494c-967c-c4e75d08553e\") " pod="openshift-multus/multus-additional-cni-plugins-9sgr6"
Sep 17 21:58:21.557581 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.557444    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 21:58:21.557581 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.557455    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f2ab99ad-b85d-494c-967c-c4e75d08553e-cni-binary-copy\") pod \"multus-additional-cni-plugins-9sgr6\" (UID: \"f2ab99ad-b85d-494c-967c-c4e75d08553e\") " pod="openshift-multus/multus-additional-cni-plugins-9sgr6"
Sep 17 21:58:21.557581 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.557479    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t6fq\" (UniqueName: \"kubernetes.io/projected/e87296a0-2567-4764-8fd6-5edadb362d0f-kube-api-access-8t6fq\") pod \"iptables-alerter-2n8ms\" (UID: \"e87296a0-2567-4764-8fd6-5edadb362d0f\") " pod="openshift-network-operator/iptables-alerter-2n8ms"
Sep 17 21:58:21.557581 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.557504    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f2ab99ad-b85d-494c-967c-c4e75d08553e-cnibin\") pod \"multus-additional-cni-plugins-9sgr6\" (UID: \"f2ab99ad-b85d-494c-967c-c4e75d08553e\") " pod="openshift-multus/multus-additional-cni-plugins-9sgr6"
Sep 17 21:58:21.557581 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.557526    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e87296a0-2567-4764-8fd6-5edadb362d0f-iptables-alerter-script\") pod \"iptables-alerter-2n8ms\" (UID: \"e87296a0-2567-4764-8fd6-5edadb362d0f\") " pod="openshift-network-operator/iptables-alerter-2n8ms"
Sep 17 21:58:21.557581 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.557558    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-n687c\""
Sep 17 21:58:21.557581 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.557571    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e87296a0-2567-4764-8fd6-5edadb362d0f-host-slash\") pod \"iptables-alerter-2n8ms\" (UID: \"e87296a0-2567-4764-8fd6-5edadb362d0f\") " pod="openshift-network-operator/iptables-alerter-2n8ms"
Sep 17 21:58:21.557873 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.557604    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgs2x\" (UniqueName: \"kubernetes.io/projected/838fe5eb-ddc1-4cea-8a3c-0c5f0e203174-kube-api-access-kgs2x\") pod \"node-ca-rdqkk\" (UID: \"838fe5eb-ddc1-4cea-8a3c-0c5f0e203174\") " pod="openshift-image-registry/node-ca-rdqkk"
Sep 17 21:58:21.557873 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.557614    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 21:58:21.557873 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.557635    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f2ab99ad-b85d-494c-967c-c4e75d08553e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9sgr6\" (UID: \"f2ab99ad-b85d-494c-967c-c4e75d08553e\") " pod="openshift-multus/multus-additional-cni-plugins-9sgr6"
Sep 17 21:58:21.557873 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.557686    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.559649 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.559634    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 21:58:21.559726 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.559638    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-wrddd\""
Sep 17 21:58:21.559865 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.559849    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7n4d"
Sep 17 21:58:21.559943 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:21.559917    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7n4d" podUID="a236d0c6-0efa-47e2-94c1-f4ca507442c4"
Sep 17 21:58:21.560137 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.560123    2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 21:58:21.560216 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.560205    2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-157.ec2.internal"
Sep 17 21:58:21.562079 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.562062    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-shx44"
Sep 17 21:58:21.562170 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:21.562114    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-shx44" podUID="7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd"
Sep 17 21:58:21.562373 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.562355    2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 21:58:21.564381 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.564364    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zgpjm"
Sep 17 21:58:21.566275 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.566250    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 21:58:21.566356 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.566344    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 21:58:21.566437 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.566420    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-n2kq8\""
Sep 17 21:58:21.566595 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.566478    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 21:58:21.566786 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.566770    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sggv9"
Sep 17 21:58:21.568974 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.568906    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 21:58:21.568974 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.568968    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fjdmf\""
Sep 17 21:58:21.569215 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.569199    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 21:58:21.569297 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.569282    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.571285 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.571266    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 21:58:21.571381 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.571367    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 21:58:21.571638 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.571619    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8sfbd"
Sep 17 21:58:21.571763 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.571695    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-d2bsb\""
Sep 17 21:58:21.571981 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.571959    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 21:58:21.572072 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.572008    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 21:58:21.572126 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.572119    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 21:58:21.572176 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.572122    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 21:58:21.574950 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.574921    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 21:58:21.575044 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.574961    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qfrds\""
Sep 17 21:58:21.575095 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.575034    2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 21:58:21.575471 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.575453    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 21:58:21.575796 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.575781    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-157.ec2.internal"]
Sep 17 21:58:21.575854 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.575810    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-157.ec2.internal"]
Sep 17 21:58:21.575893 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.575879    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl469"
Sep 17 21:58:21.575942 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:21.575927    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl469" podUID="84426995-45a3-420d-81b6-2676aba8686b"
Sep 17 21:58:21.584036 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.584020    2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-b7mjg"
Sep 17 21:58:21.592152 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.592133    2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-b7mjg"
Sep 17 21:58:21.642325 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.642250    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 21:53:20 +0000 UTC" deadline="2028-05-12 12:35:25.459471188 +0000 UTC"
Sep 17 21:58:21.642325 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.642323    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14462h37m3.817150491s"
Sep 17 21:58:21.653661 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.653641    2555 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 21:58:21.657985 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.657966    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f2ab99ad-b85d-494c-967c-c4e75d08553e-cnibin\") pod \"multus-additional-cni-plugins-9sgr6\" (UID: \"f2ab99ad-b85d-494c-967c-c4e75d08553e\") " pod="openshift-multus/multus-additional-cni-plugins-9sgr6"
Sep 17 21:58:21.658048 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.657993    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e87296a0-2567-4764-8fd6-5edadb362d0f-host-slash\") pod \"iptables-alerter-2n8ms\" (UID: \"e87296a0-2567-4764-8fd6-5edadb362d0f\") " pod="openshift-network-operator/iptables-alerter-2n8ms"
Sep 17 21:58:21.658048 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658010    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/468939af-9933-402a-aaca-455a88a366d3-etc-systemd\") pod \"tuned-cr4rb\" (UID: \"468939af-9933-402a-aaca-455a88a366d3\") " pod="openshift-cluster-node-tuning-operator/tuned-cr4rb"
Sep 17 21:58:21.658048 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658026    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-host-kubelet\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.658148 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658079    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e87296a0-2567-4764-8fd6-5edadb362d0f-host-slash\") pod \"iptables-alerter-2n8ms\" (UID: \"e87296a0-2567-4764-8fd6-5edadb362d0f\") " pod="openshift-network-operator/iptables-alerter-2n8ms"
Sep 17 21:58:21.658148 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658079    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f2ab99ad-b85d-494c-967c-c4e75d08553e-cnibin\") pod \"multus-additional-cni-plugins-9sgr6\" (UID: \"f2ab99ad-b85d-494c-967c-c4e75d08553e\") " pod="openshift-multus/multus-additional-cni-plugins-9sgr6"
Sep 17 21:58:21.658148 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658102    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-log-socket\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.658148 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658134    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgs2x\" (UniqueName: \"kubernetes.io/projected/838fe5eb-ddc1-4cea-8a3c-0c5f0e203174-kube-api-access-kgs2x\") pod \"node-ca-rdqkk\" (UID: \"838fe5eb-ddc1-4cea-8a3c-0c5f0e203174\") " pod="openshift-image-registry/node-ca-rdqkk"
Sep 17 21:58:21.658284 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658150    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66effb77-e868-4e17-9bab-d9ed02f73237-cnibin\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.658284 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658166    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzdrk\" (UniqueName: \"kubernetes.io/projected/7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd-kube-api-access-tzdrk\") pod \"network-check-target-shx44\" (UID: \"7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd\") " pod="openshift-network-diagnostics/network-check-target-shx44"
Sep 17 21:58:21.658284 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658195    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkljs\" (UniqueName: \"kubernetes.io/projected/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-kube-api-access-fkljs\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.658284 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658211    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/65875e8d-a9e5-4031-bedd-b304efbafde8-konnectivity-ca\") pod \"konnectivity-agent-8sfbd\" (UID: \"65875e8d-a9e5-4031-bedd-b304efbafde8\") " pod="kube-system/konnectivity-agent-8sfbd"
Sep 17 21:58:21.658284 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658224    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfdv8\" (UniqueName: \"kubernetes.io/projected/cc6119ef-2509-4a37-9edb-1c6da2fdef14-kube-api-access-dfdv8\") pod \"node-resolver-sggv9\" (UID: \"cc6119ef-2509-4a37-9edb-1c6da2fdef14\") " pod="openshift-dns/node-resolver-sggv9"
Sep 17 21:58:21.658284 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658237    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-run-openvswitch\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.658284 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658270    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f2ab99ad-b85d-494c-967c-c4e75d08553e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9sgr6\" (UID: \"f2ab99ad-b85d-494c-967c-c4e75d08553e\") " pod="openshift-multus/multus-additional-cni-plugins-9sgr6"
Sep 17 21:58:21.658551 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658297    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7zh2\" (UniqueName: \"kubernetes.io/projected/f2ab99ad-b85d-494c-967c-c4e75d08553e-kube-api-access-q7zh2\") pod \"multus-additional-cni-plugins-9sgr6\" (UID: \"f2ab99ad-b85d-494c-967c-c4e75d08553e\") " pod="openshift-multus/multus-additional-cni-plugins-9sgr6"
Sep 17 21:58:21.658551 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658321    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/468939af-9933-402a-aaca-455a88a366d3-var-lib-kubelet\") pod \"tuned-cr4rb\" (UID: \"468939af-9933-402a-aaca-455a88a366d3\") " pod="openshift-cluster-node-tuning-operator/tuned-cr4rb"
Sep 17 21:58:21.658551 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658345    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66effb77-e868-4e17-9bab-d9ed02f73237-system-cni-dir\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.658551 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658369    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-host-run-ovn-kubernetes\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.658551 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658392    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-ovn-node-metrics-cert\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.658551 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658419    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/838fe5eb-ddc1-4cea-8a3c-0c5f0e203174-serviceca\") pod \"node-ca-rdqkk\" (UID: \"838fe5eb-ddc1-4cea-8a3c-0c5f0e203174\") " pod="openshift-image-registry/node-ca-rdqkk"
Sep 17 21:58:21.658551 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658437    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f2ab99ad-b85d-494c-967c-c4e75d08553e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9sgr6\" (UID: \"f2ab99ad-b85d-494c-967c-c4e75d08553e\") " pod="openshift-multus/multus-additional-cni-plugins-9sgr6"
Sep 17 21:58:21.658551 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658444    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t6fq\" (UniqueName: \"kubernetes.io/projected/e87296a0-2567-4764-8fd6-5edadb362d0f-kube-api-access-8t6fq\") pod \"iptables-alerter-2n8ms\" (UID: \"e87296a0-2567-4764-8fd6-5edadb362d0f\") " pod="openshift-network-operator/iptables-alerter-2n8ms"
Sep 17 21:58:21.658551 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658482    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/468939af-9933-402a-aaca-455a88a366d3-lib-modules\") pod \"tuned-cr4rb\" (UID: \"468939af-9933-402a-aaca-455a88a366d3\") " pod="openshift-cluster-node-tuning-operator/tuned-cr4rb"
Sep 17 21:58:21.658551 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658504    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cc6119ef-2509-4a37-9edb-1c6da2fdef14-hosts-file\") pod \"node-resolver-sggv9\" (UID: \"cc6119ef-2509-4a37-9edb-1c6da2fdef14\") " pod="openshift-dns/node-resolver-sggv9"
Sep 17 21:58:21.658551 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658526    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/468939af-9933-402a-aaca-455a88a366d3-run\") pod \"tuned-cr4rb\" (UID: \"468939af-9933-402a-aaca-455a88a366d3\") " pod="openshift-cluster-node-tuning-operator/tuned-cr4rb"
Sep 17 21:58:21.658551 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658544    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/66effb77-e868-4e17-9bab-d9ed02f73237-host-run-netns\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.659061 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658558    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/66effb77-e868-4e17-9bab-d9ed02f73237-host-var-lib-cni-bin\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.659061 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658582    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/66effb77-e868-4e17-9bab-d9ed02f73237-hostroot\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.659061 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658604    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-systemd-units\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.659061 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658637    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/65875e8d-a9e5-4031-bedd-b304efbafde8-agent-certs\") pod \"konnectivity-agent-8sfbd\" (UID: \"65875e8d-a9e5-4031-bedd-b304efbafde8\") " pod="kube-system/konnectivity-agent-8sfbd"
Sep 17 21:58:21.659061 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658662    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a236d0c6-0efa-47e2-94c1-f4ca507442c4-metrics-certs\") pod \"network-metrics-daemon-n7n4d\" (UID: \"a236d0c6-0efa-47e2-94c1-f4ca507442c4\") " pod="openshift-multus/network-metrics-daemon-n7n4d"
Sep 17 21:58:21.659061 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658682    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d563612f-869f-4e82-9ba2-690fd3e34052-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zgpjm\" (UID: \"d563612f-869f-4e82-9ba2-690fd3e34052\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zgpjm"
Sep 17 21:58:21.659061 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658707    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/468939af-9933-402a-aaca-455a88a366d3-sys\") pod \"tuned-cr4rb\" (UID: \"468939af-9933-402a-aaca-455a88a366d3\") " pod="openshift-cluster-node-tuning-operator/tuned-cr4rb"
Sep 17 21:58:21.659061 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658732    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66effb77-e868-4e17-9bab-d9ed02f73237-multus-cni-dir\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.659061 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658753    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc6119ef-2509-4a37-9edb-1c6da2fdef14-tmp-dir\") pod \"node-resolver-sggv9\" (UID: \"cc6119ef-2509-4a37-9edb-1c6da2fdef14\") " pod="openshift-dns/node-resolver-sggv9"
Sep 17 21:58:21.659061 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658766    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/84426995-45a3-420d-81b6-2676aba8686b-dbus\") pod \"global-pull-secret-syncer-vl469\" (UID: \"84426995-45a3-420d-81b6-2676aba8686b\") " pod="kube-system/global-pull-secret-syncer-vl469"
Sep 17 21:58:21.659061 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658781    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/66effb77-e868-4e17-9bab-d9ed02f73237-host-var-lib-cni-multus\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.659061 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658795    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-env-overrides\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.659061 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658810    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d563612f-869f-4e82-9ba2-690fd3e34052-socket-dir\") pod \"aws-ebs-csi-driver-node-zgpjm\" (UID: \"d563612f-869f-4e82-9ba2-690fd3e34052\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zgpjm"
Sep 17 21:58:21.659061 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658836    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d563612f-869f-4e82-9ba2-690fd3e34052-etc-selinux\") pod \"aws-ebs-csi-driver-node-zgpjm\" (UID: \"d563612f-869f-4e82-9ba2-690fd3e34052\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zgpjm"
Sep 17 21:58:21.659061 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658866    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f2ab99ad-b85d-494c-967c-c4e75d08553e-system-cni-dir\") pod \"multus-additional-cni-plugins-9sgr6\" (UID: \"f2ab99ad-b85d-494c-967c-c4e75d08553e\") " pod="openshift-multus/multus-additional-cni-plugins-9sgr6"
Sep 17 21:58:21.659061 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658883    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/468939af-9933-402a-aaca-455a88a366d3-host\") pod \"tuned-cr4rb\" (UID: \"468939af-9933-402a-aaca-455a88a366d3\") " pod="openshift-cluster-node-tuning-operator/tuned-cr4rb"
Sep 17 21:58:21.659061 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658861    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/838fe5eb-ddc1-4cea-8a3c-0c5f0e203174-serviceca\") pod \"node-ca-rdqkk\" (UID: \"838fe5eb-ddc1-4cea-8a3c-0c5f0e203174\") " pod="openshift-image-registry/node-ca-rdqkk"
Sep 17 21:58:21.659743 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658901    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/468939af-9933-402a-aaca-455a88a366d3-etc-tuned\") pod \"tuned-cr4rb\" (UID: \"468939af-9933-402a-aaca-455a88a366d3\") " pod="openshift-cluster-node-tuning-operator/tuned-cr4rb"
Sep 17 21:58:21.659743 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658907    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f2ab99ad-b85d-494c-967c-c4e75d08553e-system-cni-dir\") pod \"multus-additional-cni-plugins-9sgr6\" (UID: \"f2ab99ad-b85d-494c-967c-c4e75d08553e\") " pod="openshift-multus/multus-additional-cni-plugins-9sgr6"
Sep 17 21:58:21.659743 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658916    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/66effb77-e868-4e17-9bab-d9ed02f73237-multus-socket-dir-parent\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.659743 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658933    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d563612f-869f-4e82-9ba2-690fd3e34052-registration-dir\") pod \"aws-ebs-csi-driver-node-zgpjm\" (UID: \"d563612f-869f-4e82-9ba2-690fd3e34052\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zgpjm"
Sep 17 21:58:21.659743 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658954    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e87296a0-2567-4764-8fd6-5edadb362d0f-iptables-alerter-script\") pod \"iptables-alerter-2n8ms\" (UID: \"e87296a0-2567-4764-8fd6-5edadb362d0f\") " pod="openshift-network-operator/iptables-alerter-2n8ms"
Sep 17 21:58:21.659743 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.658995    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/468939af-9933-402a-aaca-455a88a366d3-etc-sysconfig\") pod \"tuned-cr4rb\" (UID: \"468939af-9933-402a-aaca-455a88a366d3\") " pod="openshift-cluster-node-tuning-operator/tuned-cr4rb"
Sep 17 21:58:21.659743 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.659042    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/66effb77-e868-4e17-9bab-d9ed02f73237-host-run-multus-certs\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.659743 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.659084    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f2ab99ad-b85d-494c-967c-c4e75d08553e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9sgr6\" (UID: \"f2ab99ad-b85d-494c-967c-c4e75d08553e\") " pod="openshift-multus/multus-additional-cni-plugins-9sgr6"
Sep 17 21:58:21.659743 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.659125    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/468939af-9933-402a-aaca-455a88a366d3-etc-modprobe-d\") pod \"tuned-cr4rb\" (UID: \"468939af-9933-402a-aaca-455a88a366d3\") " pod="openshift-cluster-node-tuning-operator/tuned-cr4rb"
Sep 17 21:58:21.659743 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.659165    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/468939af-9933-402a-aaca-455a88a366d3-etc-sysctl-d\") pod \"tuned-cr4rb\" (UID: \"468939af-9933-402a-aaca-455a88a366d3\") " pod="openshift-cluster-node-tuning-operator/tuned-cr4rb"
Sep 17 21:58:21.659743 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.659207    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/468939af-9933-402a-aaca-455a88a366d3-tmp\") pod \"tuned-cr4rb\" (UID: \"468939af-9933-402a-aaca-455a88a366d3\") " pod="openshift-cluster-node-tuning-operator/tuned-cr4rb"
Sep 17 21:58:21.659743 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.659240    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d563612f-869f-4e82-9ba2-690fd3e34052-sys-fs\") pod \"aws-ebs-csi-driver-node-zgpjm\" (UID: \"d563612f-869f-4e82-9ba2-690fd3e34052\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zgpjm"
Sep 17 21:58:21.659743 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.659272    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-etc-openvswitch\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.659743 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.659311    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-run-ovn\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.659743 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.659324    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e87296a0-2567-4764-8fd6-5edadb362d0f-iptables-alerter-script\") pod \"iptables-alerter-2n8ms\" (UID: \"e87296a0-2567-4764-8fd6-5edadb362d0f\") " pod="openshift-network-operator/iptables-alerter-2n8ms"
Sep 17 21:58:21.659743 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.659337    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/838fe5eb-ddc1-4cea-8a3c-0c5f0e203174-host\") pod \"node-ca-rdqkk\" (UID: \"838fe5eb-ddc1-4cea-8a3c-0c5f0e203174\") " pod="openshift-image-registry/node-ca-rdqkk"
Sep 17 21:58:21.660303 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.659390    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f2ab99ad-b85d-494c-967c-c4e75d08553e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9sgr6\" (UID: \"f2ab99ad-b85d-494c-967c-c4e75d08553e\") " pod="openshift-multus/multus-additional-cni-plugins-9sgr6"
Sep 17 21:58:21.660303 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.659398    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/838fe5eb-ddc1-4cea-8a3c-0c5f0e203174-host\") pod \"node-ca-rdqkk\" (UID: \"838fe5eb-ddc1-4cea-8a3c-0c5f0e203174\") " pod="openshift-image-registry/node-ca-rdqkk"
Sep 17 21:58:21.660303 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.659442    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/66effb77-e868-4e17-9bab-d9ed02f73237-host-run-k8s-cni-cncf-io\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.660303 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.659506    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/66effb77-e868-4e17-9bab-d9ed02f73237-multus-daemon-config\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.660303 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.659534    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hq7w\" (UniqueName: \"kubernetes.io/projected/a236d0c6-0efa-47e2-94c1-f4ca507442c4-kube-api-access-6hq7w\") pod \"network-metrics-daemon-n7n4d\" (UID: \"a236d0c6-0efa-47e2-94c1-f4ca507442c4\") " pod="openshift-multus/network-metrics-daemon-n7n4d"
Sep 17 21:58:21.660303 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.659542    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f2ab99ad-b85d-494c-967c-c4e75d08553e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9sgr6\" (UID: \"f2ab99ad-b85d-494c-967c-c4e75d08553e\") " pod="openshift-multus/multus-additional-cni-plugins-9sgr6"
Sep 17 21:58:21.660303 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.659576    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f2ab99ad-b85d-494c-967c-c4e75d08553e-cni-binary-copy\") pod \"multus-additional-cni-plugins-9sgr6\" (UID: \"f2ab99ad-b85d-494c-967c-c4e75d08553e\") " pod="openshift-multus/multus-additional-cni-plugins-9sgr6"
Sep 17 21:58:21.660303 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.659610    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66effb77-e868-4e17-9bab-d9ed02f73237-multus-conf-dir\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.660303 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.659644    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-host-slash\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.660303 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.659672    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.660303 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.659688    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d563612f-869f-4e82-9ba2-690fd3e34052-device-dir\") pod \"aws-ebs-csi-driver-node-zgpjm\" (UID: \"d563612f-869f-4e82-9ba2-690fd3e34052\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zgpjm"
Sep 17 21:58:21.660303 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.659702    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6czqb\" (UniqueName: \"kubernetes.io/projected/d563612f-869f-4e82-9ba2-690fd3e34052-kube-api-access-6czqb\") pod \"aws-ebs-csi-driver-node-zgpjm\" (UID: \"d563612f-869f-4e82-9ba2-690fd3e34052\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zgpjm"
Sep 17 21:58:21.660303 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.659717    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84426995-45a3-420d-81b6-2676aba8686b-original-pull-secret\") pod \"global-pull-secret-syncer-vl469\" (UID: \"84426995-45a3-420d-81b6-2676aba8686b\") " pod="kube-system/global-pull-secret-syncer-vl469"
Sep 17 21:58:21.660303 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.659731    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltvjx\" (UniqueName: \"kubernetes.io/projected/468939af-9933-402a-aaca-455a88a366d3-kube-api-access-ltvjx\") pod \"tuned-cr4rb\" (UID: \"468939af-9933-402a-aaca-455a88a366d3\") " pod="openshift-cluster-node-tuning-operator/tuned-cr4rb"
Sep 17 21:58:21.660303 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.659744    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66effb77-e868-4e17-9bab-d9ed02f73237-cni-binary-copy\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.660303 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.659775    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8fvj\" (UniqueName: \"kubernetes.io/projected/66effb77-e868-4e17-9bab-d9ed02f73237-kube-api-access-v8fvj\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.660753 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.659791    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f2ab99ad-b85d-494c-967c-c4e75d08553e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9sgr6\" (UID: \"f2ab99ad-b85d-494c-967c-c4e75d08553e\") " pod="openshift-multus/multus-additional-cni-plugins-9sgr6"
Sep 17 21:58:21.660753 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.659803    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-host-cni-bin\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.660753 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.659828    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/84426995-45a3-420d-81b6-2676aba8686b-kubelet-config\") pod \"global-pull-secret-syncer-vl469\" (UID: \"84426995-45a3-420d-81b6-2676aba8686b\") " pod="kube-system/global-pull-secret-syncer-vl469"
Sep 17 21:58:21.660753 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.659853    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/468939af-9933-402a-aaca-455a88a366d3-etc-kubernetes\") pod \"tuned-cr4rb\" (UID: \"468939af-9933-402a-aaca-455a88a366d3\") " pod="openshift-cluster-node-tuning-operator/tuned-cr4rb"
Sep 17 21:58:21.660753 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.659892    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/468939af-9933-402a-aaca-455a88a366d3-etc-sysctl-conf\") pod \"tuned-cr4rb\" (UID: \"468939af-9933-402a-aaca-455a88a366d3\") " pod="openshift-cluster-node-tuning-operator/tuned-cr4rb"
Sep 17 21:58:21.660753 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.659924    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-node-log\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.660753 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.659964    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-host-cni-netd\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.660753 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.659992    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-ovnkube-script-lib\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.660753 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.660001    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f2ab99ad-b85d-494c-967c-c4e75d08553e-cni-binary-copy\") pod \"multus-additional-cni-plugins-9sgr6\" (UID: \"f2ab99ad-b85d-494c-967c-c4e75d08553e\") " pod="openshift-multus/multus-additional-cni-plugins-9sgr6"
Sep 17 21:58:21.660753 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.660014    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66effb77-e868-4e17-9bab-d9ed02f73237-host-var-lib-kubelet\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.660753 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.660043    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66effb77-e868-4e17-9bab-d9ed02f73237-etc-kubernetes\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.660753 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.660062    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-host-run-netns\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.660753 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.660077    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-ovnkube-config\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.660753 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.660091    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-run-systemd\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.660753 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.660111    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-var-lib-openvswitch\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.669037 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.669018    2555 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 21:58:21.672278 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.672257    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7zh2\" (UniqueName: \"kubernetes.io/projected/f2ab99ad-b85d-494c-967c-c4e75d08553e-kube-api-access-q7zh2\") pod \"multus-additional-cni-plugins-9sgr6\" (UID: \"f2ab99ad-b85d-494c-967c-c4e75d08553e\") " pod="openshift-multus/multus-additional-cni-plugins-9sgr6"
Sep 17 21:58:21.672353 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.672315    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t6fq\" (UniqueName: \"kubernetes.io/projected/e87296a0-2567-4764-8fd6-5edadb362d0f-kube-api-access-8t6fq\") pod \"iptables-alerter-2n8ms\" (UID: \"e87296a0-2567-4764-8fd6-5edadb362d0f\") " pod="openshift-network-operator/iptables-alerter-2n8ms"
Sep 17 21:58:21.672391 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.672348    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgs2x\" (UniqueName: \"kubernetes.io/projected/838fe5eb-ddc1-4cea-8a3c-0c5f0e203174-kube-api-access-kgs2x\") pod \"node-ca-rdqkk\" (UID: \"838fe5eb-ddc1-4cea-8a3c-0c5f0e203174\") " pod="openshift-image-registry/node-ca-rdqkk"
Sep 17 21:58:21.684290 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.684267    2555 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:58:21.717668 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:21.717645    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod745ac86812f97f62360d667f4353a619.slice/crio-5dc4e15b1e1be2f6eed4af0b42297141ce1d91ab9ebb9b3dd58c433c70d3fe48 WatchSource:0}: Error finding container 5dc4e15b1e1be2f6eed4af0b42297141ce1d91ab9ebb9b3dd58c433c70d3fe48: Status 404 returned error can't find the container with id 5dc4e15b1e1be2f6eed4af0b42297141ce1d91ab9ebb9b3dd58c433c70d3fe48
Sep 17 21:58:21.717980 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:21.717966    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd837547dfeda455efa71ea03001f07b0.slice/crio-a5e1cd6ebb9e155a68ca9eb5aa8f9d778c76b38852950f3a7dddce69198af259 WatchSource:0}: Error finding container a5e1cd6ebb9e155a68ca9eb5aa8f9d778c76b38852950f3a7dddce69198af259: Status 404 returned error can't find the container with id a5e1cd6ebb9e155a68ca9eb5aa8f9d778c76b38852950f3a7dddce69198af259
Sep 17 21:58:21.722244 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.722227    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 21:58:21.760978 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.760957    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/468939af-9933-402a-aaca-455a88a366d3-etc-kubernetes\") pod \"tuned-cr4rb\" (UID: \"468939af-9933-402a-aaca-455a88a366d3\") " pod="openshift-cluster-node-tuning-operator/tuned-cr4rb"
Sep 17 21:58:21.761082 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.760982    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/468939af-9933-402a-aaca-455a88a366d3-etc-sysctl-conf\") pod \"tuned-cr4rb\" (UID: \"468939af-9933-402a-aaca-455a88a366d3\") " pod="openshift-cluster-node-tuning-operator/tuned-cr4rb"
Sep 17 21:58:21.761082 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.760999    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-node-log\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.761082 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761012    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-host-cni-netd\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.761082 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761025    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-ovnkube-script-lib\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.761082 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761039    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66effb77-e868-4e17-9bab-d9ed02f73237-host-var-lib-kubelet\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.761082 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761054    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66effb77-e868-4e17-9bab-d9ed02f73237-etc-kubernetes\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.761082 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761074    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-host-run-netns\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.761481 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761088    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/468939af-9933-402a-aaca-455a88a366d3-etc-kubernetes\") pod \"tuned-cr4rb\" (UID: \"468939af-9933-402a-aaca-455a88a366d3\") " pod="openshift-cluster-node-tuning-operator/tuned-cr4rb"
Sep 17 21:58:21.761481 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761101    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-node-log\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.761481 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761113    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/468939af-9933-402a-aaca-455a88a366d3-etc-sysctl-conf\") pod \"tuned-cr4rb\" (UID: \"468939af-9933-402a-aaca-455a88a366d3\") " pod="openshift-cluster-node-tuning-operator/tuned-cr4rb"
Sep 17 21:58:21.761481 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761121    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-ovnkube-config\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.761481 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761142    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66effb77-e868-4e17-9bab-d9ed02f73237-etc-kubernetes\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.761481 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761152    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-run-systemd\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.761481 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761141    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66effb77-e868-4e17-9bab-d9ed02f73237-host-var-lib-kubelet\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.761481 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761176    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-var-lib-openvswitch\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.761481 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761207    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-host-run-netns\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.761481 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761220    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/468939af-9933-402a-aaca-455a88a366d3-etc-systemd\") pod \"tuned-cr4rb\" (UID: \"468939af-9933-402a-aaca-455a88a366d3\") " pod="openshift-cluster-node-tuning-operator/tuned-cr4rb"
Sep 17 21:58:21.761481 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761248    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-host-cni-netd\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.761481 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761250    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-host-kubelet\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.761481 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761262    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-var-lib-openvswitch\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.761481 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761224    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-run-systemd\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.761481 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761287    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-host-kubelet\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.761481 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761290    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-log-socket\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.761481 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761314    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/468939af-9933-402a-aaca-455a88a366d3-etc-systemd\") pod \"tuned-cr4rb\" (UID: \"468939af-9933-402a-aaca-455a88a366d3\") " pod="openshift-cluster-node-tuning-operator/tuned-cr4rb"
Sep 17 21:58:21.761481 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761332    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66effb77-e868-4e17-9bab-d9ed02f73237-cnibin\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.762357 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761363    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-log-socket\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.762357 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761388    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzdrk\" (UniqueName: \"kubernetes.io/projected/7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd-kube-api-access-tzdrk\") pod \"network-check-target-shx44\" (UID: \"7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd\") " pod="openshift-network-diagnostics/network-check-target-shx44"
Sep 17 21:58:21.762357 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761407    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkljs\" (UniqueName: \"kubernetes.io/projected/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-kube-api-access-fkljs\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.762357 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761469    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66effb77-e868-4e17-9bab-d9ed02f73237-cnibin\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.762357 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761477    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/65875e8d-a9e5-4031-bedd-b304efbafde8-konnectivity-ca\") pod \"konnectivity-agent-8sfbd\" (UID: \"65875e8d-a9e5-4031-bedd-b304efbafde8\") " pod="kube-system/konnectivity-agent-8sfbd"
Sep 17 21:58:21.762357 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761508    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfdv8\" (UniqueName: \"kubernetes.io/projected/cc6119ef-2509-4a37-9edb-1c6da2fdef14-kube-api-access-dfdv8\") pod \"node-resolver-sggv9\" (UID: \"cc6119ef-2509-4a37-9edb-1c6da2fdef14\") " pod="openshift-dns/node-resolver-sggv9"
Sep 17 21:58:21.762357 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761543    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-run-openvswitch\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.762357 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761568    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/468939af-9933-402a-aaca-455a88a366d3-var-lib-kubelet\") pod \"tuned-cr4rb\" (UID: \"468939af-9933-402a-aaca-455a88a366d3\") " pod="openshift-cluster-node-tuning-operator/tuned-cr4rb"
Sep 17 21:58:21.762357 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761591    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66effb77-e868-4e17-9bab-d9ed02f73237-system-cni-dir\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.762357 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761645    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-host-run-ovn-kubernetes\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.762357 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761677    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-ovn-node-metrics-cert\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.762357 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761703    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/468939af-9933-402a-aaca-455a88a366d3-lib-modules\") pod \"tuned-cr4rb\" (UID: \"468939af-9933-402a-aaca-455a88a366d3\") " pod="openshift-cluster-node-tuning-operator/tuned-cr4rb"
Sep 17 21:58:21.762357 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761720    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/468939af-9933-402a-aaca-455a88a366d3-var-lib-kubelet\") pod \"tuned-cr4rb\" (UID: \"468939af-9933-402a-aaca-455a88a366d3\") " pod="openshift-cluster-node-tuning-operator/tuned-cr4rb"
Sep 17 21:58:21.762357 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761721    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-ovnkube-config\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.762357 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761740    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-run-openvswitch\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.762357 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761743    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cc6119ef-2509-4a37-9edb-1c6da2fdef14-hosts-file\") pod \"node-resolver-sggv9\" (UID: \"cc6119ef-2509-4a37-9edb-1c6da2fdef14\") " pod="openshift-dns/node-resolver-sggv9"
Sep 17 21:58:21.762357 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761782    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/468939af-9933-402a-aaca-455a88a366d3-run\") pod \"tuned-cr4rb\" (UID: \"468939af-9933-402a-aaca-455a88a366d3\") " pod="openshift-cluster-node-tuning-operator/tuned-cr4rb"
Sep 17 21:58:21.763934 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761785    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-ovnkube-script-lib\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.763934 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761804    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cc6119ef-2509-4a37-9edb-1c6da2fdef14-hosts-file\") pod \"node-resolver-sggv9\" (UID: \"cc6119ef-2509-4a37-9edb-1c6da2fdef14\") " pod="openshift-dns/node-resolver-sggv9"
Sep 17 21:58:21.763934 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761806    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/66effb77-e868-4e17-9bab-d9ed02f73237-host-run-netns\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.763934 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761845    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/66effb77-e868-4e17-9bab-d9ed02f73237-host-var-lib-cni-bin\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.763934 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761858    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66effb77-e868-4e17-9bab-d9ed02f73237-system-cni-dir\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.763934 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761871    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/66effb77-e868-4e17-9bab-d9ed02f73237-hostroot\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.763934 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761808    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-host-run-ovn-kubernetes\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.763934 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761847    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/66effb77-e868-4e17-9bab-d9ed02f73237-host-run-netns\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.763934 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761907    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/468939af-9933-402a-aaca-455a88a366d3-run\") pod \"tuned-cr4rb\" (UID: \"468939af-9933-402a-aaca-455a88a366d3\") " pod="openshift-cluster-node-tuning-operator/tuned-cr4rb"
Sep 17 21:58:21.763934 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761911    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-systemd-units\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.763934 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761952    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/65875e8d-a9e5-4031-bedd-b304efbafde8-agent-certs\") pod \"konnectivity-agent-8sfbd\" (UID: \"65875e8d-a9e5-4031-bedd-b304efbafde8\") " pod="kube-system/konnectivity-agent-8sfbd"
Sep 17 21:58:21.763934 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761976    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a236d0c6-0efa-47e2-94c1-f4ca507442c4-metrics-certs\") pod \"network-metrics-daemon-n7n4d\" (UID: \"a236d0c6-0efa-47e2-94c1-f4ca507442c4\") " pod="openshift-multus/network-metrics-daemon-n7n4d"
Sep 17 21:58:21.763934 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761986    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/468939af-9933-402a-aaca-455a88a366d3-lib-modules\") pod \"tuned-cr4rb\" (UID: \"468939af-9933-402a-aaca-455a88a366d3\") " pod="openshift-cluster-node-tuning-operator/tuned-cr4rb"
Sep 17 21:58:21.763934 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761999    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d563612f-869f-4e82-9ba2-690fd3e34052-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zgpjm\" (UID: \"d563612f-869f-4e82-9ba2-690fd3e34052\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zgpjm"
Sep 17 21:58:21.763934 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761952    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-systemd-units\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.763934 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761997    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/65875e8d-a9e5-4031-bedd-b304efbafde8-konnectivity-ca\") pod \"konnectivity-agent-8sfbd\" (UID: \"65875e8d-a9e5-4031-bedd-b304efbafde8\") " pod="kube-system/konnectivity-agent-8sfbd"
Sep 17 21:58:21.763934 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.761989    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/66effb77-e868-4e17-9bab-d9ed02f73237-hostroot\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.763934 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762029    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/66effb77-e868-4e17-9bab-d9ed02f73237-host-var-lib-cni-bin\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.764692 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762055    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/468939af-9933-402a-aaca-455a88a366d3-sys\") pod \"tuned-cr4rb\" (UID: \"468939af-9933-402a-aaca-455a88a366d3\") " pod="openshift-cluster-node-tuning-operator/tuned-cr4rb"
Sep 17 21:58:21.764692 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762068    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d563612f-869f-4e82-9ba2-690fd3e34052-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zgpjm\" (UID: \"d563612f-869f-4e82-9ba2-690fd3e34052\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zgpjm"
Sep 17 21:58:21.764692 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:21.762084    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:21.764692 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762115    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/468939af-9933-402a-aaca-455a88a366d3-sys\") pod \"tuned-cr4rb\" (UID: \"468939af-9933-402a-aaca-455a88a366d3\") " pod="openshift-cluster-node-tuning-operator/tuned-cr4rb"
Sep 17 21:58:21.764692 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762083    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66effb77-e868-4e17-9bab-d9ed02f73237-multus-cni-dir\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.764692 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:21.762168    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a236d0c6-0efa-47e2-94c1-f4ca507442c4-metrics-certs podName:a236d0c6-0efa-47e2-94c1-f4ca507442c4 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:22.262126228 +0000 UTC m=+3.708171495 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a236d0c6-0efa-47e2-94c1-f4ca507442c4-metrics-certs") pod "network-metrics-daemon-n7n4d" (UID: "a236d0c6-0efa-47e2-94c1-f4ca507442c4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:21.764692 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762199    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66effb77-e868-4e17-9bab-d9ed02f73237-multus-cni-dir\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.764692 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762228    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc6119ef-2509-4a37-9edb-1c6da2fdef14-tmp-dir\") pod \"node-resolver-sggv9\" (UID: \"cc6119ef-2509-4a37-9edb-1c6da2fdef14\") " pod="openshift-dns/node-resolver-sggv9"
Sep 17 21:58:21.764692 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762243    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/84426995-45a3-420d-81b6-2676aba8686b-dbus\") pod \"global-pull-secret-syncer-vl469\" (UID: \"84426995-45a3-420d-81b6-2676aba8686b\") " pod="kube-system/global-pull-secret-syncer-vl469"
Sep 17 21:58:21.764692 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762258    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/66effb77-e868-4e17-9bab-d9ed02f73237-host-var-lib-cni-multus\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.764692 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762276    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-env-overrides\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.764692 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762319    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d563612f-869f-4e82-9ba2-690fd3e34052-socket-dir\") pod \"aws-ebs-csi-driver-node-zgpjm\" (UID: \"d563612f-869f-4e82-9ba2-690fd3e34052\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zgpjm"
Sep 17 21:58:21.764692 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762352    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d563612f-869f-4e82-9ba2-690fd3e34052-etc-selinux\") pod \"aws-ebs-csi-driver-node-zgpjm\" (UID: \"d563612f-869f-4e82-9ba2-690fd3e34052\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zgpjm"
Sep 17 21:58:21.764692 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762377    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/468939af-9933-402a-aaca-455a88a366d3-host\") pod \"tuned-cr4rb\" (UID: \"468939af-9933-402a-aaca-455a88a366d3\") " pod="openshift-cluster-node-tuning-operator/tuned-cr4rb"
Sep 17 21:58:21.764692 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762401    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/468939af-9933-402a-aaca-455a88a366d3-etc-tuned\") pod \"tuned-cr4rb\" (UID: \"468939af-9933-402a-aaca-455a88a366d3\") " pod="openshift-cluster-node-tuning-operator/tuned-cr4rb"
Sep 17 21:58:21.764692 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762423    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/66effb77-e868-4e17-9bab-d9ed02f73237-multus-socket-dir-parent\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.764692 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762448    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d563612f-869f-4e82-9ba2-690fd3e34052-registration-dir\") pod \"aws-ebs-csi-driver-node-zgpjm\" (UID: \"d563612f-869f-4e82-9ba2-690fd3e34052\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zgpjm"
Sep 17 21:58:21.765168 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762462    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d563612f-869f-4e82-9ba2-690fd3e34052-socket-dir\") pod \"aws-ebs-csi-driver-node-zgpjm\" (UID: \"d563612f-869f-4e82-9ba2-690fd3e34052\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zgpjm"
Sep 17 21:58:21.765168 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762476    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/468939af-9933-402a-aaca-455a88a366d3-etc-sysconfig\") pod \"tuned-cr4rb\" (UID: \"468939af-9933-402a-aaca-455a88a366d3\") " pod="openshift-cluster-node-tuning-operator/tuned-cr4rb"
Sep 17 21:58:21.765168 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762520    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/468939af-9933-402a-aaca-455a88a366d3-host\") pod \"tuned-cr4rb\" (UID: \"468939af-9933-402a-aaca-455a88a366d3\") " pod="openshift-cluster-node-tuning-operator/tuned-cr4rb"
Sep 17 21:58:21.765168 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762531    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/468939af-9933-402a-aaca-455a88a366d3-etc-sysconfig\") pod \"tuned-cr4rb\" (UID: \"468939af-9933-402a-aaca-455a88a366d3\") " pod="openshift-cluster-node-tuning-operator/tuned-cr4rb"
Sep 17 21:58:21.765168 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762548    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/66effb77-e868-4e17-9bab-d9ed02f73237-host-run-multus-certs\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.765168 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762557    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc6119ef-2509-4a37-9edb-1c6da2fdef14-tmp-dir\") pod \"node-resolver-sggv9\" (UID: \"cc6119ef-2509-4a37-9edb-1c6da2fdef14\") " pod="openshift-dns/node-resolver-sggv9"
Sep 17 21:58:21.765168 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762326    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/66effb77-e868-4e17-9bab-d9ed02f73237-host-var-lib-cni-multus\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.765168 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762638    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/84426995-45a3-420d-81b6-2676aba8686b-dbus\") pod \"global-pull-secret-syncer-vl469\" (UID: \"84426995-45a3-420d-81b6-2676aba8686b\") " pod="kube-system/global-pull-secret-syncer-vl469"
Sep 17 21:58:21.765168 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762643    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d563612f-869f-4e82-9ba2-690fd3e34052-etc-selinux\") pod \"aws-ebs-csi-driver-node-zgpjm\" (UID: \"d563612f-869f-4e82-9ba2-690fd3e34052\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zgpjm"
Sep 17 21:58:21.765168 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762668    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-env-overrides\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.765168 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762676    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/66effb77-e868-4e17-9bab-d9ed02f73237-host-run-multus-certs\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.765168 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762695    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/66effb77-e868-4e17-9bab-d9ed02f73237-multus-socket-dir-parent\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.765168 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762716    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/468939af-9933-402a-aaca-455a88a366d3-etc-modprobe-d\") pod \"tuned-cr4rb\" (UID: \"468939af-9933-402a-aaca-455a88a366d3\") " pod="openshift-cluster-node-tuning-operator/tuned-cr4rb"
Sep 17 21:58:21.765168 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762725    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d563612f-869f-4e82-9ba2-690fd3e34052-registration-dir\") pod \"aws-ebs-csi-driver-node-zgpjm\" (UID: \"d563612f-869f-4e82-9ba2-690fd3e34052\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zgpjm"
Sep 17 21:58:21.765168 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762739    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/468939af-9933-402a-aaca-455a88a366d3-etc-sysctl-d\") pod \"tuned-cr4rb\" (UID: \"468939af-9933-402a-aaca-455a88a366d3\") " pod="openshift-cluster-node-tuning-operator/tuned-cr4rb"
Sep 17 21:58:21.765168 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762777    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/468939af-9933-402a-aaca-455a88a366d3-tmp\") pod \"tuned-cr4rb\" (UID: \"468939af-9933-402a-aaca-455a88a366d3\") " pod="openshift-cluster-node-tuning-operator/tuned-cr4rb"
Sep 17 21:58:21.765168 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762799    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d563612f-869f-4e82-9ba2-690fd3e34052-sys-fs\") pod \"aws-ebs-csi-driver-node-zgpjm\" (UID: \"d563612f-869f-4e82-9ba2-690fd3e34052\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zgpjm"
Sep 17 21:58:21.765168 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762807    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/468939af-9933-402a-aaca-455a88a366d3-etc-modprobe-d\") pod \"tuned-cr4rb\" (UID: \"468939af-9933-402a-aaca-455a88a366d3\") " pod="openshift-cluster-node-tuning-operator/tuned-cr4rb"
Sep 17 21:58:21.765678 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762822    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-etc-openvswitch\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.765678 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762847    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-run-ovn\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.765678 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762853    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d563612f-869f-4e82-9ba2-690fd3e34052-sys-fs\") pod \"aws-ebs-csi-driver-node-zgpjm\" (UID: \"d563612f-869f-4e82-9ba2-690fd3e34052\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zgpjm"
Sep 17 21:58:21.765678 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762867    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/468939af-9933-402a-aaca-455a88a366d3-etc-sysctl-d\") pod \"tuned-cr4rb\" (UID: \"468939af-9933-402a-aaca-455a88a366d3\") " pod="openshift-cluster-node-tuning-operator/tuned-cr4rb"
Sep 17 21:58:21.765678 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762875    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/66effb77-e868-4e17-9bab-d9ed02f73237-host-run-k8s-cni-cncf-io\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.765678 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762890    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-etc-openvswitch\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.765678 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762910    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/66effb77-e868-4e17-9bab-d9ed02f73237-host-run-k8s-cni-cncf-io\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.765678 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762915    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/66effb77-e868-4e17-9bab-d9ed02f73237-multus-daemon-config\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.765678 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762930    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-run-ovn\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.765678 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762945    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hq7w\" (UniqueName: \"kubernetes.io/projected/a236d0c6-0efa-47e2-94c1-f4ca507442c4-kube-api-access-6hq7w\") pod \"network-metrics-daemon-n7n4d\" (UID: \"a236d0c6-0efa-47e2-94c1-f4ca507442c4\") " pod="openshift-multus/network-metrics-daemon-n7n4d"
Sep 17 21:58:21.765678 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762974    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66effb77-e868-4e17-9bab-d9ed02f73237-multus-conf-dir\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.765678 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.762997    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-host-slash\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.765678 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.763043    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-host-slash\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.765678 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.763054    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66effb77-e868-4e17-9bab-d9ed02f73237-multus-conf-dir\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.765678 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.763072    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.765678 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.763100    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d563612f-869f-4e82-9ba2-690fd3e34052-device-dir\") pod \"aws-ebs-csi-driver-node-zgpjm\" (UID: \"d563612f-869f-4e82-9ba2-690fd3e34052\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zgpjm"
Sep 17 21:58:21.765678 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.763123    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6czqb\" (UniqueName: \"kubernetes.io/projected/d563612f-869f-4e82-9ba2-690fd3e34052-kube-api-access-6czqb\") pod \"aws-ebs-csi-driver-node-zgpjm\" (UID: \"d563612f-869f-4e82-9ba2-690fd3e34052\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zgpjm"
Sep 17 21:58:21.766140 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.763149    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84426995-45a3-420d-81b6-2676aba8686b-original-pull-secret\") pod \"global-pull-secret-syncer-vl469\" (UID: \"84426995-45a3-420d-81b6-2676aba8686b\") " pod="kube-system/global-pull-secret-syncer-vl469"
Sep 17 21:58:21.766140 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.763166    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.766140 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.763208    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d563612f-869f-4e82-9ba2-690fd3e34052-device-dir\") pod \"aws-ebs-csi-driver-node-zgpjm\" (UID: \"d563612f-869f-4e82-9ba2-690fd3e34052\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zgpjm"
Sep 17 21:58:21.766140 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.763217    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltvjx\" (UniqueName: \"kubernetes.io/projected/468939af-9933-402a-aaca-455a88a366d3-kube-api-access-ltvjx\") pod \"tuned-cr4rb\" (UID: \"468939af-9933-402a-aaca-455a88a366d3\") " pod="openshift-cluster-node-tuning-operator/tuned-cr4rb"
Sep 17 21:58:21.766140 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.763248    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66effb77-e868-4e17-9bab-d9ed02f73237-cni-binary-copy\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.766140 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:21.763263    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:21.766140 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.763273    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8fvj\" (UniqueName: \"kubernetes.io/projected/66effb77-e868-4e17-9bab-d9ed02f73237-kube-api-access-v8fvj\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.766140 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.763298    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-host-cni-bin\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.766140 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:21.763316    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84426995-45a3-420d-81b6-2676aba8686b-original-pull-secret podName:84426995-45a3-420d-81b6-2676aba8686b nodeName:}" failed. No retries permitted until 2026-09-17 21:58:22.263301113 +0000 UTC m=+3.709346385 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/84426995-45a3-420d-81b6-2676aba8686b-original-pull-secret") pod "global-pull-secret-syncer-vl469" (UID: "84426995-45a3-420d-81b6-2676aba8686b") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:21.766140 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.763334    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-host-cni-bin\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.766140 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.763342    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/84426995-45a3-420d-81b6-2676aba8686b-kubelet-config\") pod \"global-pull-secret-syncer-vl469\" (UID: \"84426995-45a3-420d-81b6-2676aba8686b\") " pod="kube-system/global-pull-secret-syncer-vl469"
Sep 17 21:58:21.766140 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.763399    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/66effb77-e868-4e17-9bab-d9ed02f73237-multus-daemon-config\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.766140 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.763411    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/84426995-45a3-420d-81b6-2676aba8686b-kubelet-config\") pod \"global-pull-secret-syncer-vl469\" (UID: \"84426995-45a3-420d-81b6-2676aba8686b\") " pod="kube-system/global-pull-secret-syncer-vl469"
Sep 17 21:58:21.766140 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.763737    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66effb77-e868-4e17-9bab-d9ed02f73237-cni-binary-copy\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.766140 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.764452    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/65875e8d-a9e5-4031-bedd-b304efbafde8-agent-certs\") pod \"konnectivity-agent-8sfbd\" (UID: \"65875e8d-a9e5-4031-bedd-b304efbafde8\") " pod="kube-system/konnectivity-agent-8sfbd"
Sep 17 21:58:21.766140 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.764618    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/468939af-9933-402a-aaca-455a88a366d3-etc-tuned\") pod \"tuned-cr4rb\" (UID: \"468939af-9933-402a-aaca-455a88a366d3\") " pod="openshift-cluster-node-tuning-operator/tuned-cr4rb"
Sep 17 21:58:21.766140 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.764677    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-ovn-node-metrics-cert\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.766617 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.765106    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/468939af-9933-402a-aaca-455a88a366d3-tmp\") pod \"tuned-cr4rb\" (UID: \"468939af-9933-402a-aaca-455a88a366d3\") " pod="openshift-cluster-node-tuning-operator/tuned-cr4rb"
Sep 17 21:58:21.780919 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.780897    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfdv8\" (UniqueName: \"kubernetes.io/projected/cc6119ef-2509-4a37-9edb-1c6da2fdef14-kube-api-access-dfdv8\") pod \"node-resolver-sggv9\" (UID: \"cc6119ef-2509-4a37-9edb-1c6da2fdef14\") " pod="openshift-dns/node-resolver-sggv9"
Sep 17 21:58:21.781319 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:21.781305    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:21.781358 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:21.781323    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:21.781358 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:21.781332    2555 projected.go:196] Error preparing data for projected volume kube-api-access-tzdrk for pod openshift-network-diagnostics/network-check-target-shx44: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:21.781442 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:21.781430    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd-kube-api-access-tzdrk podName:7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd nodeName:}" failed. No retries permitted until 2026-09-17 21:58:22.281387183 +0000 UTC m=+3.727432435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-tzdrk" (UniqueName: "kubernetes.io/projected/7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd-kube-api-access-tzdrk") pod "network-check-target-shx44" (UID: "7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:21.783365 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.783334    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6czqb\" (UniqueName: \"kubernetes.io/projected/d563612f-869f-4e82-9ba2-690fd3e34052-kube-api-access-6czqb\") pod \"aws-ebs-csi-driver-node-zgpjm\" (UID: \"d563612f-869f-4e82-9ba2-690fd3e34052\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zgpjm"
Sep 17 21:58:21.783883 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.783852    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkljs\" (UniqueName: \"kubernetes.io/projected/4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf-kube-api-access-fkljs\") pod \"ovnkube-node-5s7f5\" (UID: \"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.784040 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.784026    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltvjx\" (UniqueName: \"kubernetes.io/projected/468939af-9933-402a-aaca-455a88a366d3-kube-api-access-ltvjx\") pod \"tuned-cr4rb\" (UID: \"468939af-9933-402a-aaca-455a88a366d3\") " pod="openshift-cluster-node-tuning-operator/tuned-cr4rb"
Sep 17 21:58:21.784189 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.784157    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8fvj\" (UniqueName: \"kubernetes.io/projected/66effb77-e868-4e17-9bab-d9ed02f73237-kube-api-access-v8fvj\") pod \"multus-cn4mv\" (UID: \"66effb77-e868-4e17-9bab-d9ed02f73237\") " pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.784291 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.784275    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hq7w\" (UniqueName: \"kubernetes.io/projected/a236d0c6-0efa-47e2-94c1-f4ca507442c4-kube-api-access-6hq7w\") pod \"network-metrics-daemon-n7n4d\" (UID: \"a236d0c6-0efa-47e2-94c1-f4ca507442c4\") " pod="openshift-multus/network-metrics-daemon-n7n4d"
Sep 17 21:58:21.880228 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.880160    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rdqkk"
Sep 17 21:58:21.889683 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:21.889660    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod838fe5eb_ddc1_4cea_8a3c_0c5f0e203174.slice/crio-d97da64bd5ec4c62c565b400310313ed3ed28f14589ee0b37808f4af0583404f WatchSource:0}: Error finding container d97da64bd5ec4c62c565b400310313ed3ed28f14589ee0b37808f4af0583404f: Status 404 returned error can't find the container with id d97da64bd5ec4c62c565b400310313ed3ed28f14589ee0b37808f4af0583404f
Sep 17 21:58:21.895925 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.895907    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9sgr6"
Sep 17 21:58:21.901722 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.901706    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2n8ms"
Sep 17 21:58:21.903414 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:21.903395    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2ab99ad_b85d_494c_967c_c4e75d08553e.slice/crio-58f810d3ff3ab399ed1472bb8cdb674780e27334700aadc0c8f804eced1c8f25 WatchSource:0}: Error finding container 58f810d3ff3ab399ed1472bb8cdb674780e27334700aadc0c8f804eced1c8f25: Status 404 returned error can't find the container with id 58f810d3ff3ab399ed1472bb8cdb674780e27334700aadc0c8f804eced1c8f25
Sep 17 21:58:21.909555 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:21.909535    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode87296a0_2567_4764_8fd6_5edadb362d0f.slice/crio-88a3abb542102e03c91e047b24182999afb06efcee98e26a3940a011dc8f80ad WatchSource:0}: Error finding container 88a3abb542102e03c91e047b24182999afb06efcee98e26a3940a011dc8f80ad: Status 404 returned error can't find the container with id 88a3abb542102e03c91e047b24182999afb06efcee98e26a3940a011dc8f80ad
Sep 17 21:58:21.913876 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.913860    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cr4rb"
Sep 17 21:58:21.918779 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.918757    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cn4mv"
Sep 17 21:58:21.922631 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:21.922614    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468939af_9933_402a_aaca_455a88a366d3.slice/crio-a43a82e54f0e8df73a736bd3c42649786dcd5a599364c614997761b16c7437e3 WatchSource:0}: Error finding container a43a82e54f0e8df73a736bd3c42649786dcd5a599364c614997761b16c7437e3: Status 404 returned error can't find the container with id a43a82e54f0e8df73a736bd3c42649786dcd5a599364c614997761b16c7437e3
Sep 17 21:58:21.927907 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:21.927885    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66effb77_e868_4e17_9bab_d9ed02f73237.slice/crio-c345bb20483be8804f43b3d3c1173fc140ad52cfe868aba959e0118e92a7ca2f WatchSource:0}: Error finding container c345bb20483be8804f43b3d3c1173fc140ad52cfe868aba959e0118e92a7ca2f: Status 404 returned error can't find the container with id c345bb20483be8804f43b3d3c1173fc140ad52cfe868aba959e0118e92a7ca2f
Sep 17 21:58:21.934370 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.934354    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zgpjm"
Sep 17 21:58:21.939894 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.939877    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sggv9"
Sep 17 21:58:21.942229 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:21.942169    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd563612f_869f_4e82_9ba2_690fd3e34052.slice/crio-1f6a3834a7d04a24934a5f8a2d7a1289153cf4e75944d8d0c1ed0c78d09251fc WatchSource:0}: Error finding container 1f6a3834a7d04a24934a5f8a2d7a1289153cf4e75944d8d0c1ed0c78d09251fc: Status 404 returned error can't find the container with id 1f6a3834a7d04a24934a5f8a2d7a1289153cf4e75944d8d0c1ed0c78d09251fc
Sep 17 21:58:21.947741 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:21.947721    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6119ef_2509_4a37_9edb_1c6da2fdef14.slice/crio-06e99e373ecab4c7c0525599456b963368cd4021af304991bd336c9042ecb22c WatchSource:0}: Error finding container 06e99e373ecab4c7c0525599456b963368cd4021af304991bd336c9042ecb22c: Status 404 returned error can't find the container with id 06e99e373ecab4c7c0525599456b963368cd4021af304991bd336c9042ecb22c
Sep 17 21:58:21.966506 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.966486    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:21.971124 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:21.971109    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8sfbd"
Sep 17 21:58:21.974464 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:21.974446    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e6c570d_7932_49e8_b0dc_6cf8dd77d1cf.slice/crio-41cd6dee5a158a6ec283a1177ec40197a62cadd071f222089d56fa8c249a421f WatchSource:0}: Error finding container 41cd6dee5a158a6ec283a1177ec40197a62cadd071f222089d56fa8c249a421f: Status 404 returned error can't find the container with id 41cd6dee5a158a6ec283a1177ec40197a62cadd071f222089d56fa8c249a421f
Sep 17 21:58:21.978939 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:21.978922    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65875e8d_a9e5_4031_bedd_b304efbafde8.slice/crio-eede7792a6fea07073939cf7b940683821c729971767df948a29326d5ce68f33 WatchSource:0}: Error finding container eede7792a6fea07073939cf7b940683821c729971767df948a29326d5ce68f33: Status 404 returned error can't find the container with id eede7792a6fea07073939cf7b940683821c729971767df948a29326d5ce68f33
Sep 17 21:58:22.266950 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:22.266860    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a236d0c6-0efa-47e2-94c1-f4ca507442c4-metrics-certs\") pod \"network-metrics-daemon-n7n4d\" (UID: \"a236d0c6-0efa-47e2-94c1-f4ca507442c4\") " pod="openshift-multus/network-metrics-daemon-n7n4d"
Sep 17 21:58:22.266950 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:22.266946    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84426995-45a3-420d-81b6-2676aba8686b-original-pull-secret\") pod \"global-pull-secret-syncer-vl469\" (UID: \"84426995-45a3-420d-81b6-2676aba8686b\") " pod="kube-system/global-pull-secret-syncer-vl469"
Sep 17 21:58:22.267156 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:22.266986    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:22.267156 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:22.267063    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a236d0c6-0efa-47e2-94c1-f4ca507442c4-metrics-certs podName:a236d0c6-0efa-47e2-94c1-f4ca507442c4 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:23.267044745 +0000 UTC m=+4.713090013 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a236d0c6-0efa-47e2-94c1-f4ca507442c4-metrics-certs") pod "network-metrics-daemon-n7n4d" (UID: "a236d0c6-0efa-47e2-94c1-f4ca507442c4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:22.267156 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:22.267076    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:22.267156 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:22.267148    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84426995-45a3-420d-81b6-2676aba8686b-original-pull-secret podName:84426995-45a3-420d-81b6-2676aba8686b nodeName:}" failed. No retries permitted until 2026-09-17 21:58:23.26711384 +0000 UTC m=+4.713159089 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/84426995-45a3-420d-81b6-2676aba8686b-original-pull-secret") pod "global-pull-secret-syncer-vl469" (UID: "84426995-45a3-420d-81b6-2676aba8686b") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:22.367886 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:22.367847    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzdrk\" (UniqueName: \"kubernetes.io/projected/7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd-kube-api-access-tzdrk\") pod \"network-check-target-shx44\" (UID: \"7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd\") " pod="openshift-network-diagnostics/network-check-target-shx44"
Sep 17 21:58:22.368056 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:22.368043    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:22.368121 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:22.368067    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:22.368121 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:22.368081    2555 projected.go:196] Error preparing data for projected volume kube-api-access-tzdrk for pod openshift-network-diagnostics/network-check-target-shx44: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:22.368236 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:22.368143    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd-kube-api-access-tzdrk podName:7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd nodeName:}" failed. No retries permitted until 2026-09-17 21:58:23.368125413 +0000 UTC m=+4.814170677 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-tzdrk" (UniqueName: "kubernetes.io/projected/7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd-kube-api-access-tzdrk") pod "network-check-target-shx44" (UID: "7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:22.535248 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:22.534466    2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:58:22.595344 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:22.595262    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 21:53:21 +0000 UTC" deadline="2028-05-11 21:57:20.168014876 +0000 UTC"
Sep 17 21:58:22.595344 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:22.595298    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14447h58m57.572721525s"
Sep 17 21:58:22.675489 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:22.675459    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-shx44"
Sep 17 21:58:22.675668 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:22.675585    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-shx44" podUID="7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd"
Sep 17 21:58:22.696761 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:22.696707    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sggv9" event={"ID":"cc6119ef-2509-4a37-9edb-1c6da2fdef14","Type":"ContainerStarted","Data":"06e99e373ecab4c7c0525599456b963368cd4021af304991bd336c9042ecb22c"}
Sep 17 21:58:22.697440 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:22.697371    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zgpjm" event={"ID":"d563612f-869f-4e82-9ba2-690fd3e34052","Type":"ContainerStarted","Data":"1f6a3834a7d04a24934a5f8a2d7a1289153cf4e75944d8d0c1ed0c78d09251fc"}
Sep 17 21:58:22.719540 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:22.719513    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cn4mv" event={"ID":"66effb77-e868-4e17-9bab-d9ed02f73237","Type":"ContainerStarted","Data":"c345bb20483be8804f43b3d3c1173fc140ad52cfe868aba959e0118e92a7ca2f"}
Sep 17 21:58:22.737533 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:22.737503    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cr4rb" event={"ID":"468939af-9933-402a-aaca-455a88a366d3","Type":"ContainerStarted","Data":"a43a82e54f0e8df73a736bd3c42649786dcd5a599364c614997761b16c7437e3"}
Sep 17 21:58:22.767306 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:22.767266    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2n8ms" event={"ID":"e87296a0-2567-4764-8fd6-5edadb362d0f","Type":"ContainerStarted","Data":"88a3abb542102e03c91e047b24182999afb06efcee98e26a3940a011dc8f80ad"}
Sep 17 21:58:22.782320 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:22.782292    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9sgr6" event={"ID":"f2ab99ad-b85d-494c-967c-c4e75d08553e","Type":"ContainerStarted","Data":"58f810d3ff3ab399ed1472bb8cdb674780e27334700aadc0c8f804eced1c8f25"}
Sep 17 21:58:22.810340 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:22.810274    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8sfbd" event={"ID":"65875e8d-a9e5-4031-bedd-b304efbafde8","Type":"ContainerStarted","Data":"eede7792a6fea07073939cf7b940683821c729971767df948a29326d5ce68f33"}
Sep 17 21:58:22.825394 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:22.825369    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5" event={"ID":"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf","Type":"ContainerStarted","Data":"41cd6dee5a158a6ec283a1177ec40197a62cadd071f222089d56fa8c249a421f"}
Sep 17 21:58:22.842856 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:22.842828    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rdqkk" event={"ID":"838fe5eb-ddc1-4cea-8a3c-0c5f0e203174","Type":"ContainerStarted","Data":"d97da64bd5ec4c62c565b400310313ed3ed28f14589ee0b37808f4af0583404f"}
Sep 17 21:58:22.862397 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:22.862369    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-157.ec2.internal" event={"ID":"745ac86812f97f62360d667f4353a619","Type":"ContainerStarted","Data":"5dc4e15b1e1be2f6eed4af0b42297141ce1d91ab9ebb9b3dd58c433c70d3fe48"}
Sep 17 21:58:22.886545 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:22.886516    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-157.ec2.internal" event={"ID":"d837547dfeda455efa71ea03001f07b0","Type":"ContainerStarted","Data":"a5e1cd6ebb9e155a68ca9eb5aa8f9d778c76b38852950f3a7dddce69198af259"}
Sep 17 21:58:23.275040 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:23.274954    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84426995-45a3-420d-81b6-2676aba8686b-original-pull-secret\") pod \"global-pull-secret-syncer-vl469\" (UID: \"84426995-45a3-420d-81b6-2676aba8686b\") " pod="kube-system/global-pull-secret-syncer-vl469"
Sep 17 21:58:23.275207 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:23.275049    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a236d0c6-0efa-47e2-94c1-f4ca507442c4-metrics-certs\") pod \"network-metrics-daemon-n7n4d\" (UID: \"a236d0c6-0efa-47e2-94c1-f4ca507442c4\") " pod="openshift-multus/network-metrics-daemon-n7n4d"
Sep 17 21:58:23.275207 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:23.275198    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:23.275326 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:23.275271    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a236d0c6-0efa-47e2-94c1-f4ca507442c4-metrics-certs podName:a236d0c6-0efa-47e2-94c1-f4ca507442c4 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:25.27524803 +0000 UTC m=+6.721293281 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a236d0c6-0efa-47e2-94c1-f4ca507442c4-metrics-certs") pod "network-metrics-daemon-n7n4d" (UID: "a236d0c6-0efa-47e2-94c1-f4ca507442c4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:23.275746 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:23.275721    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:23.275837 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:23.275790    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84426995-45a3-420d-81b6-2676aba8686b-original-pull-secret podName:84426995-45a3-420d-81b6-2676aba8686b nodeName:}" failed. No retries permitted until 2026-09-17 21:58:25.275773264 +0000 UTC m=+6.721818514 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/84426995-45a3-420d-81b6-2676aba8686b-original-pull-secret") pod "global-pull-secret-syncer-vl469" (UID: "84426995-45a3-420d-81b6-2676aba8686b") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:23.375836 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:23.375787    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzdrk\" (UniqueName: \"kubernetes.io/projected/7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd-kube-api-access-tzdrk\") pod \"network-check-target-shx44\" (UID: \"7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd\") " pod="openshift-network-diagnostics/network-check-target-shx44"
Sep 17 21:58:23.376061 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:23.376004    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:23.376061 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:23.376028    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:23.376061 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:23.376042    2555 projected.go:196] Error preparing data for projected volume kube-api-access-tzdrk for pod openshift-network-diagnostics/network-check-target-shx44: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:23.376271 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:23.376104    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd-kube-api-access-tzdrk podName:7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd nodeName:}" failed. No retries permitted until 2026-09-17 21:58:25.376086392 +0000 UTC m=+6.822131643 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-tzdrk" (UniqueName: "kubernetes.io/projected/7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd-kube-api-access-tzdrk") pod "network-check-target-shx44" (UID: "7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:23.595910 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:23.595786    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 21:53:21 +0000 UTC" deadline="2028-06-10 07:27:10.305122866 +0000 UTC"
Sep 17 21:58:23.595910 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:23.595850    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15153h28m46.709277088s"
Sep 17 21:58:23.601755 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:23.601726    2555 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:58:23.673013 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:23.672959    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl469"
Sep 17 21:58:23.673221 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:23.673092    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl469" podUID="84426995-45a3-420d-81b6-2676aba8686b"
Sep 17 21:58:23.673529 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:23.673508    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7n4d"
Sep 17 21:58:23.673652 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:23.673631    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7n4d" podUID="a236d0c6-0efa-47e2-94c1-f4ca507442c4"
Sep 17 21:58:24.158053 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:24.158024    2555 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:58:24.678779 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:24.678745    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-shx44"
Sep 17 21:58:24.679227 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:24.678876    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-shx44" podUID="7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd"
Sep 17 21:58:25.293215 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:25.292542    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a236d0c6-0efa-47e2-94c1-f4ca507442c4-metrics-certs\") pod \"network-metrics-daemon-n7n4d\" (UID: \"a236d0c6-0efa-47e2-94c1-f4ca507442c4\") " pod="openshift-multus/network-metrics-daemon-n7n4d"
Sep 17 21:58:25.293215 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:25.292614    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84426995-45a3-420d-81b6-2676aba8686b-original-pull-secret\") pod \"global-pull-secret-syncer-vl469\" (UID: \"84426995-45a3-420d-81b6-2676aba8686b\") " pod="kube-system/global-pull-secret-syncer-vl469"
Sep 17 21:58:25.293215 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:25.292744    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:25.293215 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:25.292807    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84426995-45a3-420d-81b6-2676aba8686b-original-pull-secret podName:84426995-45a3-420d-81b6-2676aba8686b nodeName:}" failed. No retries permitted until 2026-09-17 21:58:29.2927886 +0000 UTC m=+10.738833919 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/84426995-45a3-420d-81b6-2676aba8686b-original-pull-secret") pod "global-pull-secret-syncer-vl469" (UID: "84426995-45a3-420d-81b6-2676aba8686b") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:25.293215 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:25.293053    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:25.293215 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:25.293115    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a236d0c6-0efa-47e2-94c1-f4ca507442c4-metrics-certs podName:a236d0c6-0efa-47e2-94c1-f4ca507442c4 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:29.293098054 +0000 UTC m=+10.739143307 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a236d0c6-0efa-47e2-94c1-f4ca507442c4-metrics-certs") pod "network-metrics-daemon-n7n4d" (UID: "a236d0c6-0efa-47e2-94c1-f4ca507442c4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:25.392985 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:25.392950    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzdrk\" (UniqueName: \"kubernetes.io/projected/7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd-kube-api-access-tzdrk\") pod \"network-check-target-shx44\" (UID: \"7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd\") " pod="openshift-network-diagnostics/network-check-target-shx44"
Sep 17 21:58:25.393166 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:25.393141    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:25.393260 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:25.393174    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:25.393260 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:25.393201    2555 projected.go:196] Error preparing data for projected volume kube-api-access-tzdrk for pod openshift-network-diagnostics/network-check-target-shx44: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:25.393370 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:25.393259    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd-kube-api-access-tzdrk podName:7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd nodeName:}" failed. No retries permitted until 2026-09-17 21:58:29.393241825 +0000 UTC m=+10.839287088 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-tzdrk" (UniqueName: "kubernetes.io/projected/7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd-kube-api-access-tzdrk") pod "network-check-target-shx44" (UID: "7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:25.673737 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:25.673174    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl469"
Sep 17 21:58:25.673737 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:25.673329    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl469" podUID="84426995-45a3-420d-81b6-2676aba8686b"
Sep 17 21:58:25.673737 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:25.673360    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7n4d"
Sep 17 21:58:25.673737 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:25.673492    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7n4d" podUID="a236d0c6-0efa-47e2-94c1-f4ca507442c4"
Sep 17 21:58:26.672502 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:26.672469    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-shx44"
Sep 17 21:58:26.673066 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:26.672600    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-shx44" podUID="7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd"
Sep 17 21:58:27.672756 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:27.672722    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl469"
Sep 17 21:58:27.673223 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:27.672772    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7n4d"
Sep 17 21:58:27.673223 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:27.672870    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl469" podUID="84426995-45a3-420d-81b6-2676aba8686b"
Sep 17 21:58:27.673223 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:27.673002    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7n4d" podUID="a236d0c6-0efa-47e2-94c1-f4ca507442c4"
Sep 17 21:58:28.677039 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:28.676811    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-shx44"
Sep 17 21:58:28.677039 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:28.676924    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-shx44" podUID="7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd"
Sep 17 21:58:29.328343 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:29.328099    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84426995-45a3-420d-81b6-2676aba8686b-original-pull-secret\") pod \"global-pull-secret-syncer-vl469\" (UID: \"84426995-45a3-420d-81b6-2676aba8686b\") " pod="kube-system/global-pull-secret-syncer-vl469"
Sep 17 21:58:29.328343 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:29.328204    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a236d0c6-0efa-47e2-94c1-f4ca507442c4-metrics-certs\") pod \"network-metrics-daemon-n7n4d\" (UID: \"a236d0c6-0efa-47e2-94c1-f4ca507442c4\") " pod="openshift-multus/network-metrics-daemon-n7n4d"
Sep 17 21:58:29.328343 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:29.328324    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:29.328621 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:29.328384    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84426995-45a3-420d-81b6-2676aba8686b-original-pull-secret podName:84426995-45a3-420d-81b6-2676aba8686b nodeName:}" failed. No retries permitted until 2026-09-17 21:58:37.328366924 +0000 UTC m=+18.774412187 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/84426995-45a3-420d-81b6-2676aba8686b-original-pull-secret") pod "global-pull-secret-syncer-vl469" (UID: "84426995-45a3-420d-81b6-2676aba8686b") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:29.328720 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:29.328324    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:29.328774 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:29.328752    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a236d0c6-0efa-47e2-94c1-f4ca507442c4-metrics-certs podName:a236d0c6-0efa-47e2-94c1-f4ca507442c4 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:37.328733715 +0000 UTC m=+18.774778979 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a236d0c6-0efa-47e2-94c1-f4ca507442c4-metrics-certs") pod "network-metrics-daemon-n7n4d" (UID: "a236d0c6-0efa-47e2-94c1-f4ca507442c4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:29.429394 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:29.429358    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzdrk\" (UniqueName: \"kubernetes.io/projected/7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd-kube-api-access-tzdrk\") pod \"network-check-target-shx44\" (UID: \"7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd\") " pod="openshift-network-diagnostics/network-check-target-shx44"
Sep 17 21:58:29.429681 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:29.429593    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:29.429681 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:29.429621    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:29.429681 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:29.429634    2555 projected.go:196] Error preparing data for projected volume kube-api-access-tzdrk for pod openshift-network-diagnostics/network-check-target-shx44: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:29.429868 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:29.429706    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd-kube-api-access-tzdrk podName:7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd nodeName:}" failed. No retries permitted until 2026-09-17 21:58:37.429685553 +0000 UTC m=+18.875730820 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-tzdrk" (UniqueName: "kubernetes.io/projected/7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd-kube-api-access-tzdrk") pod "network-check-target-shx44" (UID: "7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:29.673782 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:29.673622    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl469"
Sep 17 21:58:29.673782 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:29.673761    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl469" podUID="84426995-45a3-420d-81b6-2676aba8686b"
Sep 17 21:58:29.674129 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:29.674089    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7n4d"
Sep 17 21:58:29.674252 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:29.674230    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7n4d" podUID="a236d0c6-0efa-47e2-94c1-f4ca507442c4"
Sep 17 21:58:30.673585 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:30.673552    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-shx44"
Sep 17 21:58:30.674027 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:30.673675    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-shx44" podUID="7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd"
Sep 17 21:58:31.673375 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:31.673339    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7n4d"
Sep 17 21:58:31.673555 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:31.673352    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl469"
Sep 17 21:58:31.673555 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:31.673477    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7n4d" podUID="a236d0c6-0efa-47e2-94c1-f4ca507442c4"
Sep 17 21:58:31.673652 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:31.673570    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl469" podUID="84426995-45a3-420d-81b6-2676aba8686b"
Sep 17 21:58:32.675491 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:32.675459    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-shx44"
Sep 17 21:58:32.675970 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:32.675571    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-shx44" podUID="7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd"
Sep 17 21:58:33.672521 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:33.672492    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl469"
Sep 17 21:58:33.672697 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:33.672492    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7n4d"
Sep 17 21:58:33.672697 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:33.672590    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl469" podUID="84426995-45a3-420d-81b6-2676aba8686b"
Sep 17 21:58:33.672823 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:33.672713    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7n4d" podUID="a236d0c6-0efa-47e2-94c1-f4ca507442c4"
Sep 17 21:58:34.673010 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:34.672973    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-shx44"
Sep 17 21:58:34.673583 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:34.673095    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-shx44" podUID="7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd"
Sep 17 21:58:35.672510 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:35.672478    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7n4d"
Sep 17 21:58:35.672693 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:35.672651    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7n4d" podUID="a236d0c6-0efa-47e2-94c1-f4ca507442c4"
Sep 17 21:58:35.672774 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:35.672684    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl469"
Sep 17 21:58:35.672829 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:35.672789    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl469" podUID="84426995-45a3-420d-81b6-2676aba8686b"
Sep 17 21:58:36.673373 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:36.673337    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-shx44"
Sep 17 21:58:36.673877 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:36.673475    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-shx44" podUID="7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd"
Sep 17 21:58:37.382422 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:37.382384    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a236d0c6-0efa-47e2-94c1-f4ca507442c4-metrics-certs\") pod \"network-metrics-daemon-n7n4d\" (UID: \"a236d0c6-0efa-47e2-94c1-f4ca507442c4\") " pod="openshift-multus/network-metrics-daemon-n7n4d"
Sep 17 21:58:37.382635 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:37.382438    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84426995-45a3-420d-81b6-2676aba8686b-original-pull-secret\") pod \"global-pull-secret-syncer-vl469\" (UID: \"84426995-45a3-420d-81b6-2676aba8686b\") " pod="kube-system/global-pull-secret-syncer-vl469"
Sep 17 21:58:37.382635 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:37.382548    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:37.382635 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:37.382614    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84426995-45a3-420d-81b6-2676aba8686b-original-pull-secret podName:84426995-45a3-420d-81b6-2676aba8686b nodeName:}" failed. No retries permitted until 2026-09-17 21:58:53.382597633 +0000 UTC m=+34.828642897 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/84426995-45a3-420d-81b6-2676aba8686b-original-pull-secret") pod "global-pull-secret-syncer-vl469" (UID: "84426995-45a3-420d-81b6-2676aba8686b") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:37.382795 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:37.382548    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:37.382795 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:37.382732    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a236d0c6-0efa-47e2-94c1-f4ca507442c4-metrics-certs podName:a236d0c6-0efa-47e2-94c1-f4ca507442c4 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:53.382708977 +0000 UTC m=+34.828754227 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a236d0c6-0efa-47e2-94c1-f4ca507442c4-metrics-certs") pod "network-metrics-daemon-n7n4d" (UID: "a236d0c6-0efa-47e2-94c1-f4ca507442c4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:37.483128 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:37.483093    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzdrk\" (UniqueName: \"kubernetes.io/projected/7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd-kube-api-access-tzdrk\") pod \"network-check-target-shx44\" (UID: \"7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd\") " pod="openshift-network-diagnostics/network-check-target-shx44"
Sep 17 21:58:37.483293 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:37.483277    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:37.483337 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:37.483297    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:37.483337 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:37.483307    2555 projected.go:196] Error preparing data for projected volume kube-api-access-tzdrk for pod openshift-network-diagnostics/network-check-target-shx44: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:37.483450 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:37.483367    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd-kube-api-access-tzdrk podName:7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd nodeName:}" failed. No retries permitted until 2026-09-17 21:58:53.483352134 +0000 UTC m=+34.929397400 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-tzdrk" (UniqueName: "kubernetes.io/projected/7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd-kube-api-access-tzdrk") pod "network-check-target-shx44" (UID: "7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:37.673604 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:37.673528    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7n4d"
Sep 17 21:58:37.673604 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:37.673558    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl469"
Sep 17 21:58:37.674059 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:37.673683    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7n4d" podUID="a236d0c6-0efa-47e2-94c1-f4ca507442c4"
Sep 17 21:58:37.674059 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:37.673797    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl469" podUID="84426995-45a3-420d-81b6-2676aba8686b"
Sep 17 21:58:38.672789 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:38.672757    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-shx44"
Sep 17 21:58:38.672952 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:38.672892    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-shx44" podUID="7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd"
Sep 17 21:58:39.673281 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:39.673249    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7n4d"
Sep 17 21:58:39.673724 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:39.673249    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl469"
Sep 17 21:58:39.673724 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:39.673388    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7n4d" podUID="a236d0c6-0efa-47e2-94c1-f4ca507442c4"
Sep 17 21:58:39.673724 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:39.673465    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl469" podUID="84426995-45a3-420d-81b6-2676aba8686b"
Sep 17 21:58:40.674338 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:40.674151    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-shx44"
Sep 17 21:58:40.674909 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:40.674882    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-shx44" podUID="7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd"
Sep 17 21:58:40.928467 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:40.928441    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cn4mv" event={"ID":"66effb77-e868-4e17-9bab-d9ed02f73237","Type":"ContainerStarted","Data":"b203b599b5298e9f3523c75ed8a14e1630fb8c8d3452fae1bd1f1dfd4c922cae"}
Sep 17 21:58:40.930023 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:40.929998    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cr4rb" event={"ID":"468939af-9933-402a-aaca-455a88a366d3","Type":"ContainerStarted","Data":"240bc9a7ace379d885abac13aae4d84ae348bd3fb96e50a7881c29eb65f7cff2"}
Sep 17 21:58:40.932390 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:40.932367    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5" event={"ID":"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf","Type":"ContainerStarted","Data":"ac8be9121f7856807b544802ab05bda9559286dcf15d83898835101bf174e6e7"}
Sep 17 21:58:40.932480 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:40.932439    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5" event={"ID":"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf","Type":"ContainerStarted","Data":"1ecb20093e8fc5f064c313e085df738d6272fcdd91098569028a7d2b43c8d9a5"}
Sep 17 21:58:40.932537 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:40.932489    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5" event={"ID":"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf","Type":"ContainerStarted","Data":"c7d012fc2745d2275206ec10fcb220b068898a991d241405dcc6d28c409983f2"}
Sep 17 21:58:40.936047 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:40.936013    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-157.ec2.internal" event={"ID":"745ac86812f97f62360d667f4353a619","Type":"ContainerStarted","Data":"1b7d723bb1be61cde1bec108f6adbab97ac75ec2dce72c627bdc3b96f4578dc4"}
Sep 17 21:58:40.946918 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:40.946880    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-cn4mv" podStartSLOduration=2.409366292 podStartE2EDuration="20.946868653s" totalImagesPullingTime="18.537502361s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:40.946333342 +0000 UTC m=+22.392378615" watchObservedRunningTime="2026-09-17 21:58:40.946868653 +0000 UTC m=+22.392913924"
Sep 17 21:58:40.959923 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:40.959773    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-157.ec2.internal" podStartSLOduration=19.959756562 podStartE2EDuration="19.959756562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:40.958883563 +0000 UTC m=+22.404928839" watchObservedRunningTime="2026-09-17 21:58:40.959756562 +0000 UTC m=+22.405801836"
Sep 17 21:58:40.975743 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:40.975707    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-cr4rb" podStartSLOduration=2.433894434 podStartE2EDuration="20.975694651s" totalImagesPullingTime="18.541800217s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:40.975267557 +0000 UTC m=+22.421312829" watchObservedRunningTime="2026-09-17 21:58:40.975694651 +0000 UTC m=+22.421739923"
Sep 17 21:58:41.672673 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:41.672470    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl469"
Sep 17 21:58:41.672815 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:41.672525    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7n4d"
Sep 17 21:58:41.672815 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:41.672713    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl469" podUID="84426995-45a3-420d-81b6-2676aba8686b"
Sep 17 21:58:41.672815 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:41.672790    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7n4d" podUID="a236d0c6-0efa-47e2-94c1-f4ca507442c4"
Sep 17 21:58:41.948213 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:41.948157    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s7f5_4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf/ovn-acl-logging/0.log"
Sep 17 21:58:41.948616 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:41.948592    2555 generic.go:356] "Generic (PLEG): container finished" podID="4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf" containerID="1ecb20093e8fc5f064c313e085df738d6272fcdd91098569028a7d2b43c8d9a5" exitCode=1
Sep 17 21:58:41.948705 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:41.948679    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5" event={"ID":"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf","Type":"ContainerDied","Data":"1ecb20093e8fc5f064c313e085df738d6272fcdd91098569028a7d2b43c8d9a5"}
Sep 17 21:58:41.948826 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:41.948808    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5" event={"ID":"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf","Type":"ContainerStarted","Data":"4e755645a4eabb822c5823204d0bead2a2d74486c65164702e02f40e0b2f5f48"}
Sep 17 21:58:41.948882 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:41.948869    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5" event={"ID":"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf","Type":"ContainerStarted","Data":"c15c37a90828a2fc1450fe8e56859c8ad411466d33365fa9a417a6d78de0903b"}
Sep 17 21:58:41.948920 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:41.948913    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5" event={"ID":"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf","Type":"ContainerStarted","Data":"0223b4b0a3949e1d4d3be39662d6f4eb4733bda3b41c50dfe2cb1030c4131b64"}
Sep 17 21:58:41.950176 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:41.950151    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2n8ms" event={"ID":"e87296a0-2567-4764-8fd6-5edadb362d0f","Type":"ContainerStarted","Data":"e9a49f35774b3dec978e8c64cbc8ef7f4fdea397cf918aee9a86ba7a70109d51"}
Sep 17 21:58:42.672985 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:42.672959    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-shx44"
Sep 17 21:58:42.673119 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:42.673084    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-shx44" podUID="7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd"
Sep 17 21:58:42.953334 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:42.953304    2555 generic.go:356] "Generic (PLEG): container finished" podID="f2ab99ad-b85d-494c-967c-c4e75d08553e" containerID="b0a9f2d8eb62ee2d26fe82470ddd2e2ee188562a74c60f2d40d690053bb8d159" exitCode=0
Sep 17 21:58:42.954022 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:42.953380    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9sgr6" event={"ID":"f2ab99ad-b85d-494c-967c-c4e75d08553e","Type":"ContainerDied","Data":"b0a9f2d8eb62ee2d26fe82470ddd2e2ee188562a74c60f2d40d690053bb8d159"}
Sep 17 21:58:42.955027 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:42.955005    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8sfbd" event={"ID":"65875e8d-a9e5-4031-bedd-b304efbafde8","Type":"ContainerStarted","Data":"b635cdd10fe7a54a0da1043dc6820f4d62671a5c7b5050b04aef7313d91c7788"}
Sep 17 21:58:42.956346 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:42.956327    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rdqkk" event={"ID":"838fe5eb-ddc1-4cea-8a3c-0c5f0e203174","Type":"ContainerStarted","Data":"4b622749a888d3059560796fefe404b27375e3ec8306bad2555911c77701fd0a"}
Sep 17 21:58:42.958148 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:42.957869    2555 generic.go:356] "Generic (PLEG): container finished" podID="d837547dfeda455efa71ea03001f07b0" containerID="df0817d87ad52778852f0c1300b995bea9d0a0d21f3c6076e2d7983cd6c64120" exitCode=0
Sep 17 21:58:42.958148 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:42.957908    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-157.ec2.internal" event={"ID":"d837547dfeda455efa71ea03001f07b0","Type":"ContainerDied","Data":"df0817d87ad52778852f0c1300b995bea9d0a0d21f3c6076e2d7983cd6c64120"}
Sep 17 21:58:42.959632 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:42.959593    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sggv9" event={"ID":"cc6119ef-2509-4a37-9edb-1c6da2fdef14","Type":"ContainerStarted","Data":"5757f090d94c5a13483070e2b940092d427c7f1271508d0f204c2f66dd728236"}
Sep 17 21:58:42.961040 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:42.961022    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zgpjm" event={"ID":"d563612f-869f-4e82-9ba2-690fd3e34052","Type":"ContainerStarted","Data":"7e9df302900464786f97ed9fa3e08495eca571dc78001bda1535bf1a246ab08d"}
Sep 17 21:58:42.976257 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:42.976223    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2n8ms" podStartSLOduration=4.420867271 podStartE2EDuration="22.976211142s" totalImagesPullingTime="18.555343871s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:41.964769711 +0000 UTC m=+23.410814983" watchObservedRunningTime="2026-09-17 21:58:42.976211142 +0000 UTC m=+24.422256415"
Sep 17 21:58:42.990722 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:42.990681    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8sfbd" podStartSLOduration=3.526705688 podStartE2EDuration="21.990663655s" totalImagesPullingTime="18.463957967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:58:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:42.990476449 +0000 UTC m=+24.436521720" watchObservedRunningTime="2026-09-17 21:58:42.990663655 +0000 UTC m=+24.436708928"
Sep 17 21:58:43.003622 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:43.003582    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rdqkk" podStartSLOduration=4.428019772 podStartE2EDuration="23.003567673s" totalImagesPullingTime="18.575547901s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:43.003432398 +0000 UTC m=+24.449477670" watchObservedRunningTime="2026-09-17 21:58:43.003567673 +0000 UTC m=+24.449612946"
Sep 17 21:58:43.032255 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:43.032206    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-sggv9" podStartSLOduration=3.510795836 podStartE2EDuration="22.032169857s" totalImagesPullingTime="18.521374021s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:43.031773692 +0000 UTC m=+24.477818965" watchObservedRunningTime="2026-09-17 21:58:43.032169857 +0000 UTC m=+24.478215130"
Sep 17 21:58:43.039318 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:43.039288    2555 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 21:58:43.641950 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:43.641866    2555 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T21:58:43.039311824Z","UUID":"986f11be-e772-4fdb-a231-e5d33dd3222c","Handler":null,"Name":"","Endpoint":""}
Sep 17 21:58:43.643347 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:43.643329    2555 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 21:58:43.643479 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:43.643355    2555 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 21:58:43.672700 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:43.672672    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl469"
Sep 17 21:58:43.672700 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:43.672702    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7n4d"
Sep 17 21:58:43.673068 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:43.672799    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl469" podUID="84426995-45a3-420d-81b6-2676aba8686b"
Sep 17 21:58:43.673068 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:43.672933    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7n4d" podUID="a236d0c6-0efa-47e2-94c1-f4ca507442c4"
Sep 17 21:58:43.965111 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:43.965075    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zgpjm" event={"ID":"d563612f-869f-4e82-9ba2-690fd3e34052","Type":"ContainerStarted","Data":"55d03d4c8c347659e6c45c83b323b3d7412622f41b709ab98f72c4b6b002dbd8"}
Sep 17 21:58:43.968688 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:43.968664    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s7f5_4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf/ovn-acl-logging/0.log"
Sep 17 21:58:43.969236 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:43.969211    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5" event={"ID":"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf","Type":"ContainerStarted","Data":"930eb1eae699e2909a9a72d92bb1d83c308d1023be6ca7f548a5ed883cb3c96b"}
Sep 17 21:58:43.971302 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:43.971279    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-157.ec2.internal" event={"ID":"d837547dfeda455efa71ea03001f07b0","Type":"ContainerStarted","Data":"f09b85e740ff9a7fe61a294d46f6f80a8036d007e610b61c2922e7c83f9a0658"}
Sep 17 21:58:43.988001 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:43.987959    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-157.ec2.internal" podStartSLOduration=22.987946839 podStartE2EDuration="22.987946839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:43.98663658 +0000 UTC m=+25.432681852" watchObservedRunningTime="2026-09-17 21:58:43.987946839 +0000 UTC m=+25.433992111"
Sep 17 21:58:44.673346 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:44.673313    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-shx44"
Sep 17 21:58:44.673526 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:44.673448    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-shx44" podUID="7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd"
Sep 17 21:58:44.975755 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:44.975666    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zgpjm" event={"ID":"d563612f-869f-4e82-9ba2-690fd3e34052","Type":"ContainerStarted","Data":"b393f3a41adabb92b70a5e0ed324248da64831519ea4ed9e897a594792d1073e"}
Sep 17 21:58:45.672789 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:45.672769    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl469"
Sep 17 21:58:45.672911 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:45.672886    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl469" podUID="84426995-45a3-420d-81b6-2676aba8686b"
Sep 17 21:58:45.672975 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:45.672934    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7n4d"
Sep 17 21:58:45.673064 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:45.673045    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7n4d" podUID="a236d0c6-0efa-47e2-94c1-f4ca507442c4"
Sep 17 21:58:46.673433 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:46.673394    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-shx44"
Sep 17 21:58:46.673867 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:46.673522    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-shx44" podUID="7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd"
Sep 17 21:58:46.770667 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:46.770633    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8sfbd"
Sep 17 21:58:46.771261 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:46.771241    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8sfbd"
Sep 17 21:58:46.785699 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:46.785655    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zgpjm" podStartSLOduration=6.278479763 podStartE2EDuration="26.785642177s" totalImagesPullingTime="20.507162414s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:45.008431618 +0000 UTC m=+26.454476890" watchObservedRunningTime="2026-09-17 21:58:46.785642177 +0000 UTC m=+28.231687448"
Sep 17 21:58:46.980074 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:46.979986    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8sfbd"
Sep 17 21:58:46.980561 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:46.980537    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8sfbd"
Sep 17 21:58:47.673628 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:47.673448    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl469"
Sep 17 21:58:47.674032 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:47.673480    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7n4d"
Sep 17 21:58:47.674032 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:47.673732    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl469" podUID="84426995-45a3-420d-81b6-2676aba8686b"
Sep 17 21:58:47.674032 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:47.673771    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7n4d" podUID="a236d0c6-0efa-47e2-94c1-f4ca507442c4"
Sep 17 21:58:47.984197 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:47.984113    2555 generic.go:356] "Generic (PLEG): container finished" podID="f2ab99ad-b85d-494c-967c-c4e75d08553e" containerID="698a2eb996c1d1cd00c4cf608ad623f2aa65fa462085c09afb02f5b161916321" exitCode=0
Sep 17 21:58:47.984197 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:47.984168    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9sgr6" event={"ID":"f2ab99ad-b85d-494c-967c-c4e75d08553e","Type":"ContainerDied","Data":"698a2eb996c1d1cd00c4cf608ad623f2aa65fa462085c09afb02f5b161916321"}
Sep 17 21:58:47.988114 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:47.988092    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s7f5_4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf/ovn-acl-logging/0.log"
Sep 17 21:58:47.988521 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:47.988489    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5" event={"ID":"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf","Type":"ContainerStarted","Data":"23b2a4c8336eecd7ec8aa355bef2d3fddb3ae2273fa21733fc0511d109dfc26a"}
Sep 17 21:58:47.988958 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:47.988939    2555 scope.go:117] "RemoveContainer" containerID="1ecb20093e8fc5f064c313e085df738d6272fcdd91098569028a7d2b43c8d9a5"
Sep 17 21:58:48.673435 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:48.673405    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-shx44"
Sep 17 21:58:48.673564 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:48.673534    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-shx44" podUID="7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd"
Sep 17 21:58:48.992608 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:48.992530    2555 generic.go:356] "Generic (PLEG): container finished" podID="f2ab99ad-b85d-494c-967c-c4e75d08553e" containerID="e9c7a2715013b445826b69f82210ceaa6fc5be7d817bd579c53cc561a9999120" exitCode=0
Sep 17 21:58:48.993280 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:48.992615    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9sgr6" event={"ID":"f2ab99ad-b85d-494c-967c-c4e75d08553e","Type":"ContainerDied","Data":"e9c7a2715013b445826b69f82210ceaa6fc5be7d817bd579c53cc561a9999120"}
Sep 17 21:58:48.996481 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:48.996464    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s7f5_4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf/ovn-acl-logging/0.log"
Sep 17 21:58:48.996892 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:48.996866    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5" event={"ID":"4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf","Type":"ContainerStarted","Data":"4f73bb29ca4ce77c93596bf02720bab45471cec145175cc2c99825b213e73b87"}
Sep 17 21:58:48.996999 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:48.996979    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:48.997228 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:48.997209    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:48.997329 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:48.997290    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:49.011368 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:49.011350    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:49.011501 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:49.011488    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:58:49.049506 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:49.049462    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5" podStartSLOduration=9.555214063 podStartE2EDuration="28.049449874s" totalImagesPullingTime="18.494235811s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:49.04904433 +0000 UTC m=+30.495089602" watchObservedRunningTime="2026-09-17 21:58:49.049449874 +0000 UTC m=+30.495495146"
Sep 17 21:58:49.344845 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:49.344772    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n7n4d"]
Sep 17 21:58:49.344982 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:49.344881    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7n4d"
Sep 17 21:58:49.345040 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:49.344973    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7n4d" podUID="a236d0c6-0efa-47e2-94c1-f4ca507442c4"
Sep 17 21:58:49.348021 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:49.347993    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-shx44"]
Sep 17 21:58:49.348170 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:49.348077    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-shx44"
Sep 17 21:58:49.348253 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:49.348197    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-shx44" podUID="7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd"
Sep 17 21:58:49.348793 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:49.348763    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vl469"]
Sep 17 21:58:49.348886 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:49.348869    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl469"
Sep 17 21:58:49.348975 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:49.348954    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl469" podUID="84426995-45a3-420d-81b6-2676aba8686b"
Sep 17 21:58:50.001462 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:50.001431    2555 generic.go:356] "Generic (PLEG): container finished" podID="f2ab99ad-b85d-494c-967c-c4e75d08553e" containerID="d3aebc816290dc6962ba81f894fb9c81e3ada7cbd412289cb9c51bc2e0229a82" exitCode=0
Sep 17 21:58:50.001885 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:50.001503    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9sgr6" event={"ID":"f2ab99ad-b85d-494c-967c-c4e75d08553e","Type":"ContainerDied","Data":"d3aebc816290dc6962ba81f894fb9c81e3ada7cbd412289cb9c51bc2e0229a82"}
Sep 17 21:58:50.673888 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:50.673777    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl469"
Sep 17 21:58:50.674042 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:50.673913    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl469" podUID="84426995-45a3-420d-81b6-2676aba8686b"
Sep 17 21:58:51.673022 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:51.672991    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-shx44"
Sep 17 21:58:51.673447 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:51.673025    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7n4d"
Sep 17 21:58:51.673447 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:51.673105    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-shx44" podUID="7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd"
Sep 17 21:58:51.673447 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:51.673409    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7n4d" podUID="a236d0c6-0efa-47e2-94c1-f4ca507442c4"
Sep 17 21:58:52.676502 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:52.676472    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl469"
Sep 17 21:58:52.676913 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:52.676575    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl469" podUID="84426995-45a3-420d-81b6-2676aba8686b"
Sep 17 21:58:53.399086 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:53.399053    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a236d0c6-0efa-47e2-94c1-f4ca507442c4-metrics-certs\") pod \"network-metrics-daemon-n7n4d\" (UID: \"a236d0c6-0efa-47e2-94c1-f4ca507442c4\") " pod="openshift-multus/network-metrics-daemon-n7n4d"
Sep 17 21:58:53.399086 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:53.399103    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84426995-45a3-420d-81b6-2676aba8686b-original-pull-secret\") pod \"global-pull-secret-syncer-vl469\" (UID: \"84426995-45a3-420d-81b6-2676aba8686b\") " pod="kube-system/global-pull-secret-syncer-vl469"
Sep 17 21:58:53.399313 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:53.399237    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:53.399313 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:53.399240    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:53.399313 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:53.399289    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84426995-45a3-420d-81b6-2676aba8686b-original-pull-secret podName:84426995-45a3-420d-81b6-2676aba8686b nodeName:}" failed. No retries permitted until 2026-09-17 21:59:25.399277241 +0000 UTC m=+66.845322491 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/84426995-45a3-420d-81b6-2676aba8686b-original-pull-secret") pod "global-pull-secret-syncer-vl469" (UID: "84426995-45a3-420d-81b6-2676aba8686b") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:53.399313 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:53.399304    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a236d0c6-0efa-47e2-94c1-f4ca507442c4-metrics-certs podName:a236d0c6-0efa-47e2-94c1-f4ca507442c4 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:25.39929736 +0000 UTC m=+66.845342610 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a236d0c6-0efa-47e2-94c1-f4ca507442c4-metrics-certs") pod "network-metrics-daemon-n7n4d" (UID: "a236d0c6-0efa-47e2-94c1-f4ca507442c4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:53.499664 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:53.499634    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzdrk\" (UniqueName: \"kubernetes.io/projected/7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd-kube-api-access-tzdrk\") pod \"network-check-target-shx44\" (UID: \"7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd\") " pod="openshift-network-diagnostics/network-check-target-shx44"
Sep 17 21:58:53.499870 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:53.499846    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:53.499870 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:53.499867    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:53.499963 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:53.499877    2555 projected.go:196] Error preparing data for projected volume kube-api-access-tzdrk for pod openshift-network-diagnostics/network-check-target-shx44: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:53.499963 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:53.499927    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd-kube-api-access-tzdrk podName:7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd nodeName:}" failed. No retries permitted until 2026-09-17 21:59:25.499913244 +0000 UTC m=+66.945958506 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-tzdrk" (UniqueName: "kubernetes.io/projected/7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd-kube-api-access-tzdrk") pod "network-check-target-shx44" (UID: "7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:53.673562 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:53.673481    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7n4d"
Sep 17 21:58:53.673562 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:53.673522    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-shx44"
Sep 17 21:58:53.673719 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:53.673598    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7n4d" podUID="a236d0c6-0efa-47e2-94c1-f4ca507442c4"
Sep 17 21:58:53.673777 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:53.673723    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-shx44" podUID="7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd"
Sep 17 21:58:53.997816 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:53.997793    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-157.ec2.internal" event="NodeReady"
Sep 17 21:58:53.998120 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:53.997923    2555 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 21:58:54.041717 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.041692    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6975d6b4b9-m69t4"]
Sep 17 21:58:54.045202 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.045172    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6975d6b4b9-m69t4"
Sep 17 21:58:54.047865 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.047842    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-h2m4k\""
Sep 17 21:58:54.047953 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.047879    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 21:58:54.048176 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.048161    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 21:58:54.048257 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.048211    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 21:58:54.053268 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.053249    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 21:58:54.062056 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.062016    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6975d6b4b9-m69t4"]
Sep 17 21:58:54.062934 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.062911    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7jzhx"]
Sep 17 21:58:54.066850 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.066828    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7jzhx"
Sep 17 21:58:54.068492 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.068470    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-f87z5"]
Sep 17 21:58:54.069772 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.069752    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 21:58:54.069836 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.069792    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mg4m2\""
Sep 17 21:58:54.069878 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.069766    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 21:58:54.071734 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.071720    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-f87z5"
Sep 17 21:58:54.077475 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.077455    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4j597\""
Sep 17 21:58:54.077573 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.077550    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 21:58:54.077680 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.077663    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 21:58:54.077773 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.077758    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 21:58:54.081670 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.081653    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7jzhx"]
Sep 17 21:58:54.084680 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.084645    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-f87z5"]
Sep 17 21:58:54.102661 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.102637    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8f7439f6-b83c-437d-9d4b-536775fee0d2-trusted-ca\") pod \"image-registry-6975d6b4b9-m69t4\" (UID: \"8f7439f6-b83c-437d-9d4b-536775fee0d2\") " pod="openshift-image-registry/image-registry-6975d6b4b9-m69t4"
Sep 17 21:58:54.102770 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.102682    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8f7439f6-b83c-437d-9d4b-536775fee0d2-ca-trust-extracted\") pod \"image-registry-6975d6b4b9-m69t4\" (UID: \"8f7439f6-b83c-437d-9d4b-536775fee0d2\") " pod="openshift-image-registry/image-registry-6975d6b4b9-m69t4"
Sep 17 21:58:54.102770 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.102708    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn7x5\" (UniqueName: \"kubernetes.io/projected/8f7439f6-b83c-437d-9d4b-536775fee0d2-kube-api-access-qn7x5\") pod \"image-registry-6975d6b4b9-m69t4\" (UID: \"8f7439f6-b83c-437d-9d4b-536775fee0d2\") " pod="openshift-image-registry/image-registry-6975d6b4b9-m69t4"
Sep 17 21:58:54.102770 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.102758    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8f7439f6-b83c-437d-9d4b-536775fee0d2-registry-certificates\") pod \"image-registry-6975d6b4b9-m69t4\" (UID: \"8f7439f6-b83c-437d-9d4b-536775fee0d2\") " pod="openshift-image-registry/image-registry-6975d6b4b9-m69t4"
Sep 17 21:58:54.102871 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.102796    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f7439f6-b83c-437d-9d4b-536775fee0d2-registry-tls\") pod \"image-registry-6975d6b4b9-m69t4\" (UID: \"8f7439f6-b83c-437d-9d4b-536775fee0d2\") " pod="openshift-image-registry/image-registry-6975d6b4b9-m69t4"
Sep 17 21:58:54.102871 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.102814    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8f7439f6-b83c-437d-9d4b-536775fee0d2-image-registry-private-configuration\") pod \"image-registry-6975d6b4b9-m69t4\" (UID: \"8f7439f6-b83c-437d-9d4b-536775fee0d2\") " pod="openshift-image-registry/image-registry-6975d6b4b9-m69t4"
Sep 17 21:58:54.102932 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.102891    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f7439f6-b83c-437d-9d4b-536775fee0d2-bound-sa-token\") pod \"image-registry-6975d6b4b9-m69t4\" (UID: \"8f7439f6-b83c-437d-9d4b-536775fee0d2\") " pod="openshift-image-registry/image-registry-6975d6b4b9-m69t4"
Sep 17 21:58:54.102932 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.102925    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8f7439f6-b83c-437d-9d4b-536775fee0d2-installation-pull-secrets\") pod \"image-registry-6975d6b4b9-m69t4\" (UID: \"8f7439f6-b83c-437d-9d4b-536775fee0d2\") " pod="openshift-image-registry/image-registry-6975d6b4b9-m69t4"
Sep 17 21:58:54.203249 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.203176    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f7439f6-b83c-437d-9d4b-536775fee0d2-registry-tls\") pod \"image-registry-6975d6b4b9-m69t4\" (UID: \"8f7439f6-b83c-437d-9d4b-536775fee0d2\") " pod="openshift-image-registry/image-registry-6975d6b4b9-m69t4"
Sep 17 21:58:54.203249 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.203225    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8f7439f6-b83c-437d-9d4b-536775fee0d2-image-registry-private-configuration\") pod \"image-registry-6975d6b4b9-m69t4\" (UID: \"8f7439f6-b83c-437d-9d4b-536775fee0d2\") " pod="openshift-image-registry/image-registry-6975d6b4b9-m69t4"
Sep 17 21:58:54.203399 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.203255    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f7439f6-b83c-437d-9d4b-536775fee0d2-bound-sa-token\") pod \"image-registry-6975d6b4b9-m69t4\" (UID: \"8f7439f6-b83c-437d-9d4b-536775fee0d2\") " pod="openshift-image-registry/image-registry-6975d6b4b9-m69t4"
Sep 17 21:58:54.203399 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.203276    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8f7439f6-b83c-437d-9d4b-536775fee0d2-installation-pull-secrets\") pod \"image-registry-6975d6b4b9-m69t4\" (UID: \"8f7439f6-b83c-437d-9d4b-536775fee0d2\") " pod="openshift-image-registry/image-registry-6975d6b4b9-m69t4"
Sep 17 21:58:54.203399 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.203294    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfvng\" (UniqueName: \"kubernetes.io/projected/a53be88f-7ad0-4542-9c47-4ec5728fed7b-kube-api-access-tfvng\") pod \"ingress-canary-f87z5\" (UID: \"a53be88f-7ad0-4542-9c47-4ec5728fed7b\") " pod="openshift-ingress-canary/ingress-canary-f87z5"
Sep 17 21:58:54.203399 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:54.203304    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 21:58:54.203399 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:54.203320    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6975d6b4b9-m69t4: secret "image-registry-tls" not found
Sep 17 21:58:54.203399 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.203334    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8f7439f6-b83c-437d-9d4b-536775fee0d2-trusted-ca\") pod \"image-registry-6975d6b4b9-m69t4\" (UID: \"8f7439f6-b83c-437d-9d4b-536775fee0d2\") " pod="openshift-image-registry/image-registry-6975d6b4b9-m69t4"
Sep 17 21:58:54.203399 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:54.203367    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f7439f6-b83c-437d-9d4b-536775fee0d2-registry-tls podName:8f7439f6-b83c-437d-9d4b-536775fee0d2 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:54.703348924 +0000 UTC m=+36.149394173 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8f7439f6-b83c-437d-9d4b-536775fee0d2-registry-tls") pod "image-registry-6975d6b4b9-m69t4" (UID: "8f7439f6-b83c-437d-9d4b-536775fee0d2") : secret "image-registry-tls" not found
Sep 17 21:58:54.203399 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.203385    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a53be88f-7ad0-4542-9c47-4ec5728fed7b-cert\") pod \"ingress-canary-f87z5\" (UID: \"a53be88f-7ad0-4542-9c47-4ec5728fed7b\") " pod="openshift-ingress-canary/ingress-canary-f87z5"
Sep 17 21:58:54.203747 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.203412    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c8d2e30-748c-4f7c-a848-f5f0a327726a-tmp-dir\") pod \"dns-default-7jzhx\" (UID: \"8c8d2e30-748c-4f7c-a848-f5f0a327726a\") " pod="openshift-dns/dns-default-7jzhx"
Sep 17 21:58:54.203747 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.203445    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8f7439f6-b83c-437d-9d4b-536775fee0d2-ca-trust-extracted\") pod \"image-registry-6975d6b4b9-m69t4\" (UID: \"8f7439f6-b83c-437d-9d4b-536775fee0d2\") " pod="openshift-image-registry/image-registry-6975d6b4b9-m69t4"
Sep 17 21:58:54.203747 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.203470    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn7x5\" (UniqueName: \"kubernetes.io/projected/8f7439f6-b83c-437d-9d4b-536775fee0d2-kube-api-access-qn7x5\") pod \"image-registry-6975d6b4b9-m69t4\" (UID: \"8f7439f6-b83c-437d-9d4b-536775fee0d2\") " pod="openshift-image-registry/image-registry-6975d6b4b9-m69t4"
Sep 17 21:58:54.203747 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.203497    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9k7d\" (UniqueName: \"kubernetes.io/projected/8c8d2e30-748c-4f7c-a848-f5f0a327726a-kube-api-access-z9k7d\") pod \"dns-default-7jzhx\" (UID: \"8c8d2e30-748c-4f7c-a848-f5f0a327726a\") " pod="openshift-dns/dns-default-7jzhx"
Sep 17 21:58:54.203747 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.203515    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c8d2e30-748c-4f7c-a848-f5f0a327726a-metrics-tls\") pod \"dns-default-7jzhx\" (UID: \"8c8d2e30-748c-4f7c-a848-f5f0a327726a\") " pod="openshift-dns/dns-default-7jzhx"
Sep 17 21:58:54.203747 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.203533    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8c8d2e30-748c-4f7c-a848-f5f0a327726a-config-volume\") pod \"dns-default-7jzhx\" (UID: \"8c8d2e30-748c-4f7c-a848-f5f0a327726a\") " pod="openshift-dns/dns-default-7jzhx"
Sep 17 21:58:54.203747 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.203566    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8f7439f6-b83c-437d-9d4b-536775fee0d2-registry-certificates\") pod \"image-registry-6975d6b4b9-m69t4\" (UID: \"8f7439f6-b83c-437d-9d4b-536775fee0d2\") " pod="openshift-image-registry/image-registry-6975d6b4b9-m69t4"
Sep 17 21:58:54.206439 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.204310    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8f7439f6-b83c-437d-9d4b-536775fee0d2-ca-trust-extracted\") pod \"image-registry-6975d6b4b9-m69t4\" (UID: \"8f7439f6-b83c-437d-9d4b-536775fee0d2\") " pod="openshift-image-registry/image-registry-6975d6b4b9-m69t4"
Sep 17 21:58:54.206439 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.204659    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8f7439f6-b83c-437d-9d4b-536775fee0d2-registry-certificates\") pod \"image-registry-6975d6b4b9-m69t4\" (UID: \"8f7439f6-b83c-437d-9d4b-536775fee0d2\") " pod="openshift-image-registry/image-registry-6975d6b4b9-m69t4"
Sep 17 21:58:54.206439 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.204968    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8f7439f6-b83c-437d-9d4b-536775fee0d2-trusted-ca\") pod \"image-registry-6975d6b4b9-m69t4\" (UID: \"8f7439f6-b83c-437d-9d4b-536775fee0d2\") " pod="openshift-image-registry/image-registry-6975d6b4b9-m69t4"
Sep 17 21:58:54.208616 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.208588    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8f7439f6-b83c-437d-9d4b-536775fee0d2-installation-pull-secrets\") pod \"image-registry-6975d6b4b9-m69t4\" (UID: \"8f7439f6-b83c-437d-9d4b-536775fee0d2\") " pod="openshift-image-registry/image-registry-6975d6b4b9-m69t4"
Sep 17 21:58:54.208709 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.208640    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8f7439f6-b83c-437d-9d4b-536775fee0d2-image-registry-private-configuration\") pod \"image-registry-6975d6b4b9-m69t4\" (UID: \"8f7439f6-b83c-437d-9d4b-536775fee0d2\") " pod="openshift-image-registry/image-registry-6975d6b4b9-m69t4"
Sep 17 21:58:54.214816 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.214797    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f7439f6-b83c-437d-9d4b-536775fee0d2-bound-sa-token\") pod \"image-registry-6975d6b4b9-m69t4\" (UID: \"8f7439f6-b83c-437d-9d4b-536775fee0d2\") " pod="openshift-image-registry/image-registry-6975d6b4b9-m69t4"
Sep 17 21:58:54.216512 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.216492    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn7x5\" (UniqueName: \"kubernetes.io/projected/8f7439f6-b83c-437d-9d4b-536775fee0d2-kube-api-access-qn7x5\") pod \"image-registry-6975d6b4b9-m69t4\" (UID: \"8f7439f6-b83c-437d-9d4b-536775fee0d2\") " pod="openshift-image-registry/image-registry-6975d6b4b9-m69t4"
Sep 17 21:58:54.304508 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.304478    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c8d2e30-748c-4f7c-a848-f5f0a327726a-tmp-dir\") pod \"dns-default-7jzhx\" (UID: \"8c8d2e30-748c-4f7c-a848-f5f0a327726a\") " pod="openshift-dns/dns-default-7jzhx"
Sep 17 21:58:54.304618 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.304514    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9k7d\" (UniqueName: \"kubernetes.io/projected/8c8d2e30-748c-4f7c-a848-f5f0a327726a-kube-api-access-z9k7d\") pod \"dns-default-7jzhx\" (UID: \"8c8d2e30-748c-4f7c-a848-f5f0a327726a\") " pod="openshift-dns/dns-default-7jzhx"
Sep 17 21:58:54.304618 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.304530    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c8d2e30-748c-4f7c-a848-f5f0a327726a-metrics-tls\") pod \"dns-default-7jzhx\" (UID: \"8c8d2e30-748c-4f7c-a848-f5f0a327726a\") " pod="openshift-dns/dns-default-7jzhx"
Sep 17 21:58:54.304618 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.304546    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8c8d2e30-748c-4f7c-a848-f5f0a327726a-config-volume\") pod \"dns-default-7jzhx\" (UID: \"8c8d2e30-748c-4f7c-a848-f5f0a327726a\") " pod="openshift-dns/dns-default-7jzhx"
Sep 17 21:58:54.304746 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:54.304642    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:58:54.304746 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:54.304715    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c8d2e30-748c-4f7c-a848-f5f0a327726a-metrics-tls podName:8c8d2e30-748c-4f7c-a848-f5f0a327726a nodeName:}" failed. No retries permitted until 2026-09-17 21:58:54.804694207 +0000 UTC m=+36.250739470 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8c8d2e30-748c-4f7c-a848-f5f0a327726a-metrics-tls") pod "dns-default-7jzhx" (UID: "8c8d2e30-748c-4f7c-a848-f5f0a327726a") : secret "dns-default-metrics-tls" not found
Sep 17 21:58:54.304839 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.304800    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfvng\" (UniqueName: \"kubernetes.io/projected/a53be88f-7ad0-4542-9c47-4ec5728fed7b-kube-api-access-tfvng\") pod \"ingress-canary-f87z5\" (UID: \"a53be88f-7ad0-4542-9c47-4ec5728fed7b\") " pod="openshift-ingress-canary/ingress-canary-f87z5"
Sep 17 21:58:54.304839 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.304804    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c8d2e30-748c-4f7c-a848-f5f0a327726a-tmp-dir\") pod \"dns-default-7jzhx\" (UID: \"8c8d2e30-748c-4f7c-a848-f5f0a327726a\") " pod="openshift-dns/dns-default-7jzhx"
Sep 17 21:58:54.304900 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.304882    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a53be88f-7ad0-4542-9c47-4ec5728fed7b-cert\") pod \"ingress-canary-f87z5\" (UID: \"a53be88f-7ad0-4542-9c47-4ec5728fed7b\") " pod="openshift-ingress-canary/ingress-canary-f87z5"
Sep 17 21:58:54.305112 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.305096    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8c8d2e30-748c-4f7c-a848-f5f0a327726a-config-volume\") pod \"dns-default-7jzhx\" (UID: \"8c8d2e30-748c-4f7c-a848-f5f0a327726a\") " pod="openshift-dns/dns-default-7jzhx"
Sep 17 21:58:54.306945 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.306930    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a53be88f-7ad0-4542-9c47-4ec5728fed7b-cert\") pod \"ingress-canary-f87z5\" (UID: \"a53be88f-7ad0-4542-9c47-4ec5728fed7b\") " pod="openshift-ingress-canary/ingress-canary-f87z5"
Sep 17 21:58:54.312548 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.312528    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfvng\" (UniqueName: \"kubernetes.io/projected/a53be88f-7ad0-4542-9c47-4ec5728fed7b-kube-api-access-tfvng\") pod \"ingress-canary-f87z5\" (UID: \"a53be88f-7ad0-4542-9c47-4ec5728fed7b\") " pod="openshift-ingress-canary/ingress-canary-f87z5"
Sep 17 21:58:54.313010 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.312995    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9k7d\" (UniqueName: \"kubernetes.io/projected/8c8d2e30-748c-4f7c-a848-f5f0a327726a-kube-api-access-z9k7d\") pod \"dns-default-7jzhx\" (UID: \"8c8d2e30-748c-4f7c-a848-f5f0a327726a\") " pod="openshift-dns/dns-default-7jzhx"
Sep 17 21:58:54.382911 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.382886    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-f87z5"
Sep 17 21:58:54.663984 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.663788    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-f87z5"]
Sep 17 21:58:54.667717 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:58:54.667691    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda53be88f_7ad0_4542_9c47_4ec5728fed7b.slice/crio-ef650a39e1966a10013e7e966f619d060c393fa27c2295e3d3e68f4c165aa1ce WatchSource:0}: Error finding container ef650a39e1966a10013e7e966f619d060c393fa27c2295e3d3e68f4c165aa1ce: Status 404 returned error can't find the container with id ef650a39e1966a10013e7e966f619d060c393fa27c2295e3d3e68f4c165aa1ce
Sep 17 21:58:54.676150 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.676131    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl469"
Sep 17 21:58:54.678381 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.678359    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 21:58:54.709425 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.709403    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f7439f6-b83c-437d-9d4b-536775fee0d2-registry-tls\") pod \"image-registry-6975d6b4b9-m69t4\" (UID: \"8f7439f6-b83c-437d-9d4b-536775fee0d2\") " pod="openshift-image-registry/image-registry-6975d6b4b9-m69t4"
Sep 17 21:58:54.709591 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:54.709510    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 21:58:54.709591 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:54.709520    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6975d6b4b9-m69t4: secret "image-registry-tls" not found
Sep 17 21:58:54.709591 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:54.709560    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f7439f6-b83c-437d-9d4b-536775fee0d2-registry-tls podName:8f7439f6-b83c-437d-9d4b-536775fee0d2 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:55.709549263 +0000 UTC m=+37.155594513 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8f7439f6-b83c-437d-9d4b-536775fee0d2-registry-tls") pod "image-registry-6975d6b4b9-m69t4" (UID: "8f7439f6-b83c-437d-9d4b-536775fee0d2") : secret "image-registry-tls" not found
Sep 17 21:58:54.810153 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:54.810130    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c8d2e30-748c-4f7c-a848-f5f0a327726a-metrics-tls\") pod \"dns-default-7jzhx\" (UID: \"8c8d2e30-748c-4f7c-a848-f5f0a327726a\") " pod="openshift-dns/dns-default-7jzhx"
Sep 17 21:58:54.810274 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:54.810261    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:58:54.810321 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:54.810310    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c8d2e30-748c-4f7c-a848-f5f0a327726a-metrics-tls podName:8c8d2e30-748c-4f7c-a848-f5f0a327726a nodeName:}" failed. No retries permitted until 2026-09-17 21:58:55.810298493 +0000 UTC m=+37.256343747 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8c8d2e30-748c-4f7c-a848-f5f0a327726a-metrics-tls") pod "dns-default-7jzhx" (UID: "8c8d2e30-748c-4f7c-a848-f5f0a327726a") : secret "dns-default-metrics-tls" not found
Sep 17 21:58:55.015807 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:55.015745    2555 generic.go:356] "Generic (PLEG): container finished" podID="f2ab99ad-b85d-494c-967c-c4e75d08553e" containerID="c40f4e5862d2b705515cabb8ad9fbd553167f9e0f312a789136b098dd1f9d021" exitCode=0
Sep 17 21:58:55.016156 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:55.015817    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9sgr6" event={"ID":"f2ab99ad-b85d-494c-967c-c4e75d08553e","Type":"ContainerDied","Data":"c40f4e5862d2b705515cabb8ad9fbd553167f9e0f312a789136b098dd1f9d021"}
Sep 17 21:58:55.017014 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:55.016992    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-f87z5" event={"ID":"a53be88f-7ad0-4542-9c47-4ec5728fed7b","Type":"ContainerStarted","Data":"ef650a39e1966a10013e7e966f619d060c393fa27c2295e3d3e68f4c165aa1ce"}
Sep 17 21:58:55.673421 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:55.673388    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-shx44"
Sep 17 21:58:55.673603 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:55.673572    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7n4d"
Sep 17 21:58:55.676051 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:55.676028    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 21:58:55.676051 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:55.676038    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-q7xxp\""
Sep 17 21:58:55.676262 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:55.676037    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 21:58:55.676833 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:55.676811    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9ks7q\""
Sep 17 21:58:55.676929 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:55.676813    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 21:58:55.717357 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:55.717333    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f7439f6-b83c-437d-9d4b-536775fee0d2-registry-tls\") pod \"image-registry-6975d6b4b9-m69t4\" (UID: \"8f7439f6-b83c-437d-9d4b-536775fee0d2\") " pod="openshift-image-registry/image-registry-6975d6b4b9-m69t4"
Sep 17 21:58:55.717504 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:55.717486    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 21:58:55.717638 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:55.717507    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6975d6b4b9-m69t4: secret "image-registry-tls" not found
Sep 17 21:58:55.717638 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:55.717568    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f7439f6-b83c-437d-9d4b-536775fee0d2-registry-tls podName:8f7439f6-b83c-437d-9d4b-536775fee0d2 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:57.717548752 +0000 UTC m=+39.163594019 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8f7439f6-b83c-437d-9d4b-536775fee0d2-registry-tls") pod "image-registry-6975d6b4b9-m69t4" (UID: "8f7439f6-b83c-437d-9d4b-536775fee0d2") : secret "image-registry-tls" not found
Sep 17 21:58:55.818577 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:55.818544    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c8d2e30-748c-4f7c-a848-f5f0a327726a-metrics-tls\") pod \"dns-default-7jzhx\" (UID: \"8c8d2e30-748c-4f7c-a848-f5f0a327726a\") " pod="openshift-dns/dns-default-7jzhx"
Sep 17 21:58:55.818728 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:55.818709    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:58:55.818792 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:55.818778    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c8d2e30-748c-4f7c-a848-f5f0a327726a-metrics-tls podName:8c8d2e30-748c-4f7c-a848-f5f0a327726a nodeName:}" failed. No retries permitted until 2026-09-17 21:58:57.818756742 +0000 UTC m=+39.264802008 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8c8d2e30-748c-4f7c-a848-f5f0a327726a-metrics-tls") pod "dns-default-7jzhx" (UID: "8c8d2e30-748c-4f7c-a848-f5f0a327726a") : secret "dns-default-metrics-tls" not found
Sep 17 21:58:55.881311 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:55.881267    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:58:55.967085 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:55.967000    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:58:56.023446 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:56.023415    2555 generic.go:356] "Generic (PLEG): container finished" podID="f2ab99ad-b85d-494c-967c-c4e75d08553e" containerID="628f3c35f46ea7279ad604b4262ed414394b9011837425fe1b4bb6a113740b0e" exitCode=0
Sep 17 21:58:56.023446 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:56.023492    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9sgr6" event={"ID":"f2ab99ad-b85d-494c-967c-c4e75d08553e","Type":"ContainerDied","Data":"628f3c35f46ea7279ad604b4262ed414394b9011837425fe1b4bb6a113740b0e"}
Sep 17 21:58:57.029042 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:57.028991    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9sgr6" event={"ID":"f2ab99ad-b85d-494c-967c-c4e75d08553e","Type":"ContainerStarted","Data":"adc7b7ddbcc30cba2add29f1c3a0971cc0b66361191af1f2b608623403edcdb0"}
Sep 17 21:58:57.031675 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:57.031649    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-f87z5" event={"ID":"a53be88f-7ad0-4542-9c47-4ec5728fed7b","Type":"ContainerStarted","Data":"670247e34404d4cc7a7aeb430f7f8e77f49cf19fd981cf0d77caebb07a2c5103"}
Sep 17 21:58:57.054077 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:57.054029    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9sgr6" podStartSLOduration=9.057442575 podStartE2EDuration="37.054011634s" totalImagesPullingTime="27.996569059s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:20 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:57.053662219 +0000 UTC m=+38.499707484" watchObservedRunningTime="2026-09-17 21:58:57.054011634 +0000 UTC m=+38.500056906"
Sep 17 21:58:57.070531 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:57.070479    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-f87z5" podStartSLOduration=0.888182004 podStartE2EDuration="3.070463858s" totalImagesPullingTime="2.182281854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:58:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:57.069516631 +0000 UTC m=+38.515561904" watchObservedRunningTime="2026-09-17 21:58:57.070463858 +0000 UTC m=+38.516509130"
Sep 17 21:58:57.211094 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:57.211062    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:58:57.277820 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:57.277775    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:58:57.732600 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:57.732567    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f7439f6-b83c-437d-9d4b-536775fee0d2-registry-tls\") pod \"image-registry-6975d6b4b9-m69t4\" (UID: \"8f7439f6-b83c-437d-9d4b-536775fee0d2\") " pod="openshift-image-registry/image-registry-6975d6b4b9-m69t4"
Sep 17 21:58:57.732753 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:57.732712    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 21:58:57.732753 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:57.732728    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6975d6b4b9-m69t4: secret "image-registry-tls" not found
Sep 17 21:58:57.732889 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:57.732785    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f7439f6-b83c-437d-9d4b-536775fee0d2-registry-tls podName:8f7439f6-b83c-437d-9d4b-536775fee0d2 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:01.732767197 +0000 UTC m=+43.178812450 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8f7439f6-b83c-437d-9d4b-536775fee0d2-registry-tls") pod "image-registry-6975d6b4b9-m69t4" (UID: "8f7439f6-b83c-437d-9d4b-536775fee0d2") : secret "image-registry-tls" not found
Sep 17 21:58:57.833258 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:57.833232    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c8d2e30-748c-4f7c-a848-f5f0a327726a-metrics-tls\") pod \"dns-default-7jzhx\" (UID: \"8c8d2e30-748c-4f7c-a848-f5f0a327726a\") " pod="openshift-dns/dns-default-7jzhx"
Sep 17 21:58:57.833394 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:57.833375    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:58:57.833445 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:58:57.833435    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c8d2e30-748c-4f7c-a848-f5f0a327726a-metrics-tls podName:8c8d2e30-748c-4f7c-a848-f5f0a327726a nodeName:}" failed. No retries permitted until 2026-09-17 21:59:01.833421503 +0000 UTC m=+43.279466752 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8c8d2e30-748c-4f7c-a848-f5f0a327726a-metrics-tls") pod "dns-default-7jzhx" (UID: "8c8d2e30-748c-4f7c-a848-f5f0a327726a") : secret "dns-default-metrics-tls" not found
Sep 17 21:58:58.188564 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:58.188535    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:58:59.976328 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:58:59.976295    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:00.080340 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:00.080303    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:00.156531 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:00.156502    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:00.252084 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:00.252015    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:00.939429 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:00.939393    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:01.040862 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:01.040824    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:01.353148 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:01.353055    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:01.454407 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:01.454370    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:01.759954 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:01.759914    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f7439f6-b83c-437d-9d4b-536775fee0d2-registry-tls\") pod \"image-registry-6975d6b4b9-m69t4\" (UID: \"8f7439f6-b83c-437d-9d4b-536775fee0d2\") " pod="openshift-image-registry/image-registry-6975d6b4b9-m69t4"
Sep 17 21:59:01.760110 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:01.760061    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 21:59:01.760110 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:01.760074    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6975d6b4b9-m69t4: secret "image-registry-tls" not found
Sep 17 21:59:01.760231 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:01.760127    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f7439f6-b83c-437d-9d4b-536775fee0d2-registry-tls podName:8f7439f6-b83c-437d-9d4b-536775fee0d2 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:09.760110064 +0000 UTC m=+51.206155323 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8f7439f6-b83c-437d-9d4b-536775fee0d2-registry-tls") pod "image-registry-6975d6b4b9-m69t4" (UID: "8f7439f6-b83c-437d-9d4b-536775fee0d2") : secret "image-registry-tls" not found
Sep 17 21:59:01.861169 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:01.861136    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c8d2e30-748c-4f7c-a848-f5f0a327726a-metrics-tls\") pod \"dns-default-7jzhx\" (UID: \"8c8d2e30-748c-4f7c-a848-f5f0a327726a\") " pod="openshift-dns/dns-default-7jzhx"
Sep 17 21:59:01.861358 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:01.861289    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:59:01.861398 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:01.861365    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c8d2e30-748c-4f7c-a848-f5f0a327726a-metrics-tls podName:8c8d2e30-748c-4f7c-a848-f5f0a327726a nodeName:}" failed. No retries permitted until 2026-09-17 21:59:09.86134914 +0000 UTC m=+51.307394401 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8c8d2e30-748c-4f7c-a848-f5f0a327726a-metrics-tls") pod "dns-default-7jzhx" (UID: "8c8d2e30-748c-4f7c-a848-f5f0a327726a") : secret "dns-default-metrics-tls" not found
Sep 17 21:59:04.062424 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:04.062392    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:04.165882 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:04.165845    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:04.263437 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:04.263409    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:05.044127 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:05.044092    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:05.145040 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:05.144990    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:06.007490 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:06.007400    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:06.059708 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:06.059678    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:07.976420 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:07.976381    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:09.814475 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:09.814437    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f7439f6-b83c-437d-9d4b-536775fee0d2-registry-tls\") pod \"image-registry-6975d6b4b9-m69t4\" (UID: \"8f7439f6-b83c-437d-9d4b-536775fee0d2\") " pod="openshift-image-registry/image-registry-6975d6b4b9-m69t4"
Sep 17 21:59:09.814915 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:09.814587    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 21:59:09.814915 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:09.814603    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6975d6b4b9-m69t4: secret "image-registry-tls" not found
Sep 17 21:59:09.814915 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:09.814664    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f7439f6-b83c-437d-9d4b-536775fee0d2-registry-tls podName:8f7439f6-b83c-437d-9d4b-536775fee0d2 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:25.814643689 +0000 UTC m=+67.260688953 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8f7439f6-b83c-437d-9d4b-536775fee0d2-registry-tls") pod "image-registry-6975d6b4b9-m69t4" (UID: "8f7439f6-b83c-437d-9d4b-536775fee0d2") : secret "image-registry-tls" not found
Sep 17 21:59:09.915679 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:09.915645    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c8d2e30-748c-4f7c-a848-f5f0a327726a-metrics-tls\") pod \"dns-default-7jzhx\" (UID: \"8c8d2e30-748c-4f7c-a848-f5f0a327726a\") " pod="openshift-dns/dns-default-7jzhx"
Sep 17 21:59:09.915862 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:09.915757    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:59:09.915862 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:09.915804    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c8d2e30-748c-4f7c-a848-f5f0a327726a-metrics-tls podName:8c8d2e30-748c-4f7c-a848-f5f0a327726a nodeName:}" failed. No retries permitted until 2026-09-17 21:59:25.915792323 +0000 UTC m=+67.361837573 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8c8d2e30-748c-4f7c-a848-f5f0a327726a-metrics-tls") pod "dns-default-7jzhx" (UID: "8c8d2e30-748c-4f7c-a848-f5f0a327726a") : secret "dns-default-metrics-tls" not found
Sep 17 21:59:11.317079 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.317047    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-6z77t"]
Sep 17 21:59:11.335229 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.335201    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-6z77t"]
Sep 17 21:59:11.335334 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.335304    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-6z77t"
Sep 17 21:59:11.337653 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.337637    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-m7h7h\""
Sep 17 21:59:11.421699 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.421674    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-2tt9k"]
Sep 17 21:59:11.426291 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.426228    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nncvr\" (UniqueName: \"kubernetes.io/projected/8d568830-262f-44e9-9548-bc7884654182-kube-api-access-nncvr\") pod \"network-check-source-7cbc45fddc-6z77t\" (UID: \"8d568830-262f-44e9-9548-bc7884654182\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-6z77t"
Sep 17 21:59:11.441290 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.441267    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-x9xmw"]
Sep 17 21:59:11.441422 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.441403    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-2tt9k"
Sep 17 21:59:11.444500 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.444480    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-tpfps\""
Sep 17 21:59:11.444714 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.444696    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 17 21:59:11.444804 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.444784    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 17 21:59:11.457440 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.457412    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-db9vd"]
Sep 17 21:59:11.458099 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.458075    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-x9xmw"
Sep 17 21:59:11.460812 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.460793    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 17 21:59:11.461458 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.461441    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 17 21:59:11.461458 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.461454    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-6jsbg\""
Sep 17 21:59:11.473379 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.473363    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-7njvj"]
Sep 17 21:59:11.473497 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.473482    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-db9vd"
Sep 17 21:59:11.475721 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.475702    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 17 21:59:11.475721 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.475710    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 17 21:59:11.475896 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.475744    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 17 21:59:11.476010 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.475988    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 17 21:59:11.476089 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.476008    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-h28nm\""
Sep 17 21:59:11.487439 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.487422    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-775b7455c8-brwr9"]
Sep 17 21:59:11.487559 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.487544    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-7njvj"
Sep 17 21:59:11.489851 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.489828    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 17 21:59:11.489851 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.489828    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 17 21:59:11.490011 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.489830    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 17 21:59:11.490011 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.489890    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-tz9sm\""
Sep 17 21:59:11.490119 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.490098    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 17 21:59:11.494842 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.494826    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 17 21:59:11.505557 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.505536    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-2tt9k"]
Sep 17 21:59:11.505651 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.505568    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-x9xmw"]
Sep 17 21:59:11.505651 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.505582    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-7njvj"]
Sep 17 21:59:11.505651 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.505596    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-775b7455c8-brwr9"]
Sep 17 21:59:11.505651 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.505611    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-db9vd"]
Sep 17 21:59:11.505767 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.505683    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-775b7455c8-brwr9"
Sep 17 21:59:11.507949 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.507931    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 17 21:59:11.507949 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.507949    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wg2wj\""
Sep 17 21:59:11.508112 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.507968    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 17 21:59:11.508211 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.508196    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 17 21:59:11.508466 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.508449    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 21:59:11.508700 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.508681    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 21:59:11.508774 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.508709    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 17 21:59:11.526765 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.526748    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nncvr\" (UniqueName: \"kubernetes.io/projected/8d568830-262f-44e9-9548-bc7884654182-kube-api-access-nncvr\") pod \"network-check-source-7cbc45fddc-6z77t\" (UID: \"8d568830-262f-44e9-9548-bc7884654182\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-6z77t"
Sep 17 21:59:11.526849 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.526783    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-2tt9k\" (UID: \"d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-2tt9k"
Sep 17 21:59:11.526849 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.526814    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-2tt9k\" (UID: \"d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-2tt9k"
Sep 17 21:59:11.529515 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.529498    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-pw29d"]
Sep 17 21:59:11.547677 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.547657    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-pw29d"]
Sep 17 21:59:11.547778 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.547690    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-trltj"]
Sep 17 21:59:11.547828 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.547791    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-pw29d"
Sep 17 21:59:11.548721 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.548703    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nncvr\" (UniqueName: \"kubernetes.io/projected/8d568830-262f-44e9-9548-bc7884654182-kube-api-access-nncvr\") pod \"network-check-source-7cbc45fddc-6z77t\" (UID: \"8d568830-262f-44e9-9548-bc7884654182\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-6z77t"
Sep 17 21:59:11.551173 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.551151    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 17 21:59:11.551306 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.551218    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 17 21:59:11.551306 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.551223    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 17 21:59:11.551717 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.551566    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 17 21:59:11.552305 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.552271    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-bwdzx\""
Sep 17 21:59:11.569815 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.569796    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zddx8"]
Sep 17 21:59:11.569939 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.569922    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-trltj"
Sep 17 21:59:11.572603 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.572581    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-pvpnz\""
Sep 17 21:59:11.572717 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.572625    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 17 21:59:11.576563 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.572928    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 17 21:59:11.576563 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.573228    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 17 21:59:11.576563 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.576384    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:11.593104 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.593087    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-2kws6"]
Sep 17 21:59:11.593236 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.593221    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zddx8"
Sep 17 21:59:11.595711 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.595690    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 17 21:59:11.595916 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.595902    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 21:59:11.595990 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.595934    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 17 21:59:11.595990 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.595942    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 21:59:11.595990 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.595953    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8zj29\""
Sep 17 21:59:11.617152 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.617131    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-trltj"]
Sep 17 21:59:11.617249 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.617162    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zddx8"]
Sep 17 21:59:11.617249 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.617201    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-2kws6"]
Sep 17 21:59:11.617326 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.617254    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-2kws6"
Sep 17 21:59:11.619505 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.619490    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-9r97k\""
Sep 17 21:59:11.619581 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.619558    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 21:59:11.619740 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.619727    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 17 21:59:11.620030 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.620006    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 21:59:11.620030 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.620030    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 17 21:59:11.625360 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.625342    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 17 21:59:11.627048 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.627033    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/845db971-2b15-4713-8f19-3e9e851b0d37-default-certificate\") pod \"router-default-775b7455c8-brwr9\" (UID: \"845db971-2b15-4713-8f19-3e9e851b0d37\") " pod="openshift-ingress/router-default-775b7455c8-brwr9"
Sep 17 21:59:11.627122 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.627058    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c62743be-af33-4043-abb5-a7afcc097a8b-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-pw29d\" (UID: \"c62743be-af33-4043-abb5-a7afcc097a8b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-pw29d"
Sep 17 21:59:11.627122 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.627075    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpslk\" (UniqueName: \"kubernetes.io/projected/845db971-2b15-4713-8f19-3e9e851b0d37-kube-api-access-lpslk\") pod \"router-default-775b7455c8-brwr9\" (UID: \"845db971-2b15-4713-8f19-3e9e851b0d37\") " pod="openshift-ingress/router-default-775b7455c8-brwr9"
Sep 17 21:59:11.627122 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.627104    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdfvr\" (UniqueName: \"kubernetes.io/projected/c62743be-af33-4043-abb5-a7afcc097a8b-kube-api-access-pdfvr\") pod \"kube-storage-version-migrator-operator-55889f7b47-pw29d\" (UID: \"c62743be-af33-4043-abb5-a7afcc097a8b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-pw29d"
Sep 17 21:59:11.627255 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.627146    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4sss\" (UniqueName: \"kubernetes.io/projected/a324c9ab-5d42-4d31-8ff7-449f619be774-kube-api-access-g4sss\") pod \"console-operator-fd4bb46c-7njvj\" (UID: \"a324c9ab-5d42-4d31-8ff7-449f619be774\") " pod="openshift-console-operator/console-operator-fd4bb46c-7njvj"
Sep 17 21:59:11.627255 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.627198    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a324c9ab-5d42-4d31-8ff7-449f619be774-serving-cert\") pod \"console-operator-fd4bb46c-7njvj\" (UID: \"a324c9ab-5d42-4d31-8ff7-449f619be774\") " pod="openshift-console-operator/console-operator-fd4bb46c-7njvj"
Sep 17 21:59:11.627255 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.627217    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/845db971-2b15-4713-8f19-3e9e851b0d37-stats-auth\") pod \"router-default-775b7455c8-brwr9\" (UID: \"845db971-2b15-4713-8f19-3e9e851b0d37\") " pod="openshift-ingress/router-default-775b7455c8-brwr9"
Sep 17 21:59:11.627255 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.627236    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a324c9ab-5d42-4d31-8ff7-449f619be774-config\") pod \"console-operator-fd4bb46c-7njvj\" (UID: \"a324c9ab-5d42-4d31-8ff7-449f619be774\") " pod="openshift-console-operator/console-operator-fd4bb46c-7njvj"
Sep 17 21:59:11.627403 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.627269    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-2tt9k\" (UID: \"d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-2tt9k"
Sep 17 21:59:11.627403 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.627301    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/845db971-2b15-4713-8f19-3e9e851b0d37-service-ca-bundle\") pod \"router-default-775b7455c8-brwr9\" (UID: \"845db971-2b15-4713-8f19-3e9e851b0d37\") " pod="openshift-ingress/router-default-775b7455c8-brwr9"
Sep 17 21:59:11.627403 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.627316    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/845db971-2b15-4713-8f19-3e9e851b0d37-metrics-certs\") pod \"router-default-775b7455c8-brwr9\" (UID: \"845db971-2b15-4713-8f19-3e9e851b0d37\") " pod="openshift-ingress/router-default-775b7455c8-brwr9"
Sep 17 21:59:11.627403 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.627360    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6b79292b-9dd7-4560-8a09-80ee28ec9a4b-config\") pod \"service-ca-operator-57cdc4b768-db9vd\" (UID: \"6b79292b-9dd7-4560-8a09-80ee28ec9a4b\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-db9vd"
Sep 17 21:59:11.627403 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:11.627395    2555 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 21:59:11.627620 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:11.627442    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e-networking-console-plugin-cert podName:d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e nodeName:}" failed. No retries permitted until 2026-09-17 21:59:12.127429244 +0000 UTC m=+53.573474494 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-2tt9k" (UID: "d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e") : secret "networking-console-plugin-cert" not found
Sep 17 21:59:11.627620 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.627471    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6b79292b-9dd7-4560-8a09-80ee28ec9a4b-serving-cert\") pod \"service-ca-operator-57cdc4b768-db9vd\" (UID: \"6b79292b-9dd7-4560-8a09-80ee28ec9a4b\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-db9vd"
Sep 17 21:59:11.627620 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.627510    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99p55\" (UniqueName: \"kubernetes.io/projected/da853081-8619-4926-8610-89b3cb4d9557-kube-api-access-99p55\") pod \"volume-data-source-validator-5c4db8bdf9-x9xmw\" (UID: \"da853081-8619-4926-8610-89b3cb4d9557\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-x9xmw"
Sep 17 21:59:11.627749 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.627616    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-2tt9k\" (UID: \"d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-2tt9k"
Sep 17 21:59:11.627749 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.627640    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnjms\" (UniqueName: \"kubernetes.io/projected/6b79292b-9dd7-4560-8a09-80ee28ec9a4b-kube-api-access-lnjms\") pod \"service-ca-operator-57cdc4b768-db9vd\" (UID: \"6b79292b-9dd7-4560-8a09-80ee28ec9a4b\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-db9vd"
Sep 17 21:59:11.627749 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.627684    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c62743be-af33-4043-abb5-a7afcc097a8b-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-pw29d\" (UID: \"c62743be-af33-4043-abb5-a7afcc097a8b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-pw29d"
Sep 17 21:59:11.627749 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.627720    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a324c9ab-5d42-4d31-8ff7-449f619be774-trusted-ca\") pod \"console-operator-fd4bb46c-7njvj\" (UID: \"a324c9ab-5d42-4d31-8ff7-449f619be774\") " pod="openshift-console-operator/console-operator-fd4bb46c-7njvj"
Sep 17 21:59:11.628202 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.628171    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-2tt9k\" (UID: \"d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-2tt9k"
Sep 17 21:59:11.643481 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.643460    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-6z77t"
Sep 17 21:59:11.674092 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.674053    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:11.728655 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.728622    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqzs4\" (UniqueName: \"kubernetes.io/projected/0fab6151-72a0-4476-9275-d3637b228468-kube-api-access-lqzs4\") pod \"cluster-samples-operator-685d7d68b5-trltj\" (UID: \"0fab6151-72a0-4476-9275-d3637b228468\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-trltj"
Sep 17 21:59:11.728762 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.728665    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0fab6151-72a0-4476-9275-d3637b228468-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-trltj\" (UID: \"0fab6151-72a0-4476-9275-d3637b228468\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-trltj"
Sep 17 21:59:11.728762 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.728694    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/845db971-2b15-4713-8f19-3e9e851b0d37-service-ca-bundle\") pod \"router-default-775b7455c8-brwr9\" (UID: \"845db971-2b15-4713-8f19-3e9e851b0d37\") " pod="openshift-ingress/router-default-775b7455c8-brwr9"
Sep 17 21:59:11.728762 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.728712    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/845db971-2b15-4713-8f19-3e9e851b0d37-metrics-certs\") pod \"router-default-775b7455c8-brwr9\" (UID: \"845db971-2b15-4713-8f19-3e9e851b0d37\") " pod="openshift-ingress/router-default-775b7455c8-brwr9"
Sep 17 21:59:11.728762 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.728726    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6b79292b-9dd7-4560-8a09-80ee28ec9a4b-config\") pod \"service-ca-operator-57cdc4b768-db9vd\" (UID: \"6b79292b-9dd7-4560-8a09-80ee28ec9a4b\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-db9vd"
Sep 17 21:59:11.728762 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.728744    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrnls\" (UniqueName: \"kubernetes.io/projected/44044918-464c-4ce3-9ace-e88ffeb8b926-kube-api-access-wrnls\") pod \"cluster-monitoring-operator-847b8bd8d7-zddx8\" (UID: \"44044918-464c-4ce3-9ace-e88ffeb8b926\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zddx8"
Sep 17 21:59:11.728762 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.728760    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6b79292b-9dd7-4560-8a09-80ee28ec9a4b-serving-cert\") pod \"service-ca-operator-57cdc4b768-db9vd\" (UID: \"6b79292b-9dd7-4560-8a09-80ee28ec9a4b\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-db9vd"
Sep 17 21:59:11.728977 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.728779    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b90aafb6-01ac-4c30-a30a-530c91d1c944-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-2kws6\" (UID: \"b90aafb6-01ac-4c30-a30a-530c91d1c944\") " pod="openshift-insights/insights-operator-6f9c644dd9-2kws6"
Sep 17 21:59:11.728977 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.728797    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99p55\" (UniqueName: \"kubernetes.io/projected/da853081-8619-4926-8610-89b3cb4d9557-kube-api-access-99p55\") pod \"volume-data-source-validator-5c4db8bdf9-x9xmw\" (UID: \"da853081-8619-4926-8610-89b3cb4d9557\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-x9xmw"
Sep 17 21:59:11.728977 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.728813    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b90aafb6-01ac-4c30-a30a-530c91d1c944-serving-cert\") pod \"insights-operator-6f9c644dd9-2kws6\" (UID: \"b90aafb6-01ac-4c30-a30a-530c91d1c944\") " pod="openshift-insights/insights-operator-6f9c644dd9-2kws6"
Sep 17 21:59:11.728977 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.728831    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55bks\" (UniqueName: \"kubernetes.io/projected/b90aafb6-01ac-4c30-a30a-530c91d1c944-kube-api-access-55bks\") pod \"insights-operator-6f9c644dd9-2kws6\" (UID: \"b90aafb6-01ac-4c30-a30a-530c91d1c944\") " pod="openshift-insights/insights-operator-6f9c644dd9-2kws6"
Sep 17 21:59:11.728977 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.728849    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b90aafb6-01ac-4c30-a30a-530c91d1c944-snapshots\") pod \"insights-operator-6f9c644dd9-2kws6\" (UID: \"b90aafb6-01ac-4c30-a30a-530c91d1c944\") " pod="openshift-insights/insights-operator-6f9c644dd9-2kws6"
Sep 17 21:59:11.728977 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.728877    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnjms\" (UniqueName: \"kubernetes.io/projected/6b79292b-9dd7-4560-8a09-80ee28ec9a4b-kube-api-access-lnjms\") pod \"service-ca-operator-57cdc4b768-db9vd\" (UID: \"6b79292b-9dd7-4560-8a09-80ee28ec9a4b\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-db9vd"
Sep 17 21:59:11.728977 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.728909    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c62743be-af33-4043-abb5-a7afcc097a8b-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-pw29d\" (UID: \"c62743be-af33-4043-abb5-a7afcc097a8b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-pw29d"
Sep 17 21:59:11.728977 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.728925    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/44044918-464c-4ce3-9ace-e88ffeb8b926-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-zddx8\" (UID: \"44044918-464c-4ce3-9ace-e88ffeb8b926\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zddx8"
Sep 17 21:59:11.728977 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.728940    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b90aafb6-01ac-4c30-a30a-530c91d1c944-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-2kws6\" (UID: \"b90aafb6-01ac-4c30-a30a-530c91d1c944\") " pod="openshift-insights/insights-operator-6f9c644dd9-2kws6"
Sep 17 21:59:11.728977 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.728956    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a324c9ab-5d42-4d31-8ff7-449f619be774-trusted-ca\") pod \"console-operator-fd4bb46c-7njvj\" (UID: \"a324c9ab-5d42-4d31-8ff7-449f619be774\") " pod="openshift-console-operator/console-operator-fd4bb46c-7njvj"
Sep 17 21:59:11.729277 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.728984    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/845db971-2b15-4713-8f19-3e9e851b0d37-default-certificate\") pod \"router-default-775b7455c8-brwr9\" (UID: \"845db971-2b15-4713-8f19-3e9e851b0d37\") " pod="openshift-ingress/router-default-775b7455c8-brwr9"
Sep 17 21:59:11.729277 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.729000    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c62743be-af33-4043-abb5-a7afcc097a8b-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-pw29d\" (UID: \"c62743be-af33-4043-abb5-a7afcc097a8b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-pw29d"
Sep 17 21:59:11.729277 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.729016    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpslk\" (UniqueName: \"kubernetes.io/projected/845db971-2b15-4713-8f19-3e9e851b0d37-kube-api-access-lpslk\") pod \"router-default-775b7455c8-brwr9\" (UID: \"845db971-2b15-4713-8f19-3e9e851b0d37\") " pod="openshift-ingress/router-default-775b7455c8-brwr9"
Sep 17 21:59:11.729277 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.729041    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdfvr\" (UniqueName: \"kubernetes.io/projected/c62743be-af33-4043-abb5-a7afcc097a8b-kube-api-access-pdfvr\") pod \"kube-storage-version-migrator-operator-55889f7b47-pw29d\" (UID: \"c62743be-af33-4043-abb5-a7afcc097a8b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-pw29d"
Sep 17 21:59:11.729277 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.729055    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4sss\" (UniqueName: \"kubernetes.io/projected/a324c9ab-5d42-4d31-8ff7-449f619be774-kube-api-access-g4sss\") pod \"console-operator-fd4bb46c-7njvj\" (UID: \"a324c9ab-5d42-4d31-8ff7-449f619be774\") " pod="openshift-console-operator/console-operator-fd4bb46c-7njvj"
Sep 17 21:59:11.729277 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.729071    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b90aafb6-01ac-4c30-a30a-530c91d1c944-tmp\") pod \"insights-operator-6f9c644dd9-2kws6\" (UID: \"b90aafb6-01ac-4c30-a30a-530c91d1c944\") " pod="openshift-insights/insights-operator-6f9c644dd9-2kws6"
Sep 17 21:59:11.729277 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.729087    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/44044918-464c-4ce3-9ace-e88ffeb8b926-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-zddx8\" (UID: \"44044918-464c-4ce3-9ace-e88ffeb8b926\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zddx8"
Sep 17 21:59:11.729277 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.729107    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a324c9ab-5d42-4d31-8ff7-449f619be774-serving-cert\") pod \"console-operator-fd4bb46c-7njvj\" (UID: \"a324c9ab-5d42-4d31-8ff7-449f619be774\") " pod="openshift-console-operator/console-operator-fd4bb46c-7njvj"
Sep 17 21:59:11.729277 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.729122    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/845db971-2b15-4713-8f19-3e9e851b0d37-stats-auth\") pod \"router-default-775b7455c8-brwr9\" (UID: \"845db971-2b15-4713-8f19-3e9e851b0d37\") " pod="openshift-ingress/router-default-775b7455c8-brwr9"
Sep 17 21:59:11.729277 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.729140    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a324c9ab-5d42-4d31-8ff7-449f619be774-config\") pod \"console-operator-fd4bb46c-7njvj\" (UID: \"a324c9ab-5d42-4d31-8ff7-449f619be774\") " pod="openshift-console-operator/console-operator-fd4bb46c-7njvj"
Sep 17 21:59:11.730279 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.729887    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a324c9ab-5d42-4d31-8ff7-449f619be774-config\") pod \"console-operator-fd4bb46c-7njvj\" (UID: \"a324c9ab-5d42-4d31-8ff7-449f619be774\") " pod="openshift-console-operator/console-operator-fd4bb46c-7njvj"
Sep 17 21:59:11.730279 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:11.729968    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/845db971-2b15-4713-8f19-3e9e851b0d37-service-ca-bundle podName:845db971-2b15-4713-8f19-3e9e851b0d37 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:12.229948778 +0000 UTC m=+53.675994029 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/845db971-2b15-4713-8f19-3e9e851b0d37-service-ca-bundle") pod "router-default-775b7455c8-brwr9" (UID: "845db971-2b15-4713-8f19-3e9e851b0d37") : configmap references non-existent config key: service-ca.crt
Sep 17 21:59:11.730279 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.730114    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c62743be-af33-4043-abb5-a7afcc097a8b-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-pw29d\" (UID: \"c62743be-af33-4043-abb5-a7afcc097a8b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-pw29d"
Sep 17 21:59:11.730279 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.730114    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6b79292b-9dd7-4560-8a09-80ee28ec9a4b-config\") pod \"service-ca-operator-57cdc4b768-db9vd\" (UID: \"6b79292b-9dd7-4560-8a09-80ee28ec9a4b\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-db9vd"
Sep 17 21:59:11.730279 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:11.730231    2555 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 21:59:11.730572 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:11.730289    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/845db971-2b15-4713-8f19-3e9e851b0d37-metrics-certs podName:845db971-2b15-4713-8f19-3e9e851b0d37 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:12.230271141 +0000 UTC m=+53.676316395 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/845db971-2b15-4713-8f19-3e9e851b0d37-metrics-certs") pod "router-default-775b7455c8-brwr9" (UID: "845db971-2b15-4713-8f19-3e9e851b0d37") : secret "router-metrics-certs-default" not found
Sep 17 21:59:11.731042 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.731016    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a324c9ab-5d42-4d31-8ff7-449f619be774-trusted-ca\") pod \"console-operator-fd4bb46c-7njvj\" (UID: \"a324c9ab-5d42-4d31-8ff7-449f619be774\") " pod="openshift-console-operator/console-operator-fd4bb46c-7njvj"
Sep 17 21:59:11.732123 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.732105    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a324c9ab-5d42-4d31-8ff7-449f619be774-serving-cert\") pod \"console-operator-fd4bb46c-7njvj\" (UID: \"a324c9ab-5d42-4d31-8ff7-449f619be774\") " pod="openshift-console-operator/console-operator-fd4bb46c-7njvj"
Sep 17 21:59:11.732327 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.732290    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6b79292b-9dd7-4560-8a09-80ee28ec9a4b-serving-cert\") pod \"service-ca-operator-57cdc4b768-db9vd\" (UID: \"6b79292b-9dd7-4560-8a09-80ee28ec9a4b\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-db9vd"
Sep 17 21:59:11.732871 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.732848    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/845db971-2b15-4713-8f19-3e9e851b0d37-default-certificate\") pod \"router-default-775b7455c8-brwr9\" (UID: \"845db971-2b15-4713-8f19-3e9e851b0d37\") " pod="openshift-ingress/router-default-775b7455c8-brwr9"
Sep 17 21:59:11.733000 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.732981    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/845db971-2b15-4713-8f19-3e9e851b0d37-stats-auth\") pod \"router-default-775b7455c8-brwr9\" (UID: \"845db971-2b15-4713-8f19-3e9e851b0d37\") " pod="openshift-ingress/router-default-775b7455c8-brwr9"
Sep 17 21:59:11.733080 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.733065    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c62743be-af33-4043-abb5-a7afcc097a8b-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-pw29d\" (UID: \"c62743be-af33-4043-abb5-a7afcc097a8b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-pw29d"
Sep 17 21:59:11.740683 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.740639    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99p55\" (UniqueName: \"kubernetes.io/projected/da853081-8619-4926-8610-89b3cb4d9557-kube-api-access-99p55\") pod \"volume-data-source-validator-5c4db8bdf9-x9xmw\" (UID: \"da853081-8619-4926-8610-89b3cb4d9557\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-x9xmw"
Sep 17 21:59:11.741259 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.740977    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnjms\" (UniqueName: \"kubernetes.io/projected/6b79292b-9dd7-4560-8a09-80ee28ec9a4b-kube-api-access-lnjms\") pod \"service-ca-operator-57cdc4b768-db9vd\" (UID: \"6b79292b-9dd7-4560-8a09-80ee28ec9a4b\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-db9vd"
Sep 17 21:59:11.741259 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.741122    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpslk\" (UniqueName: \"kubernetes.io/projected/845db971-2b15-4713-8f19-3e9e851b0d37-kube-api-access-lpslk\") pod \"router-default-775b7455c8-brwr9\" (UID: \"845db971-2b15-4713-8f19-3e9e851b0d37\") " pod="openshift-ingress/router-default-775b7455c8-brwr9"
Sep 17 21:59:11.741259 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.741209    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4sss\" (UniqueName: \"kubernetes.io/projected/a324c9ab-5d42-4d31-8ff7-449f619be774-kube-api-access-g4sss\") pod \"console-operator-fd4bb46c-7njvj\" (UID: \"a324c9ab-5d42-4d31-8ff7-449f619be774\") " pod="openshift-console-operator/console-operator-fd4bb46c-7njvj"
Sep 17 21:59:11.741442 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.741380    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdfvr\" (UniqueName: \"kubernetes.io/projected/c62743be-af33-4043-abb5-a7afcc097a8b-kube-api-access-pdfvr\") pod \"kube-storage-version-migrator-operator-55889f7b47-pw29d\" (UID: \"c62743be-af33-4043-abb5-a7afcc097a8b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-pw29d"
Sep 17 21:59:11.766565 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.766543    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-x9xmw"
Sep 17 21:59:11.774750 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.774729    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:11.781350 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.781335    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-db9vd"
Sep 17 21:59:11.795398 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.795381    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-7njvj"
Sep 17 21:59:11.830222 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.830195    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b90aafb6-01ac-4c30-a30a-530c91d1c944-tmp\") pod \"insights-operator-6f9c644dd9-2kws6\" (UID: \"b90aafb6-01ac-4c30-a30a-530c91d1c944\") " pod="openshift-insights/insights-operator-6f9c644dd9-2kws6"
Sep 17 21:59:11.830334 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.830232    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/44044918-464c-4ce3-9ace-e88ffeb8b926-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-zddx8\" (UID: \"44044918-464c-4ce3-9ace-e88ffeb8b926\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zddx8"
Sep 17 21:59:11.830334 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.830278    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqzs4\" (UniqueName: \"kubernetes.io/projected/0fab6151-72a0-4476-9275-d3637b228468-kube-api-access-lqzs4\") pod \"cluster-samples-operator-685d7d68b5-trltj\" (UID: \"0fab6151-72a0-4476-9275-d3637b228468\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-trltj"
Sep 17 21:59:11.830334 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.830295    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0fab6151-72a0-4476-9275-d3637b228468-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-trltj\" (UID: \"0fab6151-72a0-4476-9275-d3637b228468\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-trltj"
Sep 17 21:59:11.830499 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.830336    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrnls\" (UniqueName: \"kubernetes.io/projected/44044918-464c-4ce3-9ace-e88ffeb8b926-kube-api-access-wrnls\") pod \"cluster-monitoring-operator-847b8bd8d7-zddx8\" (UID: \"44044918-464c-4ce3-9ace-e88ffeb8b926\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zddx8"
Sep 17 21:59:11.830499 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.830356    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b90aafb6-01ac-4c30-a30a-530c91d1c944-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-2kws6\" (UID: \"b90aafb6-01ac-4c30-a30a-530c91d1c944\") " pod="openshift-insights/insights-operator-6f9c644dd9-2kws6"
Sep 17 21:59:11.830499 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.830376    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b90aafb6-01ac-4c30-a30a-530c91d1c944-serving-cert\") pod \"insights-operator-6f9c644dd9-2kws6\" (UID: \"b90aafb6-01ac-4c30-a30a-530c91d1c944\") " pod="openshift-insights/insights-operator-6f9c644dd9-2kws6"
Sep 17 21:59:11.830499 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.830395    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55bks\" (UniqueName: \"kubernetes.io/projected/b90aafb6-01ac-4c30-a30a-530c91d1c944-kube-api-access-55bks\") pod \"insights-operator-6f9c644dd9-2kws6\" (UID: \"b90aafb6-01ac-4c30-a30a-530c91d1c944\") " pod="openshift-insights/insights-operator-6f9c644dd9-2kws6"
Sep 17 21:59:11.830499 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.830412    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b90aafb6-01ac-4c30-a30a-530c91d1c944-snapshots\") pod \"insights-operator-6f9c644dd9-2kws6\" (UID: \"b90aafb6-01ac-4c30-a30a-530c91d1c944\") " pod="openshift-insights/insights-operator-6f9c644dd9-2kws6"
Sep 17 21:59:11.830499 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:11.830428    2555 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 21:59:11.830499 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.830451    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/44044918-464c-4ce3-9ace-e88ffeb8b926-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-zddx8\" (UID: \"44044918-464c-4ce3-9ace-e88ffeb8b926\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zddx8"
Sep 17 21:59:11.830499 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.830467    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b90aafb6-01ac-4c30-a30a-530c91d1c944-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-2kws6\" (UID: \"b90aafb6-01ac-4c30-a30a-530c91d1c944\") " pod="openshift-insights/insights-operator-6f9c644dd9-2kws6"
Sep 17 21:59:11.830858 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:11.830505    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fab6151-72a0-4476-9275-d3637b228468-samples-operator-tls podName:0fab6151-72a0-4476-9275-d3637b228468 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:12.330481204 +0000 UTC m=+53.776526454 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0fab6151-72a0-4476-9275-d3637b228468-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-trltj" (UID: "0fab6151-72a0-4476-9275-d3637b228468") : secret "samples-operator-tls" not found
Sep 17 21:59:11.830858 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.830530    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b90aafb6-01ac-4c30-a30a-530c91d1c944-tmp\") pod \"insights-operator-6f9c644dd9-2kws6\" (UID: \"b90aafb6-01ac-4c30-a30a-530c91d1c944\") " pod="openshift-insights/insights-operator-6f9c644dd9-2kws6"
Sep 17 21:59:11.831304 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:11.830973    2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 21:59:11.831304 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:11.831037    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44044918-464c-4ce3-9ace-e88ffeb8b926-cluster-monitoring-operator-tls podName:44044918-464c-4ce3-9ace-e88ffeb8b926 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:12.331018172 +0000 UTC m=+53.777063429 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/44044918-464c-4ce3-9ace-e88ffeb8b926-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-zddx8" (UID: "44044918-464c-4ce3-9ace-e88ffeb8b926") : secret "cluster-monitoring-operator-tls" not found
Sep 17 21:59:11.831304 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.831123    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b90aafb6-01ac-4c30-a30a-530c91d1c944-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-2kws6\" (UID: \"b90aafb6-01ac-4c30-a30a-530c91d1c944\") " pod="openshift-insights/insights-operator-6f9c644dd9-2kws6"
Sep 17 21:59:11.831509 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.831309    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b90aafb6-01ac-4c30-a30a-530c91d1c944-snapshots\") pod \"insights-operator-6f9c644dd9-2kws6\" (UID: \"b90aafb6-01ac-4c30-a30a-530c91d1c944\") " pod="openshift-insights/insights-operator-6f9c644dd9-2kws6"
Sep 17 21:59:11.831509 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.831370    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/44044918-464c-4ce3-9ace-e88ffeb8b926-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-zddx8\" (UID: \"44044918-464c-4ce3-9ace-e88ffeb8b926\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zddx8"
Sep 17 21:59:11.831509 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.831450    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b90aafb6-01ac-4c30-a30a-530c91d1c944-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-2kws6\" (UID: \"b90aafb6-01ac-4c30-a30a-530c91d1c944\") " pod="openshift-insights/insights-operator-6f9c644dd9-2kws6"
Sep 17 21:59:11.833034 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.833006    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b90aafb6-01ac-4c30-a30a-530c91d1c944-serving-cert\") pod \"insights-operator-6f9c644dd9-2kws6\" (UID: \"b90aafb6-01ac-4c30-a30a-530c91d1c944\") " pod="openshift-insights/insights-operator-6f9c644dd9-2kws6"
Sep 17 21:59:11.843221 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.843147    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqzs4\" (UniqueName: \"kubernetes.io/projected/0fab6151-72a0-4476-9275-d3637b228468-kube-api-access-lqzs4\") pod \"cluster-samples-operator-685d7d68b5-trltj\" (UID: \"0fab6151-72a0-4476-9275-d3637b228468\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-trltj"
Sep 17 21:59:11.844251 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.844216    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrnls\" (UniqueName: \"kubernetes.io/projected/44044918-464c-4ce3-9ace-e88ffeb8b926-kube-api-access-wrnls\") pod \"cluster-monitoring-operator-847b8bd8d7-zddx8\" (UID: \"44044918-464c-4ce3-9ace-e88ffeb8b926\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zddx8"
Sep 17 21:59:11.845116 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.845092    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55bks\" (UniqueName: \"kubernetes.io/projected/b90aafb6-01ac-4c30-a30a-530c91d1c944-kube-api-access-55bks\") pod \"insights-operator-6f9c644dd9-2kws6\" (UID: \"b90aafb6-01ac-4c30-a30a-530c91d1c944\") " pod="openshift-insights/insights-operator-6f9c644dd9-2kws6"
Sep 17 21:59:11.861954 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.861882    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-pw29d"
Sep 17 21:59:11.922019 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.921961    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:11.925848 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.925686    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-2kws6"
Sep 17 21:59:11.942020 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:11.941898    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-6z77t"]
Sep 17 21:59:11.944360 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:59:11.944334    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d568830_262f_44e9_9548_bc7884654182.slice/crio-dcd34589073bbf905b438b27da4bd4c8518113a9c4137c7205879e73440eb070 WatchSource:0}: Error finding container dcd34589073bbf905b438b27da4bd4c8518113a9c4137c7205879e73440eb070: Status 404 returned error can't find the container with id dcd34589073bbf905b438b27da4bd4c8518113a9c4137c7205879e73440eb070
Sep 17 21:59:12.068123 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:12.068099    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-x9xmw"]
Sep 17 21:59:12.069556 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:12.069523    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-6z77t" event={"ID":"8d568830-262f-44e9-9548-bc7884654182","Type":"ContainerStarted","Data":"dcd34589073bbf905b438b27da4bd4c8518113a9c4137c7205879e73440eb070"}
Sep 17 21:59:12.069942 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:59:12.069925    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda853081_8619_4926_8610_89b3cb4d9557.slice/crio-3e4ae2e7aec15f05d27def3de29df7414702a241ca0982b064ac07f5bd4bbb68 WatchSource:0}: Error finding container 3e4ae2e7aec15f05d27def3de29df7414702a241ca0982b064ac07f5bd4bbb68: Status 404 returned error can't find the container with id 3e4ae2e7aec15f05d27def3de29df7414702a241ca0982b064ac07f5bd4bbb68
Sep 17 21:59:12.102527 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:12.102504    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-db9vd"]
Sep 17 21:59:12.104516 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:59:12.104491    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b79292b_9dd7_4560_8a09_80ee28ec9a4b.slice/crio-9b3633c06e97afbf93f9477cef601a0f27c00b6d6622569b78bb44070c355d48 WatchSource:0}: Error finding container 9b3633c06e97afbf93f9477cef601a0f27c00b6d6622569b78bb44070c355d48: Status 404 returned error can't find the container with id 9b3633c06e97afbf93f9477cef601a0f27c00b6d6622569b78bb44070c355d48
Sep 17 21:59:12.122122 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:12.122101    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-7njvj"]
Sep 17 21:59:12.124608 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:59:12.124584    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda324c9ab_5d42_4d31_8ff7_449f619be774.slice/crio-2212677cb975a3d6403d94cba681f8ba47e07d6139c3da34b0b9f0846bf6ecea WatchSource:0}: Error finding container 2212677cb975a3d6403d94cba681f8ba47e07d6139c3da34b0b9f0846bf6ecea: Status 404 returned error can't find the container with id 2212677cb975a3d6403d94cba681f8ba47e07d6139c3da34b0b9f0846bf6ecea
Sep 17 21:59:12.133016 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:12.132991    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-2tt9k\" (UID: \"d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-2tt9k"
Sep 17 21:59:12.133109 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:12.133082    2555 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 21:59:12.133150 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:12.133126    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e-networking-console-plugin-cert podName:d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e nodeName:}" failed. No retries permitted until 2026-09-17 21:59:13.133113326 +0000 UTC m=+54.579158576 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-2tt9k" (UID: "d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e") : secret "networking-console-plugin-cert" not found
Sep 17 21:59:12.169850 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:12.169823    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-pw29d"]
Sep 17 21:59:12.172165 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:59:12.172142    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc62743be_af33_4043_abb5_a7afcc097a8b.slice/crio-d22c1378f4e48c15f64bbac5e45f106842cf9979d4c557eb2c5a185e7f57e5a3 WatchSource:0}: Error finding container d22c1378f4e48c15f64bbac5e45f106842cf9979d4c557eb2c5a185e7f57e5a3: Status 404 returned error can't find the container with id d22c1378f4e48c15f64bbac5e45f106842cf9979d4c557eb2c5a185e7f57e5a3
Sep 17 21:59:12.213387 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:12.213364    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-2kws6"]
Sep 17 21:59:12.216963 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:59:12.216947    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb90aafb6_01ac_4c30_a30a_530c91d1c944.slice/crio-515392b28199f06cf12bfca28de0f75a50f3d21e200562e9cf5d8cc93e373cd1 WatchSource:0}: Error finding container 515392b28199f06cf12bfca28de0f75a50f3d21e200562e9cf5d8cc93e373cd1: Status 404 returned error can't find the container with id 515392b28199f06cf12bfca28de0f75a50f3d21e200562e9cf5d8cc93e373cd1
Sep 17 21:59:12.233511 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:12.233491    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/845db971-2b15-4713-8f19-3e9e851b0d37-service-ca-bundle\") pod \"router-default-775b7455c8-brwr9\" (UID: \"845db971-2b15-4713-8f19-3e9e851b0d37\") " pod="openshift-ingress/router-default-775b7455c8-brwr9"
Sep 17 21:59:12.233597 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:12.233517    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/845db971-2b15-4713-8f19-3e9e851b0d37-metrics-certs\") pod \"router-default-775b7455c8-brwr9\" (UID: \"845db971-2b15-4713-8f19-3e9e851b0d37\") " pod="openshift-ingress/router-default-775b7455c8-brwr9"
Sep 17 21:59:12.233636 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:12.233618    2555 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 21:59:12.233672 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:12.233636    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/845db971-2b15-4713-8f19-3e9e851b0d37-service-ca-bundle podName:845db971-2b15-4713-8f19-3e9e851b0d37 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:13.233620782 +0000 UTC m=+54.679666032 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/845db971-2b15-4713-8f19-3e9e851b0d37-service-ca-bundle") pod "router-default-775b7455c8-brwr9" (UID: "845db971-2b15-4713-8f19-3e9e851b0d37") : configmap references non-existent config key: service-ca.crt
Sep 17 21:59:12.233752 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:12.233741    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/845db971-2b15-4713-8f19-3e9e851b0d37-metrics-certs podName:845db971-2b15-4713-8f19-3e9e851b0d37 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:13.233729778 +0000 UTC m=+54.679775028 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/845db971-2b15-4713-8f19-3e9e851b0d37-metrics-certs") pod "router-default-775b7455c8-brwr9" (UID: "845db971-2b15-4713-8f19-3e9e851b0d37") : secret "router-metrics-certs-default" not found
Sep 17 21:59:12.334280 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:12.334218    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/44044918-464c-4ce3-9ace-e88ffeb8b926-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-zddx8\" (UID: \"44044918-464c-4ce3-9ace-e88ffeb8b926\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zddx8"
Sep 17 21:59:12.334581 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:12.334287    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0fab6151-72a0-4476-9275-d3637b228468-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-trltj\" (UID: \"0fab6151-72a0-4476-9275-d3637b228468\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-trltj"
Sep 17 21:59:12.334581 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:12.334353    2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 21:59:12.334581 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:12.334381    2555 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 21:59:12.334581 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:12.334410    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44044918-464c-4ce3-9ace-e88ffeb8b926-cluster-monitoring-operator-tls podName:44044918-464c-4ce3-9ace-e88ffeb8b926 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:13.334396356 +0000 UTC m=+54.780441606 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/44044918-464c-4ce3-9ace-e88ffeb8b926-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-zddx8" (UID: "44044918-464c-4ce3-9ace-e88ffeb8b926") : secret "cluster-monitoring-operator-tls" not found
Sep 17 21:59:12.334581 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:12.334424    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fab6151-72a0-4476-9275-d3637b228468-samples-operator-tls podName:0fab6151-72a0-4476-9275-d3637b228468 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:13.33441822 +0000 UTC m=+54.780463470 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0fab6151-72a0-4476-9275-d3637b228468-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-trltj" (UID: "0fab6151-72a0-4476-9275-d3637b228468") : secret "samples-operator-tls" not found
Sep 17 21:59:13.077192 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:13.077138    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-2kws6" event={"ID":"b90aafb6-01ac-4c30-a30a-530c91d1c944","Type":"ContainerStarted","Data":"515392b28199f06cf12bfca28de0f75a50f3d21e200562e9cf5d8cc93e373cd1"}
Sep 17 21:59:13.082588 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:13.082554    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-pw29d" event={"ID":"c62743be-af33-4043-abb5-a7afcc097a8b","Type":"ContainerStarted","Data":"d22c1378f4e48c15f64bbac5e45f106842cf9979d4c557eb2c5a185e7f57e5a3"}
Sep 17 21:59:13.088952 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:13.088917    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-x9xmw" event={"ID":"da853081-8619-4926-8610-89b3cb4d9557","Type":"ContainerStarted","Data":"3e4ae2e7aec15f05d27def3de29df7414702a241ca0982b064ac07f5bd4bbb68"}
Sep 17 21:59:13.090957 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:13.090928    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-7njvj" event={"ID":"a324c9ab-5d42-4d31-8ff7-449f619be774","Type":"ContainerStarted","Data":"2212677cb975a3d6403d94cba681f8ba47e07d6139c3da34b0b9f0846bf6ecea"}
Sep 17 21:59:13.095461 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:13.095364    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-db9vd" event={"ID":"6b79292b-9dd7-4560-8a09-80ee28ec9a4b","Type":"ContainerStarted","Data":"9b3633c06e97afbf93f9477cef601a0f27c00b6d6622569b78bb44070c355d48"}
Sep 17 21:59:13.142790 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:13.142751    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-2tt9k\" (UID: \"d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-2tt9k"
Sep 17 21:59:13.143013 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:13.142949    2555 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 21:59:13.143114 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:13.143022    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e-networking-console-plugin-cert podName:d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e nodeName:}" failed. No retries permitted until 2026-09-17 21:59:15.143002417 +0000 UTC m=+56.589047674 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-2tt9k" (UID: "d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e") : secret "networking-console-plugin-cert" not found
Sep 17 21:59:13.243930 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:13.243874    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/845db971-2b15-4713-8f19-3e9e851b0d37-service-ca-bundle\") pod \"router-default-775b7455c8-brwr9\" (UID: \"845db971-2b15-4713-8f19-3e9e851b0d37\") " pod="openshift-ingress/router-default-775b7455c8-brwr9"
Sep 17 21:59:13.243930 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:13.243921    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/845db971-2b15-4713-8f19-3e9e851b0d37-metrics-certs\") pod \"router-default-775b7455c8-brwr9\" (UID: \"845db971-2b15-4713-8f19-3e9e851b0d37\") " pod="openshift-ingress/router-default-775b7455c8-brwr9"
Sep 17 21:59:13.244165 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:13.244122    2555 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 21:59:13.244306 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:13.244196    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/845db971-2b15-4713-8f19-3e9e851b0d37-metrics-certs podName:845db971-2b15-4713-8f19-3e9e851b0d37 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:15.244164812 +0000 UTC m=+56.690210067 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/845db971-2b15-4713-8f19-3e9e851b0d37-metrics-certs") pod "router-default-775b7455c8-brwr9" (UID: "845db971-2b15-4713-8f19-3e9e851b0d37") : secret "router-metrics-certs-default" not found
Sep 17 21:59:13.244704 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:13.244597    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/845db971-2b15-4713-8f19-3e9e851b0d37-service-ca-bundle podName:845db971-2b15-4713-8f19-3e9e851b0d37 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:15.24458106 +0000 UTC m=+56.690626312 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/845db971-2b15-4713-8f19-3e9e851b0d37-service-ca-bundle") pod "router-default-775b7455c8-brwr9" (UID: "845db971-2b15-4713-8f19-3e9e851b0d37") : configmap references non-existent config key: service-ca.crt
Sep 17 21:59:13.345402 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:13.344669    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/44044918-464c-4ce3-9ace-e88ffeb8b926-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-zddx8\" (UID: \"44044918-464c-4ce3-9ace-e88ffeb8b926\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zddx8"
Sep 17 21:59:13.345402 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:13.344791    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0fab6151-72a0-4476-9275-d3637b228468-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-trltj\" (UID: \"0fab6151-72a0-4476-9275-d3637b228468\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-trltj"
Sep 17 21:59:13.345402 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:13.344966    2555 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 21:59:13.345402 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:13.345025    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fab6151-72a0-4476-9275-d3637b228468-samples-operator-tls podName:0fab6151-72a0-4476-9275-d3637b228468 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:15.345007379 +0000 UTC m=+56.791052631 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0fab6151-72a0-4476-9275-d3637b228468-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-trltj" (UID: "0fab6151-72a0-4476-9275-d3637b228468") : secret "samples-operator-tls" not found
Sep 17 21:59:13.345402 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:13.345322    2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 21:59:13.345402 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:13.345391    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44044918-464c-4ce3-9ace-e88ffeb8b926-cluster-monitoring-operator-tls podName:44044918-464c-4ce3-9ace-e88ffeb8b926 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:15.345372927 +0000 UTC m=+56.791418182 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/44044918-464c-4ce3-9ace-e88ffeb8b926-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-zddx8" (UID: "44044918-464c-4ce3-9ace-e88ffeb8b926") : secret "cluster-monitoring-operator-tls" not found
Sep 17 21:59:14.103223 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:14.103169    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-x9xmw" event={"ID":"da853081-8619-4926-8610-89b3cb4d9557","Type":"ContainerStarted","Data":"9b044f708e56ca02ef93ec2a97f95c928e99bdf4fb59a0552b29d530e67d19a6"}
Sep 17 21:59:14.118868 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:14.118566    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-x9xmw" podStartSLOduration=1.3045320839999999 podStartE2EDuration="3.118546433s" totalImagesPullingTime="1.814014349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:14.116725065 +0000 UTC m=+55.562770341" watchObservedRunningTime="2026-09-17 21:59:14.118546433 +0000 UTC m=+55.564591707"
Sep 17 21:59:14.194646 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:14.194502    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:14.253459 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:14.253420    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:15.161917 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:15.161872    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-2tt9k\" (UID: \"d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-2tt9k"
Sep 17 21:59:15.162377 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:15.162034    2555 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 21:59:15.162377 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:15.162128    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e-networking-console-plugin-cert podName:d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e nodeName:}" failed. No retries permitted until 2026-09-17 21:59:19.162106369 +0000 UTC m=+60.608151618 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-2tt9k" (UID: "d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e") : secret "networking-console-plugin-cert" not found
Sep 17 21:59:15.262685 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:15.262644    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/845db971-2b15-4713-8f19-3e9e851b0d37-service-ca-bundle\") pod \"router-default-775b7455c8-brwr9\" (UID: \"845db971-2b15-4713-8f19-3e9e851b0d37\") " pod="openshift-ingress/router-default-775b7455c8-brwr9"
Sep 17 21:59:15.262685 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:15.262678    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/845db971-2b15-4713-8f19-3e9e851b0d37-metrics-certs\") pod \"router-default-775b7455c8-brwr9\" (UID: \"845db971-2b15-4713-8f19-3e9e851b0d37\") " pod="openshift-ingress/router-default-775b7455c8-brwr9"
Sep 17 21:59:15.262913 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:15.262827    2555 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 21:59:15.262913 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:15.262884    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/845db971-2b15-4713-8f19-3e9e851b0d37-service-ca-bundle podName:845db971-2b15-4713-8f19-3e9e851b0d37 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:19.262867807 +0000 UTC m=+60.708913057 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/845db971-2b15-4713-8f19-3e9e851b0d37-service-ca-bundle") pod "router-default-775b7455c8-brwr9" (UID: "845db971-2b15-4713-8f19-3e9e851b0d37") : configmap references non-existent config key: service-ca.crt
Sep 17 21:59:15.262913 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:15.262904    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/845db971-2b15-4713-8f19-3e9e851b0d37-metrics-certs podName:845db971-2b15-4713-8f19-3e9e851b0d37 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:19.26289719 +0000 UTC m=+60.708942440 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/845db971-2b15-4713-8f19-3e9e851b0d37-metrics-certs") pod "router-default-775b7455c8-brwr9" (UID: "845db971-2b15-4713-8f19-3e9e851b0d37") : secret "router-metrics-certs-default" not found
Sep 17 21:59:15.363780 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:15.363741    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0fab6151-72a0-4476-9275-d3637b228468-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-trltj\" (UID: \"0fab6151-72a0-4476-9275-d3637b228468\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-trltj"
Sep 17 21:59:15.363936 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:15.363840    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/44044918-464c-4ce3-9ace-e88ffeb8b926-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-zddx8\" (UID: \"44044918-464c-4ce3-9ace-e88ffeb8b926\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zddx8"
Sep 17 21:59:15.363936 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:15.363902    2555 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 21:59:15.364060 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:15.363944    2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 21:59:15.364060 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:15.363975    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fab6151-72a0-4476-9275-d3637b228468-samples-operator-tls podName:0fab6151-72a0-4476-9275-d3637b228468 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:19.363952744 +0000 UTC m=+60.809998011 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0fab6151-72a0-4476-9275-d3637b228468-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-trltj" (UID: "0fab6151-72a0-4476-9275-d3637b228468") : secret "samples-operator-tls" not found
Sep 17 21:59:15.364060 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:15.363995    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44044918-464c-4ce3-9ace-e88ffeb8b926-cluster-monitoring-operator-tls podName:44044918-464c-4ce3-9ace-e88ffeb8b926 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:19.363985241 +0000 UTC m=+60.810030497 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/44044918-464c-4ce3-9ace-e88ffeb8b926-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-zddx8" (UID: "44044918-464c-4ce3-9ace-e88ffeb8b926") : secret "cluster-monitoring-operator-tls" not found
Sep 17 21:59:18.120084 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:18.120042    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-6z77t" event={"ID":"8d568830-262f-44e9-9548-bc7884654182","Type":"ContainerStarted","Data":"73a10661b114f4e8ec6dd329f09438d3ca4a9aefb81b4952d368f33adc6f3c21"}
Sep 17 21:59:18.121795 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:18.121772    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-7njvj_a324c9ab-5d42-4d31-8ff7-449f619be774/console-operator/0.log"
Sep 17 21:59:18.121906 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:18.121815    2555 generic.go:356] "Generic (PLEG): container finished" podID="a324c9ab-5d42-4d31-8ff7-449f619be774" containerID="fc7b0216c9229fcaf6f3864df60982783a6266dc3accc9ea2f36af5539ce9e5a" exitCode=255
Sep 17 21:59:18.121906 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:18.121894    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-7njvj" event={"ID":"a324c9ab-5d42-4d31-8ff7-449f619be774","Type":"ContainerDied","Data":"fc7b0216c9229fcaf6f3864df60982783a6266dc3accc9ea2f36af5539ce9e5a"}
Sep 17 21:59:18.122114 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:18.122095    2555 scope.go:117] "RemoveContainer" containerID="fc7b0216c9229fcaf6f3864df60982783a6266dc3accc9ea2f36af5539ce9e5a"
Sep 17 21:59:18.124009 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:18.123615    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-db9vd" event={"ID":"6b79292b-9dd7-4560-8a09-80ee28ec9a4b","Type":"ContainerStarted","Data":"bb554b77f4bb72d527e84bd5d3678050df2308d6fef49673fab4f21941449544"}
Sep 17 21:59:18.125791 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:18.125769    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-2kws6" event={"ID":"b90aafb6-01ac-4c30-a30a-530c91d1c944","Type":"ContainerStarted","Data":"fab6e903227d51cc327ddda970a490f8ae43fbc7534ec8c0d606f8d7e46d98c8"}
Sep 17 21:59:18.127565 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:18.127545    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-pw29d" event={"ID":"c62743be-af33-4043-abb5-a7afcc097a8b","Type":"ContainerStarted","Data":"04c768b896d81a90de4091e47a446930115cf4bf69e6ba59d382a0bff43663ef"}
Sep 17 21:59:18.139380 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:18.139337    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-6z77t" podStartSLOduration=1.707477655 podStartE2EDuration="7.13932114s" totalImagesPullingTime="5.431843485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:18.139067335 +0000 UTC m=+59.585112607" watchObservedRunningTime="2026-09-17 21:59:18.13932114 +0000 UTC m=+59.585366413"
Sep 17 21:59:18.159957 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:18.159914    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-2kws6" podStartSLOduration=2.013543112 podStartE2EDuration="7.159899085s" totalImagesPullingTime="5.146355973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:18.158762252 +0000 UTC m=+59.604807528" watchObservedRunningTime="2026-09-17 21:59:18.159899085 +0000 UTC m=+59.605944349"
Sep 17 21:59:18.192998 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:18.192947    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-pw29d" podStartSLOduration=1.9985392069999999 podStartE2EDuration="7.19292971s" totalImagesPullingTime="5.194390503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:18.175745323 +0000 UTC m=+59.621790596" watchObservedRunningTime="2026-09-17 21:59:18.19292971 +0000 UTC m=+59.638974983"
Sep 17 21:59:18.212214 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:18.211694    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-db9vd" podStartSLOduration=1.953294267 podStartE2EDuration="7.211662102s" totalImagesPullingTime="5.258367835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:18.193311123 +0000 UTC m=+59.639356396" watchObservedRunningTime="2026-09-17 21:59:18.211662102 +0000 UTC m=+59.657707377"
Sep 17 21:59:18.835662 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:18.835590    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pxjbl"]
Sep 17 21:59:18.839355 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:18.839332    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pxjbl"
Sep 17 21:59:18.841879 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:18.841817    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 21:59:18.842101 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:18.842084    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 21:59:18.842389 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:18.842358    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:18.842915 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:18.842897    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g6j2k\""
Sep 17 21:59:18.849134 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:18.849111    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pxjbl"]
Sep 17 21:59:18.903994 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:18.903966    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:18.968047 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:18.968018    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:18.997321 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:18.997276    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab4de47d-58d7-44d5-b1c8-884f9801c218-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pxjbl\" (UID: \"ab4de47d-58d7-44d5-b1c8-884f9801c218\") " pod="openshift-insights/insights-runtime-extractor-pxjbl"
Sep 17 21:59:18.997321 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:18.997318    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ab4de47d-58d7-44d5-b1c8-884f9801c218-crio-socket\") pod \"insights-runtime-extractor-pxjbl\" (UID: \"ab4de47d-58d7-44d5-b1c8-884f9801c218\") " pod="openshift-insights/insights-runtime-extractor-pxjbl"
Sep 17 21:59:18.997489 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:18.997356    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab4de47d-58d7-44d5-b1c8-884f9801c218-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pxjbl\" (UID: \"ab4de47d-58d7-44d5-b1c8-884f9801c218\") " pod="openshift-insights/insights-runtime-extractor-pxjbl"
Sep 17 21:59:18.997489 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:18.997397    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab4de47d-58d7-44d5-b1c8-884f9801c218-data-volume\") pod \"insights-runtime-extractor-pxjbl\" (UID: \"ab4de47d-58d7-44d5-b1c8-884f9801c218\") " pod="openshift-insights/insights-runtime-extractor-pxjbl"
Sep 17 21:59:18.997489 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:18.997432    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4xcw\" (UniqueName: \"kubernetes.io/projected/ab4de47d-58d7-44d5-b1c8-884f9801c218-kube-api-access-z4xcw\") pod \"insights-runtime-extractor-pxjbl\" (UID: \"ab4de47d-58d7-44d5-b1c8-884f9801c218\") " pod="openshift-insights/insights-runtime-extractor-pxjbl"
Sep 17 21:59:19.098121 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:19.098035    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab4de47d-58d7-44d5-b1c8-884f9801c218-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pxjbl\" (UID: \"ab4de47d-58d7-44d5-b1c8-884f9801c218\") " pod="openshift-insights/insights-runtime-extractor-pxjbl"
Sep 17 21:59:19.098121 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:19.098079    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ab4de47d-58d7-44d5-b1c8-884f9801c218-crio-socket\") pod \"insights-runtime-extractor-pxjbl\" (UID: \"ab4de47d-58d7-44d5-b1c8-884f9801c218\") " pod="openshift-insights/insights-runtime-extractor-pxjbl"
Sep 17 21:59:19.098333 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:19.098160    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 21:59:19.098333 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:19.098195    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab4de47d-58d7-44d5-b1c8-884f9801c218-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pxjbl\" (UID: \"ab4de47d-58d7-44d5-b1c8-884f9801c218\") " pod="openshift-insights/insights-runtime-extractor-pxjbl"
Sep 17 21:59:19.098333 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:19.098234    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ab4de47d-58d7-44d5-b1c8-884f9801c218-kube-rbac-proxy-cm podName:ab4de47d-58d7-44d5-b1c8-884f9801c218 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:19.598219493 +0000 UTC m=+61.044264766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ab4de47d-58d7-44d5-b1c8-884f9801c218-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pxjbl" (UID: "ab4de47d-58d7-44d5-b1c8-884f9801c218") : configmap "kube-rbac-proxy" not found
Sep 17 21:59:19.098333 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:19.098164    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ab4de47d-58d7-44d5-b1c8-884f9801c218-crio-socket\") pod \"insights-runtime-extractor-pxjbl\" (UID: \"ab4de47d-58d7-44d5-b1c8-884f9801c218\") " pod="openshift-insights/insights-runtime-extractor-pxjbl"
Sep 17 21:59:19.098333 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:19.098295    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:19.098333 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:19.098273    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab4de47d-58d7-44d5-b1c8-884f9801c218-data-volume\") pod \"insights-runtime-extractor-pxjbl\" (UID: \"ab4de47d-58d7-44d5-b1c8-884f9801c218\") " pod="openshift-insights/insights-runtime-extractor-pxjbl"
Sep 17 21:59:19.098651 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:19.098370    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4xcw\" (UniqueName: \"kubernetes.io/projected/ab4de47d-58d7-44d5-b1c8-884f9801c218-kube-api-access-z4xcw\") pod \"insights-runtime-extractor-pxjbl\" (UID: \"ab4de47d-58d7-44d5-b1c8-884f9801c218\") " pod="openshift-insights/insights-runtime-extractor-pxjbl"
Sep 17 21:59:19.098651 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:19.098439    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab4de47d-58d7-44d5-b1c8-884f9801c218-insights-runtime-extractor-tls podName:ab4de47d-58d7-44d5-b1c8-884f9801c218 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:19.598417824 +0000 UTC m=+61.044463075 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab4de47d-58d7-44d5-b1c8-884f9801c218-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pxjbl" (UID: "ab4de47d-58d7-44d5-b1c8-884f9801c218") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:19.098651 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:19.098567    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab4de47d-58d7-44d5-b1c8-884f9801c218-data-volume\") pod \"insights-runtime-extractor-pxjbl\" (UID: \"ab4de47d-58d7-44d5-b1c8-884f9801c218\") " pod="openshift-insights/insights-runtime-extractor-pxjbl"
Sep 17 21:59:19.106375 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:19.106350    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4xcw\" (UniqueName: \"kubernetes.io/projected/ab4de47d-58d7-44d5-b1c8-884f9801c218-kube-api-access-z4xcw\") pod \"insights-runtime-extractor-pxjbl\" (UID: \"ab4de47d-58d7-44d5-b1c8-884f9801c218\") " pod="openshift-insights/insights-runtime-extractor-pxjbl"
Sep 17 21:59:19.133270 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:19.133245    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-7njvj_a324c9ab-5d42-4d31-8ff7-449f619be774/console-operator/1.log"
Sep 17 21:59:19.133745 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:19.133727    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-7njvj_a324c9ab-5d42-4d31-8ff7-449f619be774/console-operator/0.log"
Sep 17 21:59:19.133802 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:19.133775    2555 generic.go:356] "Generic (PLEG): container finished" podID="a324c9ab-5d42-4d31-8ff7-449f619be774" containerID="f4313fef2ede7d1d3dd86cb60f9e3b7b4dfce13ee0e9491ca41de34e49d88356" exitCode=255
Sep 17 21:59:19.135435 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:19.134219    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-7njvj" event={"ID":"a324c9ab-5d42-4d31-8ff7-449f619be774","Type":"ContainerDied","Data":"f4313fef2ede7d1d3dd86cb60f9e3b7b4dfce13ee0e9491ca41de34e49d88356"}
Sep 17 21:59:19.135435 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:19.134301    2555 scope.go:117] "RemoveContainer" containerID="fc7b0216c9229fcaf6f3864df60982783a6266dc3accc9ea2f36af5539ce9e5a"
Sep 17 21:59:19.135435 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:19.135095    2555 scope.go:117] "RemoveContainer" containerID="f4313fef2ede7d1d3dd86cb60f9e3b7b4dfce13ee0e9491ca41de34e49d88356"
Sep 17 21:59:19.135731 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:19.135679    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-7njvj_openshift-console-operator(a324c9ab-5d42-4d31-8ff7-449f619be774)\"" pod="openshift-console-operator/console-operator-fd4bb46c-7njvj" podUID="a324c9ab-5d42-4d31-8ff7-449f619be774"
Sep 17 21:59:19.168531 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:19.168502    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:19.199555 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:19.199531    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-2tt9k\" (UID: \"d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-2tt9k"
Sep 17 21:59:19.199712 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:19.199696    2555 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 21:59:19.199801 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:19.199787    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e-networking-console-plugin-cert podName:d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e nodeName:}" failed. No retries permitted until 2026-09-17 21:59:27.199765268 +0000 UTC m=+68.645810522 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-2tt9k" (UID: "d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e") : secret "networking-console-plugin-cert" not found
Sep 17 21:59:19.227586 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:19.227559    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:19.300716 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:19.300577    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/845db971-2b15-4713-8f19-3e9e851b0d37-service-ca-bundle\") pod \"router-default-775b7455c8-brwr9\" (UID: \"845db971-2b15-4713-8f19-3e9e851b0d37\") " pod="openshift-ingress/router-default-775b7455c8-brwr9"
Sep 17 21:59:19.300716 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:19.300621    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/845db971-2b15-4713-8f19-3e9e851b0d37-metrics-certs\") pod \"router-default-775b7455c8-brwr9\" (UID: \"845db971-2b15-4713-8f19-3e9e851b0d37\") " pod="openshift-ingress/router-default-775b7455c8-brwr9"
Sep 17 21:59:19.300898 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:19.300759    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/845db971-2b15-4713-8f19-3e9e851b0d37-service-ca-bundle podName:845db971-2b15-4713-8f19-3e9e851b0d37 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:27.300738714 +0000 UTC m=+68.746783970 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/845db971-2b15-4713-8f19-3e9e851b0d37-service-ca-bundle") pod "router-default-775b7455c8-brwr9" (UID: "845db971-2b15-4713-8f19-3e9e851b0d37") : configmap references non-existent config key: service-ca.crt
Sep 17 21:59:19.300898 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:19.300759    2555 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 21:59:19.300898 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:19.300834    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/845db971-2b15-4713-8f19-3e9e851b0d37-metrics-certs podName:845db971-2b15-4713-8f19-3e9e851b0d37 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:27.300816677 +0000 UTC m=+68.746861947 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/845db971-2b15-4713-8f19-3e9e851b0d37-metrics-certs") pod "router-default-775b7455c8-brwr9" (UID: "845db971-2b15-4713-8f19-3e9e851b0d37") : secret "router-metrics-certs-default" not found
Sep 17 21:59:19.345002 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:19.344970    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:19.401284 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:19.401239    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/44044918-464c-4ce3-9ace-e88ffeb8b926-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-zddx8\" (UID: \"44044918-464c-4ce3-9ace-e88ffeb8b926\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zddx8"
Sep 17 21:59:19.401374 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:19.401364    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0fab6151-72a0-4476-9275-d3637b228468-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-trltj\" (UID: \"0fab6151-72a0-4476-9275-d3637b228468\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-trltj"
Sep 17 21:59:19.401430 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:19.401370    2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 21:59:19.401430 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:19.401421    2555 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 21:59:19.401501 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:19.401453    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44044918-464c-4ce3-9ace-e88ffeb8b926-cluster-monitoring-operator-tls podName:44044918-464c-4ce3-9ace-e88ffeb8b926 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:27.401435521 +0000 UTC m=+68.847480785 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/44044918-464c-4ce3-9ace-e88ffeb8b926-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-zddx8" (UID: "44044918-464c-4ce3-9ace-e88ffeb8b926") : secret "cluster-monitoring-operator-tls" not found
Sep 17 21:59:19.401501 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:19.401471    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fab6151-72a0-4476-9275-d3637b228468-samples-operator-tls podName:0fab6151-72a0-4476-9275-d3637b228468 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:27.401459227 +0000 UTC m=+68.847504480 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0fab6151-72a0-4476-9275-d3637b228468-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-trltj" (UID: "0fab6151-72a0-4476-9275-d3637b228468") : secret "samples-operator-tls" not found
Sep 17 21:59:19.446608 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:19.446579    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:19.602961 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:19.602939    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab4de47d-58d7-44d5-b1c8-884f9801c218-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pxjbl\" (UID: \"ab4de47d-58d7-44d5-b1c8-884f9801c218\") " pod="openshift-insights/insights-runtime-extractor-pxjbl"
Sep 17 21:59:19.603128 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:19.602977    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab4de47d-58d7-44d5-b1c8-884f9801c218-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pxjbl\" (UID: \"ab4de47d-58d7-44d5-b1c8-884f9801c218\") " pod="openshift-insights/insights-runtime-extractor-pxjbl"
Sep 17 21:59:19.603128 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:19.603054    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 21:59:19.603128 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:19.603076    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:19.603128 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:19.603112    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ab4de47d-58d7-44d5-b1c8-884f9801c218-kube-rbac-proxy-cm podName:ab4de47d-58d7-44d5-b1c8-884f9801c218 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:20.603098357 +0000 UTC m=+62.049143607 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ab4de47d-58d7-44d5-b1c8-884f9801c218-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pxjbl" (UID: "ab4de47d-58d7-44d5-b1c8-884f9801c218") : configmap "kube-rbac-proxy" not found
Sep 17 21:59:19.603291 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:19.603162    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab4de47d-58d7-44d5-b1c8-884f9801c218-insights-runtime-extractor-tls podName:ab4de47d-58d7-44d5-b1c8-884f9801c218 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:20.603154006 +0000 UTC m=+62.049199256 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab4de47d-58d7-44d5-b1c8-884f9801c218-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pxjbl" (UID: "ab4de47d-58d7-44d5-b1c8-884f9801c218") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:19.625888 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:19.625847    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:19.726563 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:19.726472    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:19.830836 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:19.830796    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:19.926541 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:19.926504    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:20.044988 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:20.044895    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:20.139239 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:20.139214    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-7njvj_a324c9ab-5d42-4d31-8ff7-449f619be774/console-operator/1.log"
Sep 17 21:59:20.139664 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:20.139526    2555 scope.go:117] "RemoveContainer" containerID="f4313fef2ede7d1d3dd86cb60f9e3b7b4dfce13ee0e9491ca41de34e49d88356"
Sep 17 21:59:20.139719 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:20.139675    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-7njvj_openshift-console-operator(a324c9ab-5d42-4d31-8ff7-449f619be774)\"" pod="openshift-console-operator/console-operator-fd4bb46c-7njvj" podUID="a324c9ab-5d42-4d31-8ff7-449f619be774"
Sep 17 21:59:20.591382 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:20.591353    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sggv9_cc6119ef-2509-4a37-9edb-1c6da2fdef14/dns-node-resolver/0.log"
Sep 17 21:59:20.611983 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:20.611955    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab4de47d-58d7-44d5-b1c8-884f9801c218-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pxjbl\" (UID: \"ab4de47d-58d7-44d5-b1c8-884f9801c218\") " pod="openshift-insights/insights-runtime-extractor-pxjbl"
Sep 17 21:59:20.612143 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:20.612004    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab4de47d-58d7-44d5-b1c8-884f9801c218-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pxjbl\" (UID: \"ab4de47d-58d7-44d5-b1c8-884f9801c218\") " pod="openshift-insights/insights-runtime-extractor-pxjbl"
Sep 17 21:59:20.612143 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:20.612081    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 21:59:20.612143 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:20.612106    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:20.612143 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:20.612140    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ab4de47d-58d7-44d5-b1c8-884f9801c218-kube-rbac-proxy-cm podName:ab4de47d-58d7-44d5-b1c8-884f9801c218 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:22.612125549 +0000 UTC m=+64.058170814 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ab4de47d-58d7-44d5-b1c8-884f9801c218-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pxjbl" (UID: "ab4de47d-58d7-44d5-b1c8-884f9801c218") : configmap "kube-rbac-proxy" not found
Sep 17 21:59:20.612342 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:20.612207    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab4de47d-58d7-44d5-b1c8-884f9801c218-insights-runtime-extractor-tls podName:ab4de47d-58d7-44d5-b1c8-884f9801c218 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:22.61219032 +0000 UTC m=+64.058235573 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab4de47d-58d7-44d5-b1c8-884f9801c218-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pxjbl" (UID: "ab4de47d-58d7-44d5-b1c8-884f9801c218") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:21.014926 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:21.014891    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5s7f5"
Sep 17 21:59:21.210780 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:21.210746    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:21.323805 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:21.323722    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:21.390959 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:21.390932    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rdqkk_838fe5eb-ddc1-4cea-8a3c-0c5f0e203174/node-ca/0.log"
Sep 17 21:59:21.402092 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:21.402066    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:21.504100 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:21.504070    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:21.796365 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:21.796333    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-7njvj"
Sep 17 21:59:21.796515 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:21.796380    2555 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-7njvj"
Sep 17 21:59:21.796684 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:21.796671    2555 scope.go:117] "RemoveContainer" containerID="f4313fef2ede7d1d3dd86cb60f9e3b7b4dfce13ee0e9491ca41de34e49d88356"
Sep 17 21:59:21.796824 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:21.796809    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-7njvj_openshift-console-operator(a324c9ab-5d42-4d31-8ff7-449f619be774)\"" pod="openshift-console-operator/console-operator-fd4bb46c-7njvj" podUID="a324c9ab-5d42-4d31-8ff7-449f619be774"
Sep 17 21:59:21.926892 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:21.926860    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:22.002844 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:22.002818    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:22.193869 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:22.193848    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-f87z5_a53be88f-7ad0-4542-9c47-4ec5728fed7b/serve-healthcheck-canary/0.log"
Sep 17 21:59:22.231682 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:22.231652    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:22.630109 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:22.630072    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab4de47d-58d7-44d5-b1c8-884f9801c218-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pxjbl\" (UID: \"ab4de47d-58d7-44d5-b1c8-884f9801c218\") " pod="openshift-insights/insights-runtime-extractor-pxjbl"
Sep 17 21:59:22.630307 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:22.630133    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab4de47d-58d7-44d5-b1c8-884f9801c218-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pxjbl\" (UID: \"ab4de47d-58d7-44d5-b1c8-884f9801c218\") " pod="openshift-insights/insights-runtime-extractor-pxjbl"
Sep 17 21:59:22.630307 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:22.630213    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 21:59:22.630307 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:22.630243    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:22.630307 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:22.630280    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ab4de47d-58d7-44d5-b1c8-884f9801c218-kube-rbac-proxy-cm podName:ab4de47d-58d7-44d5-b1c8-884f9801c218 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:26.630264719 +0000 UTC m=+68.076309969 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ab4de47d-58d7-44d5-b1c8-884f9801c218-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pxjbl" (UID: "ab4de47d-58d7-44d5-b1c8-884f9801c218") : configmap "kube-rbac-proxy" not found
Sep 17 21:59:22.630307 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:22.630294    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab4de47d-58d7-44d5-b1c8-884f9801c218-insights-runtime-extractor-tls podName:ab4de47d-58d7-44d5-b1c8-884f9801c218 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:26.630288543 +0000 UTC m=+68.076333793 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab4de47d-58d7-44d5-b1c8-884f9801c218-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pxjbl" (UID: "ab4de47d-58d7-44d5-b1c8-884f9801c218") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:23.779767 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:23.779731    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:23.894467 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:23.894434    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:24.837530 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:24.837496    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:25.454758 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:25.454714    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a236d0c6-0efa-47e2-94c1-f4ca507442c4-metrics-certs\") pod \"network-metrics-daemon-n7n4d\" (UID: \"a236d0c6-0efa-47e2-94c1-f4ca507442c4\") " pod="openshift-multus/network-metrics-daemon-n7n4d"
Sep 17 21:59:25.455030 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:25.454844    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84426995-45a3-420d-81b6-2676aba8686b-original-pull-secret\") pod \"global-pull-secret-syncer-vl469\" (UID: \"84426995-45a3-420d-81b6-2676aba8686b\") " pod="kube-system/global-pull-secret-syncer-vl469"
Sep 17 21:59:25.457455 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:25.457439    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 21:59:25.457513 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:25.457495    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 21:59:25.465597 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:25.465576    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 21:59:25.465665 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:25.465654    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a236d0c6-0efa-47e2-94c1-f4ca507442c4-metrics-certs podName:a236d0c6-0efa-47e2-94c1-f4ca507442c4 nodeName:}" failed. No retries permitted until 2026-09-17 22:00:29.465637396 +0000 UTC m=+130.911682651 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a236d0c6-0efa-47e2-94c1-f4ca507442c4-metrics-certs") pod "network-metrics-daemon-n7n4d" (UID: "a236d0c6-0efa-47e2-94c1-f4ca507442c4") : secret "metrics-daemon-secret" not found
Sep 17 21:59:25.467685 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:25.467667    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84426995-45a3-420d-81b6-2676aba8686b-original-pull-secret\") pod \"global-pull-secret-syncer-vl469\" (UID: \"84426995-45a3-420d-81b6-2676aba8686b\") " pod="kube-system/global-pull-secret-syncer-vl469"
Sep 17 21:59:25.555494 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:25.555460    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzdrk\" (UniqueName: \"kubernetes.io/projected/7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd-kube-api-access-tzdrk\") pod \"network-check-target-shx44\" (UID: \"7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd\") " pod="openshift-network-diagnostics/network-check-target-shx44"
Sep 17 21:59:25.557833 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:25.557807    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzdrk\" (UniqueName: \"kubernetes.io/projected/7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd-kube-api-access-tzdrk\") pod \"network-check-target-shx44\" (UID: \"7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd\") " pod="openshift-network-diagnostics/network-check-target-shx44"
Sep 17 21:59:25.585398 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:25.585375    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl469"
Sep 17 21:59:25.662714 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:25.662678    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:25.687364 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:25.687336    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-q7xxp\""
Sep 17 21:59:25.695207 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:25.695171    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-shx44"
Sep 17 21:59:25.851449 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:25.851413    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:25.858677 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:25.858654    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f7439f6-b83c-437d-9d4b-536775fee0d2-registry-tls\") pod \"image-registry-6975d6b4b9-m69t4\" (UID: \"8f7439f6-b83c-437d-9d4b-536775fee0d2\") " pod="openshift-image-registry/image-registry-6975d6b4b9-m69t4"
Sep 17 21:59:25.858879 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:25.858810    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 21:59:25.858879 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:25.858832    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6975d6b4b9-m69t4: secret "image-registry-tls" not found
Sep 17 21:59:25.858981 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:25.858901    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f7439f6-b83c-437d-9d4b-536775fee0d2-registry-tls podName:8f7439f6-b83c-437d-9d4b-536775fee0d2 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:57.858886577 +0000 UTC m=+99.304931831 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8f7439f6-b83c-437d-9d4b-536775fee0d2-registry-tls") pod "image-registry-6975d6b4b9-m69t4" (UID: "8f7439f6-b83c-437d-9d4b-536775fee0d2") : secret "image-registry-tls" not found
Sep 17 21:59:25.867883 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:25.867860    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vl469"]
Sep 17 21:59:25.870666 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:59:25.870645    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84426995_45a3_420d_81b6_2676aba8686b.slice/crio-33206003d33cce384167ad8f02ac53dd9062a2440e60cc91fc388a4c51e7ef71 WatchSource:0}: Error finding container 33206003d33cce384167ad8f02ac53dd9062a2440e60cc91fc388a4c51e7ef71: Status 404 returned error can't find the container with id 33206003d33cce384167ad8f02ac53dd9062a2440e60cc91fc388a4c51e7ef71
Sep 17 21:59:25.953906 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:25.953879    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:25.959833 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:25.959812    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c8d2e30-748c-4f7c-a848-f5f0a327726a-metrics-tls\") pod \"dns-default-7jzhx\" (UID: \"8c8d2e30-748c-4f7c-a848-f5f0a327726a\") " pod="openshift-dns/dns-default-7jzhx"
Sep 17 21:59:25.959937 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:25.959910    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:59:25.959993 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:25.959959    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c8d2e30-748c-4f7c-a848-f5f0a327726a-metrics-tls podName:8c8d2e30-748c-4f7c-a848-f5f0a327726a nodeName:}" failed. No retries permitted until 2026-09-17 21:59:57.959945552 +0000 UTC m=+99.405990802 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8c8d2e30-748c-4f7c-a848-f5f0a327726a-metrics-tls") pod "dns-default-7jzhx" (UID: "8c8d2e30-748c-4f7c-a848-f5f0a327726a") : secret "dns-default-metrics-tls" not found
Sep 17 21:59:25.966343 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:25.966323    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-shx44"]
Sep 17 21:59:25.968962 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:59:25.968941    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ff7d0a0_c08e_4155_8c55_b6d83d5f16cd.slice/crio-a3e96656f4d57dc463b0385e94592b5935cac2c470eeccd727983ec63ddd884b WatchSource:0}: Error finding container a3e96656f4d57dc463b0385e94592b5935cac2c470eeccd727983ec63ddd884b: Status 404 returned error can't find the container with id a3e96656f4d57dc463b0385e94592b5935cac2c470eeccd727983ec63ddd884b
Sep 17 21:59:26.046892 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:26.046864    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:26.158694 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:26.158671    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-shx44" event={"ID":"7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd","Type":"ContainerStarted","Data":"713ef3038d70bedc91ed7d94b66c5049a9e9ef05de712f04de9fa8cd9d4d247b"}
Sep 17 21:59:26.158810 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:26.158716    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-shx44" event={"ID":"7ff7d0a0-c08e-4155-8c55-b6d83d5f16cd","Type":"ContainerStarted","Data":"a3e96656f4d57dc463b0385e94592b5935cac2c470eeccd727983ec63ddd884b"}
Sep 17 21:59:26.158870 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:26.158852    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-shx44"
Sep 17 21:59:26.159833 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:26.159813    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vl469" event={"ID":"84426995-45a3-420d-81b6-2676aba8686b","Type":"ContainerStarted","Data":"33206003d33cce384167ad8f02ac53dd9062a2440e60cc91fc388a4c51e7ef71"}
Sep 17 21:59:26.186920 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:26.186893    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:26.198483 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:26.198443    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-shx44" podStartSLOduration=66.198430077 podStartE2EDuration="1m6.198430077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:58:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:26.197560472 +0000 UTC m=+67.643605747" watchObservedRunningTime="2026-09-17 21:59:26.198430077 +0000 UTC m=+67.644475421"
Sep 17 21:59:26.433869 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:26.433841    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:26.665893 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:26.665851    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab4de47d-58d7-44d5-b1c8-884f9801c218-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pxjbl\" (UID: \"ab4de47d-58d7-44d5-b1c8-884f9801c218\") " pod="openshift-insights/insights-runtime-extractor-pxjbl"
Sep 17 21:59:26.666080 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:26.665921    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab4de47d-58d7-44d5-b1c8-884f9801c218-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pxjbl\" (UID: \"ab4de47d-58d7-44d5-b1c8-884f9801c218\") " pod="openshift-insights/insights-runtime-extractor-pxjbl"
Sep 17 21:59:26.666080 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:26.665978    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 21:59:26.666080 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:26.666056    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ab4de47d-58d7-44d5-b1c8-884f9801c218-kube-rbac-proxy-cm podName:ab4de47d-58d7-44d5-b1c8-884f9801c218 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:34.666040154 +0000 UTC m=+76.112085404 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ab4de47d-58d7-44d5-b1c8-884f9801c218-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pxjbl" (UID: "ab4de47d-58d7-44d5-b1c8-884f9801c218") : configmap "kube-rbac-proxy" not found
Sep 17 21:59:26.666278 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:26.666088    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:26.666278 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:26.666149    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab4de47d-58d7-44d5-b1c8-884f9801c218-insights-runtime-extractor-tls podName:ab4de47d-58d7-44d5-b1c8-884f9801c218 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:34.666131382 +0000 UTC m=+76.112176635 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab4de47d-58d7-44d5-b1c8-884f9801c218-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pxjbl" (UID: "ab4de47d-58d7-44d5-b1c8-884f9801c218") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:27.269504 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:27.269468    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-2tt9k\" (UID: \"d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-2tt9k"
Sep 17 21:59:27.269904 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:27.269640    2555 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 21:59:27.269904 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:27.269731    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e-networking-console-plugin-cert podName:d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e nodeName:}" failed. No retries permitted until 2026-09-17 21:59:43.26971051 +0000 UTC m=+84.715755761 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-2tt9k" (UID: "d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e") : secret "networking-console-plugin-cert" not found
Sep 17 21:59:27.357192 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:27.357158    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:27.370188 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:27.370159    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/845db971-2b15-4713-8f19-3e9e851b0d37-service-ca-bundle\") pod \"router-default-775b7455c8-brwr9\" (UID: \"845db971-2b15-4713-8f19-3e9e851b0d37\") " pod="openshift-ingress/router-default-775b7455c8-brwr9"
Sep 17 21:59:27.370334 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:27.370201    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/845db971-2b15-4713-8f19-3e9e851b0d37-metrics-certs\") pod \"router-default-775b7455c8-brwr9\" (UID: \"845db971-2b15-4713-8f19-3e9e851b0d37\") " pod="openshift-ingress/router-default-775b7455c8-brwr9"
Sep 17 21:59:27.370334 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:27.370321    2555 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 21:59:27.370451 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:27.370343    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/845db971-2b15-4713-8f19-3e9e851b0d37-service-ca-bundle podName:845db971-2b15-4713-8f19-3e9e851b0d37 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:43.370322411 +0000 UTC m=+84.816367661 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/845db971-2b15-4713-8f19-3e9e851b0d37-service-ca-bundle") pod "router-default-775b7455c8-brwr9" (UID: "845db971-2b15-4713-8f19-3e9e851b0d37") : configmap references non-existent config key: service-ca.crt
Sep 17 21:59:27.370451 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:27.370421    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/845db971-2b15-4713-8f19-3e9e851b0d37-metrics-certs podName:845db971-2b15-4713-8f19-3e9e851b0d37 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:43.370404594 +0000 UTC m=+84.816449844 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/845db971-2b15-4713-8f19-3e9e851b0d37-metrics-certs") pod "router-default-775b7455c8-brwr9" (UID: "845db971-2b15-4713-8f19-3e9e851b0d37") : secret "router-metrics-certs-default" not found
Sep 17 21:59:27.471671 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:27.471635    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0fab6151-72a0-4476-9275-d3637b228468-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-trltj\" (UID: \"0fab6151-72a0-4476-9275-d3637b228468\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-trltj"
Sep 17 21:59:27.471851 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:27.471781    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/44044918-464c-4ce3-9ace-e88ffeb8b926-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-zddx8\" (UID: \"44044918-464c-4ce3-9ace-e88ffeb8b926\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zddx8"
Sep 17 21:59:27.471918 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:27.471896    2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 21:59:27.471977 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:27.471966    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44044918-464c-4ce3-9ace-e88ffeb8b926-cluster-monitoring-operator-tls podName:44044918-464c-4ce3-9ace-e88ffeb8b926 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:43.471948088 +0000 UTC m=+84.917993360 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/44044918-464c-4ce3-9ace-e88ffeb8b926-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-zddx8" (UID: "44044918-464c-4ce3-9ace-e88ffeb8b926") : secret "cluster-monitoring-operator-tls" not found
Sep 17 21:59:27.474778 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:27.474753    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0fab6151-72a0-4476-9275-d3637b228468-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-trltj\" (UID: \"0fab6151-72a0-4476-9275-d3637b228468\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-trltj"
Sep 17 21:59:27.481960 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:27.481935    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-pvpnz\""
Sep 17 21:59:27.489845 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:27.489824    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-trltj"
Sep 17 21:59:27.770344 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:27.770306    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:27.788258 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:27.788233    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-trltj"]
Sep 17 21:59:27.995237 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:27.995203    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:28.156662 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:28.156623    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:28.168783 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:28.168749    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-trltj" event={"ID":"0fab6151-72a0-4476-9275-d3637b228468","Type":"ContainerStarted","Data":"12f26189641b8046bcabcafd93dc7eef4371e1e03dc372ce9421b8c3c965f95f"}
Sep 17 21:59:28.550241 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:28.550154    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:28.650117 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:28.650065    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:29.125825 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:29.125774    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:29.348877 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:29.348828    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:29.450309 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:29.450279    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:29.761704 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:29.761613    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:29.999959 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:29.999862    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:30.084548 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:30.084519    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:30.177879 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:30.177842    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vl469" event={"ID":"84426995-45a3-420d-81b6-2676aba8686b","Type":"ContainerStarted","Data":"d5f296ceb12eb880eeba591535bbd0be7a364256f5377b073e2db130fe09e0fb"}
Sep 17 21:59:30.180858 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:30.180818    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:30.196750 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:30.196693    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vl469" podStartSLOduration=65.464047348 podStartE2EDuration="1m9.196672688s" totalImagesPullingTime="3.73262534s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:30.196068306 +0000 UTC m=+71.642113580" watchObservedRunningTime="2026-09-17 21:59:30.196672688 +0000 UTC m=+71.642717960"
Sep 17 21:59:30.361115 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:30.361035    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:31.183276 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:31.183235    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-trltj" event={"ID":"0fab6151-72a0-4476-9275-d3637b228468","Type":"ContainerStarted","Data":"cbc14171638f691b34f3edd34d3e406e07ccaec3d482f309060abb29483f3704"}
Sep 17 21:59:31.183702 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:31.183297    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-trltj" event={"ID":"0fab6151-72a0-4476-9275-d3637b228468","Type":"ContainerStarted","Data":"1a92cf5a93890d305fe2e4b0eb97e3b49b1bdaf6a2690a7db5fbf1dd3664b874"}
Sep 17 21:59:31.198785 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:31.198732    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-trltj" podStartSLOduration=17.083516739 podStartE2EDuration="20.198714202s" totalImagesPullingTime="3.115197463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:31.198481585 +0000 UTC m=+72.644526857" watchObservedRunningTime="2026-09-17 21:59:31.198714202 +0000 UTC m=+72.644759484"
Sep 17 21:59:31.568730 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:31.568703    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:31.768925 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:31.768883    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:31.878689 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:31.878660    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:31.978924 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:31.978892    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:32.079361 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:32.079321    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:32.180570 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:32.180488    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:32.351987 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:32.351955    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:32.673798 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:32.673776    2555 scope.go:117] "RemoveContainer" containerID="f4313fef2ede7d1d3dd86cb60f9e3b7b4dfce13ee0e9491ca41de34e49d88356"
Sep 17 21:59:33.193758 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:33.192414    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-7njvj_a324c9ab-5d42-4d31-8ff7-449f619be774/console-operator/1.log"
Sep 17 21:59:33.193758 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:33.192526    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-7njvj" event={"ID":"a324c9ab-5d42-4d31-8ff7-449f619be774","Type":"ContainerStarted","Data":"5eda9987687fa0ab24dd54a623ae4cd09ff59d030212962ea9f16f235e788c02"}
Sep 17 21:59:33.193758 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:33.193594    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-7njvj"
Sep 17 21:59:33.199198 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:33.199155    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-7njvj"
Sep 17 21:59:33.212134 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:33.212091    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-7njvj" podStartSLOduration=16.973405754 podStartE2EDuration="22.212076692s" totalImagesPullingTime="5.238670938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:33.210597089 +0000 UTC m=+74.656642357" watchObservedRunningTime="2026-09-17 21:59:33.212076692 +0000 UTC m=+74.658121964"
Sep 17 21:59:33.231189 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:33.231147    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:33.317349 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:33.317315    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:33.416153 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:33.416105    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:33.551540 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:33.551455    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:33.721562 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:33.721531    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:33.847999 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:33.847915    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:33.962102 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:33.962073    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:34.216394 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:34.216367    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:34.322556 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:34.322524    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:34.422087 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:34.422063    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:34.536405 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:34.536332    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:34.642306 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:34.642273    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:34.738337 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:34.738308    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab4de47d-58d7-44d5-b1c8-884f9801c218-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pxjbl\" (UID: \"ab4de47d-58d7-44d5-b1c8-884f9801c218\") " pod="openshift-insights/insights-runtime-extractor-pxjbl"
Sep 17 21:59:34.738505 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:34.738353    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab4de47d-58d7-44d5-b1c8-884f9801c218-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pxjbl\" (UID: \"ab4de47d-58d7-44d5-b1c8-884f9801c218\") " pod="openshift-insights/insights-runtime-extractor-pxjbl"
Sep 17 21:59:34.738505 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:34.738462    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:34.738577 ip-10-0-140-157 kubenswrapper[2555]: E0917 21:59:34.738529    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab4de47d-58d7-44d5-b1c8-884f9801c218-insights-runtime-extractor-tls podName:ab4de47d-58d7-44d5-b1c8-884f9801c218 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:50.738509352 +0000 UTC m=+92.184554601 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab4de47d-58d7-44d5-b1c8-884f9801c218-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pxjbl" (UID: "ab4de47d-58d7-44d5-b1c8-884f9801c218") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:34.738940 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:34.738922    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab4de47d-58d7-44d5-b1c8-884f9801c218-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pxjbl\" (UID: \"ab4de47d-58d7-44d5-b1c8-884f9801c218\") " pod="openshift-insights/insights-runtime-extractor-pxjbl"
Sep 17 21:59:34.932875 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:34.932840    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:35.033143 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:35.033090    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:35.141803 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:35.141772    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:35.349309 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:35.349231    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:35.963367 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:35.963340    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:36.046796 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:36.046767    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:36.115146 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:36.115121    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:36.197235 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:36.197205    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:36.362103 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:36.362026    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:36.761747 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:36.761715    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:40.146323 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:40.146293    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-695b94f8cd-z8fms"]
Sep 17 21:59:40.149309 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:40.149290    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-695b94f8cd-z8fms"
Sep 17 21:59:40.152995 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:40.152973    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 21:59:40.153801 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:40.153778    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 21:59:40.153801 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:40.153790    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 21:59:40.153801 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:40.153797    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 17 21:59:40.153990 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:40.153817    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-hzxkt\""
Sep 17 21:59:40.166391 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:40.166365    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-695b94f8cd-z8fms"]
Sep 17 21:59:40.169671 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:40.169644    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:40.182593 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:40.182571    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e1404cae-3e39-4ef1-b7b8-80b19cb295ac-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-695b94f8cd-z8fms\" (UID: \"e1404cae-3e39-4ef1-b7b8-80b19cb295ac\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-695b94f8cd-z8fms"
Sep 17 21:59:40.182679 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:40.182600    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42vrl\" (UniqueName: \"kubernetes.io/projected/e1404cae-3e39-4ef1-b7b8-80b19cb295ac-kube-api-access-42vrl\") pod \"managed-serviceaccount-addon-agent-695b94f8cd-z8fms\" (UID: \"e1404cae-3e39-4ef1-b7b8-80b19cb295ac\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-695b94f8cd-z8fms"
Sep 17 21:59:40.251646 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:40.251622    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-n2h8c"]
Sep 17 21:59:40.254691 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:40.254675    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-n2h8c"
Sep 17 21:59:40.256998 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:40.256978    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-c6mqp\""
Sep 17 21:59:40.257086 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:40.256978    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 21:59:40.257128 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:40.257088    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 21:59:40.266724 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:40.266703    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-n2h8c"]
Sep 17 21:59:40.276333 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:40.276312    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:40.283787 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:40.283764    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5244\" (UniqueName: \"kubernetes.io/projected/1c48dd88-220b-45c2-a407-1240b4294030-kube-api-access-f5244\") pod \"downloads-6594f4c75-n2h8c\" (UID: \"1c48dd88-220b-45c2-a407-1240b4294030\") " pod="openshift-console/downloads-6594f4c75-n2h8c"
Sep 17 21:59:40.283874 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:40.283794    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e1404cae-3e39-4ef1-b7b8-80b19cb295ac-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-695b94f8cd-z8fms\" (UID: \"e1404cae-3e39-4ef1-b7b8-80b19cb295ac\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-695b94f8cd-z8fms"
Sep 17 21:59:40.283874 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:40.283816    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42vrl\" (UniqueName: \"kubernetes.io/projected/e1404cae-3e39-4ef1-b7b8-80b19cb295ac-kube-api-access-42vrl\") pod \"managed-serviceaccount-addon-agent-695b94f8cd-z8fms\" (UID: \"e1404cae-3e39-4ef1-b7b8-80b19cb295ac\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-695b94f8cd-z8fms"
Sep 17 21:59:40.286161 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:40.286142    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e1404cae-3e39-4ef1-b7b8-80b19cb295ac-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-695b94f8cd-z8fms\" (UID: \"e1404cae-3e39-4ef1-b7b8-80b19cb295ac\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-695b94f8cd-z8fms"
Sep 17 21:59:40.303123 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:40.303097    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42vrl\" (UniqueName: \"kubernetes.io/projected/e1404cae-3e39-4ef1-b7b8-80b19cb295ac-kube-api-access-42vrl\") pod \"managed-serviceaccount-addon-agent-695b94f8cd-z8fms\" (UID: \"e1404cae-3e39-4ef1-b7b8-80b19cb295ac\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-695b94f8cd-z8fms"
Sep 17 21:59:40.359137 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:40.359109    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:40.385249 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:40.385221    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5244\" (UniqueName: \"kubernetes.io/projected/1c48dd88-220b-45c2-a407-1240b4294030-kube-api-access-f5244\") pod \"downloads-6594f4c75-n2h8c\" (UID: \"1c48dd88-220b-45c2-a407-1240b4294030\") " pod="openshift-console/downloads-6594f4c75-n2h8c"
Sep 17 21:59:40.393712 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:40.393687    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5244\" (UniqueName: \"kubernetes.io/projected/1c48dd88-220b-45c2-a407-1240b4294030-kube-api-access-f5244\") pod \"downloads-6594f4c75-n2h8c\" (UID: \"1c48dd88-220b-45c2-a407-1240b4294030\") " pod="openshift-console/downloads-6594f4c75-n2h8c"
Sep 17 21:59:40.460201 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:40.460122    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:40.469227 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:40.469212    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-695b94f8cd-z8fms"
Sep 17 21:59:40.557609 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:40.557581    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:40.563535 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:40.563519    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-n2h8c"
Sep 17 21:59:40.736700 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:40.736626    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-695b94f8cd-z8fms"]
Sep 17 21:59:40.739801 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:59:40.739772    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1404cae_3e39_4ef1_b7b8_80b19cb295ac.slice/crio-19efd008d3893d2e8cd202792f9db51d1dacea9ebdc09ce2f32278483caf5dcc WatchSource:0}: Error finding container 19efd008d3893d2e8cd202792f9db51d1dacea9ebdc09ce2f32278483caf5dcc: Status 404 returned error can't find the container with id 19efd008d3893d2e8cd202792f9db51d1dacea9ebdc09ce2f32278483caf5dcc
Sep 17 21:59:40.832625 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:40.832604    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-n2h8c"]
Sep 17 21:59:40.836767 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:59:40.836741    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c48dd88_220b_45c2_a407_1240b4294030.slice/crio-d2282957513d5eabb2f49a106ad0e46bb13a8f6db20907648f20af9aa764f37f WatchSource:0}: Error finding container d2282957513d5eabb2f49a106ad0e46bb13a8f6db20907648f20af9aa764f37f: Status 404 returned error can't find the container with id d2282957513d5eabb2f49a106ad0e46bb13a8f6db20907648f20af9aa764f37f
Sep 17 21:59:41.219971 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:41.219913    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-n2h8c" event={"ID":"1c48dd88-220b-45c2-a407-1240b4294030","Type":"ContainerStarted","Data":"d2282957513d5eabb2f49a106ad0e46bb13a8f6db20907648f20af9aa764f37f"}
Sep 17 21:59:41.221436 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:41.221403    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-695b94f8cd-z8fms" event={"ID":"e1404cae-3e39-4ef1-b7b8-80b19cb295ac","Type":"ContainerStarted","Data":"19efd008d3893d2e8cd202792f9db51d1dacea9ebdc09ce2f32278483caf5dcc"}
Sep 17 21:59:43.231028 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:43.230997    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-695b94f8cd-z8fms" event={"ID":"e1404cae-3e39-4ef1-b7b8-80b19cb295ac","Type":"ContainerStarted","Data":"3d5e9862eedf083dee5e4aa7d53f9eb85fb534393d0d8fa196576baedbe6533e"}
Sep 17 21:59:43.246903 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:43.246847    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-695b94f8cd-z8fms" podStartSLOduration=0.836439494 podStartE2EDuration="3.246821705s" totalImagesPullingTime="2.410382211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:43.244942756 +0000 UTC m=+84.690988029" watchObservedRunningTime="2026-09-17 21:59:43.246821705 +0000 UTC m=+84.692866978"
Sep 17 21:59:43.313125 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:43.313087    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-2tt9k\" (UID: \"d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-2tt9k"
Sep 17 21:59:43.315644 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:43.315616    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-2tt9k\" (UID: \"d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-2tt9k"
Sep 17 21:59:43.413984 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:43.413903    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/845db971-2b15-4713-8f19-3e9e851b0d37-service-ca-bundle\") pod \"router-default-775b7455c8-brwr9\" (UID: \"845db971-2b15-4713-8f19-3e9e851b0d37\") " pod="openshift-ingress/router-default-775b7455c8-brwr9"
Sep 17 21:59:43.413984 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:43.413944    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/845db971-2b15-4713-8f19-3e9e851b0d37-metrics-certs\") pod \"router-default-775b7455c8-brwr9\" (UID: \"845db971-2b15-4713-8f19-3e9e851b0d37\") " pod="openshift-ingress/router-default-775b7455c8-brwr9"
Sep 17 21:59:43.414868 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:43.414834    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/845db971-2b15-4713-8f19-3e9e851b0d37-service-ca-bundle\") pod \"router-default-775b7455c8-brwr9\" (UID: \"845db971-2b15-4713-8f19-3e9e851b0d37\") " pod="openshift-ingress/router-default-775b7455c8-brwr9"
Sep 17 21:59:43.416747 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:43.416719    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/845db971-2b15-4713-8f19-3e9e851b0d37-metrics-certs\") pod \"router-default-775b7455c8-brwr9\" (UID: \"845db971-2b15-4713-8f19-3e9e851b0d37\") " pod="openshift-ingress/router-default-775b7455c8-brwr9"
Sep 17 21:59:43.515246 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:43.515211    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/44044918-464c-4ce3-9ace-e88ffeb8b926-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-zddx8\" (UID: \"44044918-464c-4ce3-9ace-e88ffeb8b926\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zddx8"
Sep 17 21:59:43.517916 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:43.517884    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/44044918-464c-4ce3-9ace-e88ffeb8b926-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-zddx8\" (UID: \"44044918-464c-4ce3-9ace-e88ffeb8b926\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zddx8"
Sep 17 21:59:43.553254 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:43.553228    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-tpfps\""
Sep 17 21:59:43.561972 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:43.561948    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-2tt9k"
Sep 17 21:59:43.617607 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:43.616828    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wg2wj\""
Sep 17 21:59:43.624899 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:43.624877    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-775b7455c8-brwr9"
Sep 17 21:59:43.703714 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:43.703688    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8zj29\""
Sep 17 21:59:43.712425 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:43.712115    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zddx8"
Sep 17 21:59:43.838727 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:43.838695    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-2tt9k"]
Sep 17 21:59:43.841103 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:59:43.841078    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6369f71_f42c_4c68_98b0_bb1dcb9e1e8e.slice/crio-22cb30f3f21b49d4804313eb8e3de66176c01663aaddfe93acf6d901d6130210 WatchSource:0}: Error finding container 22cb30f3f21b49d4804313eb8e3de66176c01663aaddfe93acf6d901d6130210: Status 404 returned error can't find the container with id 22cb30f3f21b49d4804313eb8e3de66176c01663aaddfe93acf6d901d6130210
Sep 17 21:59:43.913773 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:43.913745    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-775b7455c8-brwr9"]
Sep 17 21:59:43.917008 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:59:43.916977    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod845db971_2b15_4713_8f19_3e9e851b0d37.slice/crio-58cd56155007b4533eaf4ff18f7a6b890c021f8a7bf572ea06b7e40ed09a3fc9 WatchSource:0}: Error finding container 58cd56155007b4533eaf4ff18f7a6b890c021f8a7bf572ea06b7e40ed09a3fc9: Status 404 returned error can't find the container with id 58cd56155007b4533eaf4ff18f7a6b890c021f8a7bf572ea06b7e40ed09a3fc9
Sep 17 21:59:43.990786 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:43.990732    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zddx8"]
Sep 17 21:59:43.991663 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:59:43.991635    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44044918_464c_4ce3_9ace_e88ffeb8b926.slice/crio-6bcf6208d31897d7846534250ac6279d8feb1d52b107d2f18e0d01321769f2df WatchSource:0}: Error finding container 6bcf6208d31897d7846534250ac6279d8feb1d52b107d2f18e0d01321769f2df: Status 404 returned error can't find the container with id 6bcf6208d31897d7846534250ac6279d8feb1d52b107d2f18e0d01321769f2df
Sep 17 21:59:44.102199 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:44.102131    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:44.202571 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:44.202495    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:44.237409 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:44.237366    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zddx8" event={"ID":"44044918-464c-4ce3-9ace-e88ffeb8b926","Type":"ContainerStarted","Data":"6bcf6208d31897d7846534250ac6279d8feb1d52b107d2f18e0d01321769f2df"}
Sep 17 21:59:44.238764 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:44.238737    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-2tt9k" event={"ID":"d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e","Type":"ContainerStarted","Data":"22cb30f3f21b49d4804313eb8e3de66176c01663aaddfe93acf6d901d6130210"}
Sep 17 21:59:44.240490 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:44.240466    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-775b7455c8-brwr9" event={"ID":"845db971-2b15-4713-8f19-3e9e851b0d37","Type":"ContainerStarted","Data":"ccf5a69e41dc1962e1a15e134111f1358cf24716e26b87bfbd409ace0b84a2fa"}
Sep 17 21:59:44.240585 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:44.240512    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-775b7455c8-brwr9" event={"ID":"845db971-2b15-4713-8f19-3e9e851b0d37","Type":"ContainerStarted","Data":"58cd56155007b4533eaf4ff18f7a6b890c021f8a7bf572ea06b7e40ed09a3fc9"}
Sep 17 21:59:44.260110 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:44.260052    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-775b7455c8-brwr9" podStartSLOduration=33.260036242 podStartE2EDuration="33.260036242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:44.259095342 +0000 UTC m=+85.705140615" watchObservedRunningTime="2026-09-17 21:59:44.260036242 +0000 UTC m=+85.706081513"
Sep 17 21:59:44.625886 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:44.625770    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-775b7455c8-brwr9"
Sep 17 21:59:44.628810 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:44.628783    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-775b7455c8-brwr9"
Sep 17 21:59:45.182092 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:45.182052    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6694775df6-dlnxh"]
Sep 17 21:59:45.235462 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:45.235426    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6694775df6-dlnxh"]
Sep 17 21:59:45.235593 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:45.235542    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6694775df6-dlnxh"
Sep 17 21:59:45.238104 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:45.238073    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 21:59:45.238499 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:45.238168    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-qjgsp\""
Sep 17 21:59:45.238499 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:45.238268    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 21:59:45.238993 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:45.238866    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 21:59:45.238993 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:45.238937    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 21:59:45.238993 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:45.238942    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 21:59:45.243953 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:45.243935    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-775b7455c8-brwr9"
Sep 17 21:59:45.245470 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:45.245449    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-775b7455c8-brwr9"
Sep 17 21:59:45.331092 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:45.331061    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4b232899-c172-4e88-8bea-3c2835941d74-console-config\") pod \"console-6694775df6-dlnxh\" (UID: \"4b232899-c172-4e88-8bea-3c2835941d74\") " pod="openshift-console/console-6694775df6-dlnxh"
Sep 17 21:59:45.331258 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:45.331112    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jggxd\" (UniqueName: \"kubernetes.io/projected/4b232899-c172-4e88-8bea-3c2835941d74-kube-api-access-jggxd\") pod \"console-6694775df6-dlnxh\" (UID: \"4b232899-c172-4e88-8bea-3c2835941d74\") " pod="openshift-console/console-6694775df6-dlnxh"
Sep 17 21:59:45.331333 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:45.331243    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b232899-c172-4e88-8bea-3c2835941d74-console-serving-cert\") pod \"console-6694775df6-dlnxh\" (UID: \"4b232899-c172-4e88-8bea-3c2835941d74\") " pod="openshift-console/console-6694775df6-dlnxh"
Sep 17 21:59:45.331333 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:45.331283    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4b232899-c172-4e88-8bea-3c2835941d74-service-ca\") pod \"console-6694775df6-dlnxh\" (UID: \"4b232899-c172-4e88-8bea-3c2835941d74\") " pod="openshift-console/console-6694775df6-dlnxh"
Sep 17 21:59:45.331333 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:45.331319    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4b232899-c172-4e88-8bea-3c2835941d74-console-oauth-config\") pod \"console-6694775df6-dlnxh\" (UID: \"4b232899-c172-4e88-8bea-3c2835941d74\") " pod="openshift-console/console-6694775df6-dlnxh"
Sep 17 21:59:45.331481 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:45.331355    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4b232899-c172-4e88-8bea-3c2835941d74-oauth-serving-cert\") pod \"console-6694775df6-dlnxh\" (UID: \"4b232899-c172-4e88-8bea-3c2835941d74\") " pod="openshift-console/console-6694775df6-dlnxh"
Sep 17 21:59:45.432499 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:45.432422    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b232899-c172-4e88-8bea-3c2835941d74-console-serving-cert\") pod \"console-6694775df6-dlnxh\" (UID: \"4b232899-c172-4e88-8bea-3c2835941d74\") " pod="openshift-console/console-6694775df6-dlnxh"
Sep 17 21:59:45.432499 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:45.432463    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4b232899-c172-4e88-8bea-3c2835941d74-service-ca\") pod \"console-6694775df6-dlnxh\" (UID: \"4b232899-c172-4e88-8bea-3c2835941d74\") " pod="openshift-console/console-6694775df6-dlnxh"
Sep 17 21:59:45.432499 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:45.432485    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4b232899-c172-4e88-8bea-3c2835941d74-console-oauth-config\") pod \"console-6694775df6-dlnxh\" (UID: \"4b232899-c172-4e88-8bea-3c2835941d74\") " pod="openshift-console/console-6694775df6-dlnxh"
Sep 17 21:59:45.432762 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:45.432569    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4b232899-c172-4e88-8bea-3c2835941d74-oauth-serving-cert\") pod \"console-6694775df6-dlnxh\" (UID: \"4b232899-c172-4e88-8bea-3c2835941d74\") " pod="openshift-console/console-6694775df6-dlnxh"
Sep 17 21:59:45.432762 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:45.432717    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4b232899-c172-4e88-8bea-3c2835941d74-console-config\") pod \"console-6694775df6-dlnxh\" (UID: \"4b232899-c172-4e88-8bea-3c2835941d74\") " pod="openshift-console/console-6694775df6-dlnxh"
Sep 17 21:59:45.432947 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:45.432911    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jggxd\" (UniqueName: \"kubernetes.io/projected/4b232899-c172-4e88-8bea-3c2835941d74-kube-api-access-jggxd\") pod \"console-6694775df6-dlnxh\" (UID: \"4b232899-c172-4e88-8bea-3c2835941d74\") " pod="openshift-console/console-6694775df6-dlnxh"
Sep 17 21:59:45.433512 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:45.433486    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4b232899-c172-4e88-8bea-3c2835941d74-service-ca\") pod \"console-6694775df6-dlnxh\" (UID: \"4b232899-c172-4e88-8bea-3c2835941d74\") " pod="openshift-console/console-6694775df6-dlnxh"
Sep 17 21:59:45.433614 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:45.433486    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4b232899-c172-4e88-8bea-3c2835941d74-oauth-serving-cert\") pod \"console-6694775df6-dlnxh\" (UID: \"4b232899-c172-4e88-8bea-3c2835941d74\") " pod="openshift-console/console-6694775df6-dlnxh"
Sep 17 21:59:45.433991 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:45.433968    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4b232899-c172-4e88-8bea-3c2835941d74-console-config\") pod \"console-6694775df6-dlnxh\" (UID: \"4b232899-c172-4e88-8bea-3c2835941d74\") " pod="openshift-console/console-6694775df6-dlnxh"
Sep 17 21:59:45.435552 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:45.435529    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4b232899-c172-4e88-8bea-3c2835941d74-console-oauth-config\") pod \"console-6694775df6-dlnxh\" (UID: \"4b232899-c172-4e88-8bea-3c2835941d74\") " pod="openshift-console/console-6694775df6-dlnxh"
Sep 17 21:59:45.435670 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:45.435649    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b232899-c172-4e88-8bea-3c2835941d74-console-serving-cert\") pod \"console-6694775df6-dlnxh\" (UID: \"4b232899-c172-4e88-8bea-3c2835941d74\") " pod="openshift-console/console-6694775df6-dlnxh"
Sep 17 21:59:45.441504 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:45.441484    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jggxd\" (UniqueName: \"kubernetes.io/projected/4b232899-c172-4e88-8bea-3c2835941d74-kube-api-access-jggxd\") pod \"console-6694775df6-dlnxh\" (UID: \"4b232899-c172-4e88-8bea-3c2835941d74\") " pod="openshift-console/console-6694775df6-dlnxh"
Sep 17 21:59:45.546982 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:45.546948    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6694775df6-dlnxh"
Sep 17 21:59:45.684864 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:45.684769    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:45.777494 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:45.777408    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:46.548238 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:46.548128    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6694775df6-dlnxh"]
Sep 17 21:59:46.550930 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:59:46.550900    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b232899_c172_4e88_8bea_3c2835941d74.slice/crio-0f893748c8d0567657c31de1dc6e2315a6a7684f158ceb4b845eaa8b77a3de48 WatchSource:0}: Error finding container 0f893748c8d0567657c31de1dc6e2315a6a7684f158ceb4b845eaa8b77a3de48: Status 404 returned error can't find the container with id 0f893748c8d0567657c31de1dc6e2315a6a7684f158ceb4b845eaa8b77a3de48
Sep 17 21:59:46.655522 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:46.655492    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:46.755924 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:46.755894    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:46.831626 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:46.831544    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:47.220079 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:47.219933    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:47.256212 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:47.256157    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zddx8" event={"ID":"44044918-464c-4ce3-9ace-e88ffeb8b926","Type":"ContainerStarted","Data":"e24b8435bd1f52b60b538b29a7a034042b4b01d4feb941a690a0c21ccba2bf9b"}
Sep 17 21:59:47.259587 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:47.259559    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6694775df6-dlnxh" event={"ID":"4b232899-c172-4e88-8bea-3c2835941d74","Type":"ContainerStarted","Data":"0f893748c8d0567657c31de1dc6e2315a6a7684f158ceb4b845eaa8b77a3de48"}
Sep 17 21:59:47.263362 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:47.263340    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-2tt9k" event={"ID":"d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e","Type":"ContainerStarted","Data":"c369fd85106e19f5be1e380f3ed57e9f551436d38210c9680aee6acc45bf722b"}
Sep 17 21:59:47.298252 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:47.298201    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zddx8" podStartSLOduration=34.011058344 podStartE2EDuration="36.298170598s" totalImagesPullingTime="2.287112254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:47.277046832 +0000 UTC m=+88.723092104" watchObservedRunningTime="2026-09-17 21:59:47.298170598 +0000 UTC m=+88.744215871"
Sep 17 21:59:47.426925 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:47.426873    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:47.494814 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:47.494712    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:48.965081 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:48.965044    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:49.035117 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:49.035081    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:50.789595 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:50.789561    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab4de47d-58d7-44d5-b1c8-884f9801c218-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pxjbl\" (UID: \"ab4de47d-58d7-44d5-b1c8-884f9801c218\") " pod="openshift-insights/insights-runtime-extractor-pxjbl"
Sep 17 21:59:50.792556 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:50.792503    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab4de47d-58d7-44d5-b1c8-884f9801c218-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pxjbl\" (UID: \"ab4de47d-58d7-44d5-b1c8-884f9801c218\") " pod="openshift-insights/insights-runtime-extractor-pxjbl"
Sep 17 21:59:50.953529 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:50.953499    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g6j2k\""
Sep 17 21:59:50.961488 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:50.961459    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pxjbl"
Sep 17 21:59:51.013291 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:51.013264    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:51.070233 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:51.070094    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:51.228357 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:51.228321    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:51.329578 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:51.329470    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:52.351891 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:52.351855    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:52.415279 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:52.415244    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:54.144379 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:54.144341    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:54.245344 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:54.245298    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:54.349393 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:54.349352    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:54.410950 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:54.410870    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:54.718581 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:54.718490    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:54.818603 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:54.818572    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:55.171344 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:55.171288    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:55.876822 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:55.876779    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:55.957654 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:55.957602    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:56.024268 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:56.024225    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:56.749224 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:56.749175    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:56.811537 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:56.811506    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:57.168044 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:57.168008    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-shx44"
Sep 17 21:59:57.184582 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:57.184532    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-2tt9k" podStartSLOduration=43.753044922 podStartE2EDuration="46.184512536s" totalImagesPullingTime="2.431467614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:47.298442159 +0000 UTC m=+88.744487437" watchObservedRunningTime="2026-09-17 21:59:57.184512536 +0000 UTC m=+98.630557809"
Sep 17 21:59:57.803020 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:57.802984    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:57.951825 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:57.951790    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f7439f6-b83c-437d-9d4b-536775fee0d2-registry-tls\") pod \"image-registry-6975d6b4b9-m69t4\" (UID: \"8f7439f6-b83c-437d-9d4b-536775fee0d2\") " pod="openshift-image-registry/image-registry-6975d6b4b9-m69t4"
Sep 17 21:59:57.954553 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:57.954526    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f7439f6-b83c-437d-9d4b-536775fee0d2-registry-tls\") pod \"image-registry-6975d6b4b9-m69t4\" (UID: \"8f7439f6-b83c-437d-9d4b-536775fee0d2\") " pod="openshift-image-registry/image-registry-6975d6b4b9-m69t4"
Sep 17 21:59:57.957082 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:57.957052    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-h2m4k\""
Sep 17 21:59:57.965297 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:57.965273    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6975d6b4b9-m69t4"
Sep 17 21:59:58.053083 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:58.053014    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c8d2e30-748c-4f7c-a848-f5f0a327726a-metrics-tls\") pod \"dns-default-7jzhx\" (UID: \"8c8d2e30-748c-4f7c-a848-f5f0a327726a\") " pod="openshift-dns/dns-default-7jzhx"
Sep 17 21:59:58.055922 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:58.055895    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c8d2e30-748c-4f7c-a848-f5f0a327726a-metrics-tls\") pod \"dns-default-7jzhx\" (UID: \"8c8d2e30-748c-4f7c-a848-f5f0a327726a\") " pod="openshift-dns/dns-default-7jzhx"
Sep 17 21:59:58.278948 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:58.278919    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mg4m2\""
Sep 17 21:59:58.287199 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:58.287156    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7jzhx"
Sep 17 21:59:58.320318 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:58.320291    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:58.420650 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:58.420621    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:58.979968 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:58.979866    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7jzhx"]
Sep 17 21:59:58.982448 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:59:58.982415    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c8d2e30_748c_4f7c_a848_f5f0a327726a.slice/crio-c8212193bdc8bb3ef8015000ba04394e3045f608d35d88abcdb58c9ff3865ca3 WatchSource:0}: Error finding container c8212193bdc8bb3ef8015000ba04394e3045f608d35d88abcdb58c9ff3865ca3: Status 404 returned error can't find the container with id c8212193bdc8bb3ef8015000ba04394e3045f608d35d88abcdb58c9ff3865ca3
Sep 17 21:59:58.985056 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:58.984879    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pxjbl"]
Sep 17 21:59:58.986362 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:58.986172    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6975d6b4b9-m69t4"]
Sep 17 21:59:58.988917 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:59:58.988889    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4de47d_58d7_44d5_b1c8_884f9801c218.slice/crio-60adb1305d5a93fce55a36186960f99a65a76f37013d42f387dc851bf05c9ccb WatchSource:0}: Error finding container 60adb1305d5a93fce55a36186960f99a65a76f37013d42f387dc851bf05c9ccb: Status 404 returned error can't find the container with id 60adb1305d5a93fce55a36186960f99a65a76f37013d42f387dc851bf05c9ccb
Sep 17 21:59:58.990043 ip-10-0-140-157 kubenswrapper[2555]: W0917 21:59:58.990020    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f7439f6_b83c_437d_9d4b_536775fee0d2.slice/crio-6e136b4567883ba78ee01802f0f9e85b46e99307ba376b9e6a0966dff2bcaafc WatchSource:0}: Error finding container 6e136b4567883ba78ee01802f0f9e85b46e99307ba376b9e6a0966dff2bcaafc: Status 404 returned error can't find the container with id 6e136b4567883ba78ee01802f0f9e85b46e99307ba376b9e6a0966dff2bcaafc
Sep 17 21:59:59.315779 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:59.315675    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pxjbl" event={"ID":"ab4de47d-58d7-44d5-b1c8-884f9801c218","Type":"ContainerStarted","Data":"132c0ba40f898bc955844d99577ebc6b41ae80626799ed08acf35c21058fb3cf"}
Sep 17 21:59:59.315779 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:59.315750    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pxjbl" event={"ID":"ab4de47d-58d7-44d5-b1c8-884f9801c218","Type":"ContainerStarted","Data":"60adb1305d5a93fce55a36186960f99a65a76f37013d42f387dc851bf05c9ccb"}
Sep 17 21:59:59.317795 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:59.317754    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-n2h8c" event={"ID":"1c48dd88-220b-45c2-a407-1240b4294030","Type":"ContainerStarted","Data":"20c4662d6d9fe4f5b760f0fdc5de91c46454d9b4620e2a9c4df02d4ffe8e1518"}
Sep 17 21:59:59.318078 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:59.318010    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-n2h8c"
Sep 17 21:59:59.320392 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:59.319749    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6975d6b4b9-m69t4" event={"ID":"8f7439f6-b83c-437d-9d4b-536775fee0d2","Type":"ContainerStarted","Data":"fbb9118a647cea9bd85c7361e36f4567f41b65dd7bdd36df1fdb4ea9ed10e001"}
Sep 17 21:59:59.320392 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:59.319794    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6975d6b4b9-m69t4" event={"ID":"8f7439f6-b83c-437d-9d4b-536775fee0d2","Type":"ContainerStarted","Data":"6e136b4567883ba78ee01802f0f9e85b46e99307ba376b9e6a0966dff2bcaafc"}
Sep 17 21:59:59.320392 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:59.320332    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6975d6b4b9-m69t4"
Sep 17 21:59:59.321780 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:59.321757    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7jzhx" event={"ID":"8c8d2e30-748c-4f7c-a848-f5f0a327726a","Type":"ContainerStarted","Data":"c8212193bdc8bb3ef8015000ba04394e3045f608d35d88abcdb58c9ff3865ca3"}
Sep 17 21:59:59.323655 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:59.323631    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6694775df6-dlnxh" event={"ID":"4b232899-c172-4e88-8bea-3c2835941d74","Type":"ContainerStarted","Data":"82ec84332f90a7da879da67b4b88e2c5c61873e4b68f6dc1d047c6a1ccf2074c"}
Sep 17 21:59:59.331413 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:59.331350    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-n2h8c"
Sep 17 21:59:59.336003 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:59.335948    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-n2h8c" podStartSLOduration=1.405075896 podStartE2EDuration="19.335931163s" totalImagesPullingTime="17.930855267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:59.334922402 +0000 UTC m=+100.780967676" watchObservedRunningTime="2026-09-17 21:59:59.335931163 +0000 UTC m=+100.781976434"
Sep 17 21:59:59.357288 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:59.357237    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6694775df6-dlnxh" podStartSLOduration=2.207727014 podStartE2EDuration="14.357218592s" totalImagesPullingTime="12.149491578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:59.356561384 +0000 UTC m=+100.802606657" watchObservedRunningTime="2026-09-17 21:59:59.357218592 +0000 UTC m=+100.803263863"
Sep 17 21:59:59.379734 ip-10-0-140-157 kubenswrapper[2555]: I0917 21:59:59.379683    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6975d6b4b9-m69t4" podStartSLOduration=76.379663397 podStartE2EDuration="1m16.379663397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:58:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:59.377940005 +0000 UTC m=+100.823985277" watchObservedRunningTime="2026-09-17 21:59:59.379663397 +0000 UTC m=+100.825708671"
Sep 17 22:00:00.068251 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.068177    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:00.160332 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.160136    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:00.309211 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.309144    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:00.331344 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.331309    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pxjbl" event={"ID":"ab4de47d-58d7-44d5-b1c8-884f9801c218","Type":"ContainerStarted","Data":"8be785b16a943ab1945522ee0a2826a9f820bdd5d7047664f9104e61801416bf"}
Sep 17 22:00:00.392893 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.392837    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6694775df6-dlnxh"]
Sep 17 22:00:00.394289 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.394261    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jxgq6"]
Sep 17 22:00:00.412913 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.412848    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jxgq6"
Sep 17 22:00:00.420505 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.419015    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 22:00:00.420505 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.419441    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 22:00:00.420505 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.419666    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 22:00:00.420505 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.419850    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hzs49\""
Sep 17 22:00:00.420505 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.420026    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 22:00:00.435491 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.435461    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:00.562917 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.562833    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:00.578416 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.578012    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1d57efce-da57-4e90-8063-9781be3cda5b-root\") pod \"node-exporter-jxgq6\" (UID: \"1d57efce-da57-4e90-8063-9781be3cda5b\") " pod="openshift-monitoring/node-exporter-jxgq6"
Sep 17 22:00:00.578416 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.578053    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d57efce-da57-4e90-8063-9781be3cda5b-metrics-client-ca\") pod \"node-exporter-jxgq6\" (UID: \"1d57efce-da57-4e90-8063-9781be3cda5b\") " pod="openshift-monitoring/node-exporter-jxgq6"
Sep 17 22:00:00.578416 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.578083    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d57efce-da57-4e90-8063-9781be3cda5b-sys\") pod \"node-exporter-jxgq6\" (UID: \"1d57efce-da57-4e90-8063-9781be3cda5b\") " pod="openshift-monitoring/node-exporter-jxgq6"
Sep 17 22:00:00.578416 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.578152    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvxhj\" (UniqueName: \"kubernetes.io/projected/1d57efce-da57-4e90-8063-9781be3cda5b-kube-api-access-nvxhj\") pod \"node-exporter-jxgq6\" (UID: \"1d57efce-da57-4e90-8063-9781be3cda5b\") " pod="openshift-monitoring/node-exporter-jxgq6"
Sep 17 22:00:00.578416 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.578220    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1d57efce-da57-4e90-8063-9781be3cda5b-node-exporter-wtmp\") pod \"node-exporter-jxgq6\" (UID: \"1d57efce-da57-4e90-8063-9781be3cda5b\") " pod="openshift-monitoring/node-exporter-jxgq6"
Sep 17 22:00:00.578416 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.578250    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1d57efce-da57-4e90-8063-9781be3cda5b-node-exporter-accelerators-collector-config\") pod \"node-exporter-jxgq6\" (UID: \"1d57efce-da57-4e90-8063-9781be3cda5b\") " pod="openshift-monitoring/node-exporter-jxgq6"
Sep 17 22:00:00.578416 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.578283    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1d57efce-da57-4e90-8063-9781be3cda5b-node-exporter-textfile\") pod \"node-exporter-jxgq6\" (UID: \"1d57efce-da57-4e90-8063-9781be3cda5b\") " pod="openshift-monitoring/node-exporter-jxgq6"
Sep 17 22:00:00.578416 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.578308    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d57efce-da57-4e90-8063-9781be3cda5b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jxgq6\" (UID: \"1d57efce-da57-4e90-8063-9781be3cda5b\") " pod="openshift-monitoring/node-exporter-jxgq6"
Sep 17 22:00:00.578416 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.578338    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1d57efce-da57-4e90-8063-9781be3cda5b-node-exporter-tls\") pod \"node-exporter-jxgq6\" (UID: \"1d57efce-da57-4e90-8063-9781be3cda5b\") " pod="openshift-monitoring/node-exporter-jxgq6"
Sep 17 22:00:00.679615 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.679530    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvxhj\" (UniqueName: \"kubernetes.io/projected/1d57efce-da57-4e90-8063-9781be3cda5b-kube-api-access-nvxhj\") pod \"node-exporter-jxgq6\" (UID: \"1d57efce-da57-4e90-8063-9781be3cda5b\") " pod="openshift-monitoring/node-exporter-jxgq6"
Sep 17 22:00:00.679615 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.679579    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1d57efce-da57-4e90-8063-9781be3cda5b-node-exporter-wtmp\") pod \"node-exporter-jxgq6\" (UID: \"1d57efce-da57-4e90-8063-9781be3cda5b\") " pod="openshift-monitoring/node-exporter-jxgq6"
Sep 17 22:00:00.679615 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.679607    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1d57efce-da57-4e90-8063-9781be3cda5b-node-exporter-accelerators-collector-config\") pod \"node-exporter-jxgq6\" (UID: \"1d57efce-da57-4e90-8063-9781be3cda5b\") " pod="openshift-monitoring/node-exporter-jxgq6"
Sep 17 22:00:00.679897 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.679635    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1d57efce-da57-4e90-8063-9781be3cda5b-node-exporter-textfile\") pod \"node-exporter-jxgq6\" (UID: \"1d57efce-da57-4e90-8063-9781be3cda5b\") " pod="openshift-monitoring/node-exporter-jxgq6"
Sep 17 22:00:00.679897 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.679659    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d57efce-da57-4e90-8063-9781be3cda5b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jxgq6\" (UID: \"1d57efce-da57-4e90-8063-9781be3cda5b\") " pod="openshift-monitoring/node-exporter-jxgq6"
Sep 17 22:00:00.679897 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.679764    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1d57efce-da57-4e90-8063-9781be3cda5b-node-exporter-tls\") pod \"node-exporter-jxgq6\" (UID: \"1d57efce-da57-4e90-8063-9781be3cda5b\") " pod="openshift-monitoring/node-exporter-jxgq6"
Sep 17 22:00:00.680155 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.680135    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1d57efce-da57-4e90-8063-9781be3cda5b-root\") pod \"node-exporter-jxgq6\" (UID: \"1d57efce-da57-4e90-8063-9781be3cda5b\") " pod="openshift-monitoring/node-exporter-jxgq6"
Sep 17 22:00:00.680307 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.680259    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1d57efce-da57-4e90-8063-9781be3cda5b-node-exporter-wtmp\") pod \"node-exporter-jxgq6\" (UID: \"1d57efce-da57-4e90-8063-9781be3cda5b\") " pod="openshift-monitoring/node-exporter-jxgq6"
Sep 17 22:00:00.680555 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.680136    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1d57efce-da57-4e90-8063-9781be3cda5b-node-exporter-textfile\") pod \"node-exporter-jxgq6\" (UID: \"1d57efce-da57-4e90-8063-9781be3cda5b\") " pod="openshift-monitoring/node-exporter-jxgq6"
Sep 17 22:00:00.683217 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.680890    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1d57efce-da57-4e90-8063-9781be3cda5b-root\") pod \"node-exporter-jxgq6\" (UID: \"1d57efce-da57-4e90-8063-9781be3cda5b\") " pod="openshift-monitoring/node-exporter-jxgq6"
Sep 17 22:00:00.683217 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.680936    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d57efce-da57-4e90-8063-9781be3cda5b-metrics-client-ca\") pod \"node-exporter-jxgq6\" (UID: \"1d57efce-da57-4e90-8063-9781be3cda5b\") " pod="openshift-monitoring/node-exporter-jxgq6"
Sep 17 22:00:00.683217 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.681501    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1d57efce-da57-4e90-8063-9781be3cda5b-node-exporter-accelerators-collector-config\") pod \"node-exporter-jxgq6\" (UID: \"1d57efce-da57-4e90-8063-9781be3cda5b\") " pod="openshift-monitoring/node-exporter-jxgq6"
Sep 17 22:00:00.683217 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.681966    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d57efce-da57-4e90-8063-9781be3cda5b-sys\") pod \"node-exporter-jxgq6\" (UID: \"1d57efce-da57-4e90-8063-9781be3cda5b\") " pod="openshift-monitoring/node-exporter-jxgq6"
Sep 17 22:00:00.683217 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.682172    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d57efce-da57-4e90-8063-9781be3cda5b-sys\") pod \"node-exporter-jxgq6\" (UID: \"1d57efce-da57-4e90-8063-9781be3cda5b\") " pod="openshift-monitoring/node-exporter-jxgq6"
Sep 17 22:00:00.683217 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.682669    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d57efce-da57-4e90-8063-9781be3cda5b-metrics-client-ca\") pod \"node-exporter-jxgq6\" (UID: \"1d57efce-da57-4e90-8063-9781be3cda5b\") " pod="openshift-monitoring/node-exporter-jxgq6"
Sep 17 22:00:00.683217 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.682953    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:00.683608 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.683348    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1d57efce-da57-4e90-8063-9781be3cda5b-node-exporter-tls\") pod \"node-exporter-jxgq6\" (UID: \"1d57efce-da57-4e90-8063-9781be3cda5b\") " pod="openshift-monitoring/node-exporter-jxgq6"
Sep 17 22:00:00.684478 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.684323    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d57efce-da57-4e90-8063-9781be3cda5b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jxgq6\" (UID: \"1d57efce-da57-4e90-8063-9781be3cda5b\") " pod="openshift-monitoring/node-exporter-jxgq6"
Sep 17 22:00:00.692087 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.691981    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvxhj\" (UniqueName: \"kubernetes.io/projected/1d57efce-da57-4e90-8063-9781be3cda5b-kube-api-access-nvxhj\") pod \"node-exporter-jxgq6\" (UID: \"1d57efce-da57-4e90-8063-9781be3cda5b\") " pod="openshift-monitoring/node-exporter-jxgq6"
Sep 17 22:00:00.728652 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.728151    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jxgq6"
Sep 17 22:00:00.780820 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.780768    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:00.879977 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.879934    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:00.992371 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:00.992170    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:01.054142 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:00:01.054105    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d57efce_da57_4e90_8063_9781be3cda5b.slice/crio-f335707f6d61f59c1730b4e8f3688b15e2ea7099fe151672bb7c1a7da1cf9eb6 WatchSource:0}: Error finding container f335707f6d61f59c1730b4e8f3688b15e2ea7099fe151672bb7c1a7da1cf9eb6: Status 404 returned error can't find the container with id f335707f6d61f59c1730b4e8f3688b15e2ea7099fe151672bb7c1a7da1cf9eb6
Sep 17 22:00:01.337383 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:01.337202    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jxgq6" event={"ID":"1d57efce-da57-4e90-8063-9781be3cda5b","Type":"ContainerStarted","Data":"f335707f6d61f59c1730b4e8f3688b15e2ea7099fe151672bb7c1a7da1cf9eb6"}
Sep 17 22:00:01.429823 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:01.429786    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:01.656529 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:01.656488    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:01.787552 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:01.787449    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:02.106130 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:02.106037    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6975d6b4b9-m69t4"]
Sep 17 22:00:02.182811 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:02.182714    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:02.345505 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:02.345167    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7jzhx" event={"ID":"8c8d2e30-748c-4f7c-a848-f5f0a327726a","Type":"ContainerStarted","Data":"566138f81fea5fee04705c34ea0e557dc041d21f7db9edc18e869ee2978215b2"}
Sep 17 22:00:02.345505 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:02.345270    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7jzhx" event={"ID":"8c8d2e30-748c-4f7c-a848-f5f0a327726a","Type":"ContainerStarted","Data":"bf1ae0d1bb8379d71f538a646669a33154c6dc1c377a71867e0dd5a8272ba3d7"}
Sep 17 22:00:02.345505 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:02.345500    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7jzhx"
Sep 17 22:00:02.373263 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:02.373229    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:02.385923 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:02.385869    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-7jzhx" podStartSLOduration=65.902412637 podStartE2EDuration="1m8.385847896s" totalImagesPullingTime="2.483435259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:58:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:00:02.364550229 +0000 UTC m=+103.810595502" watchObservedRunningTime="2026-09-17 22:00:02.385847896 +0000 UTC m=+103.831893174"
Sep 17 22:00:02.623966 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:02.623888    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:03.316006 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:03.315927    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:03.356377 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:03.356335    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pxjbl" event={"ID":"ab4de47d-58d7-44d5-b1c8-884f9801c218","Type":"ContainerStarted","Data":"78d1f7b3d9a76e92b8d561bb788f428d4e4b343b8f448da5ea4438b222e2cc38"}
Sep 17 22:00:03.383233 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:03.382304    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pxjbl" podStartSLOduration=41.683234809 podStartE2EDuration="45.382280692s" totalImagesPullingTime="3.699045883s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:59:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:00:03.379439688 +0000 UTC m=+104.825484961" watchObservedRunningTime="2026-09-17 22:00:03.382280692 +0000 UTC m=+104.828325965"
Sep 17 22:00:03.726161 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:03.726125    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:04.289560 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:04.289475    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:05.087925 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:05.087888    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:05.173955 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:05.173922    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:05.365560 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:05.365471    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:05.373527 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:05.373493    2555 generic.go:356] "Generic (PLEG): container finished" podID="1d57efce-da57-4e90-8063-9781be3cda5b" containerID="8d173c18db34d9c696062044a43dce4baa10cbe62383fec0e7ad4d44d44058d9" exitCode=0
Sep 17 22:00:05.373703 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:05.373549    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jxgq6" event={"ID":"1d57efce-da57-4e90-8063-9781be3cda5b","Type":"ContainerDied","Data":"8d173c18db34d9c696062044a43dce4baa10cbe62383fec0e7ad4d44d44058d9"}
Sep 17 22:00:05.481354 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:05.481320    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:05.547526 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:05.547494    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6694775df6-dlnxh"
Sep 17 22:00:05.874040 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:05.874004    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:05.937681 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:05.937591    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:06.349909 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:06.349820    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:06.380260 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:06.380223    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jxgq6" event={"ID":"1d57efce-da57-4e90-8063-9781be3cda5b","Type":"ContainerStarted","Data":"5d5ba921376170c159085067c004a6fc0e12d5ee2a0482baf7552356c764c362"}
Sep 17 22:00:06.380451 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:06.380284    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jxgq6" event={"ID":"1d57efce-da57-4e90-8063-9781be3cda5b","Type":"ContainerStarted","Data":"a0bfe54d34177440b7cc6f58f591de3dfaa654fe85288fb55530406c39ef5f8d"}
Sep 17 22:00:06.413053 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:06.412990    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jxgq6" podStartSLOduration=3.14981973 podStartE2EDuration="6.412972545s" totalImagesPullingTime="3.175214706s" totalInitContainerRuntime="87.938109ms" isStatefulPod=true podCreationTimestamp="2026-09-17 22:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:00:06.410671654 +0000 UTC m=+107.856716931" watchObservedRunningTime="2026-09-17 22:00:06.412972545 +0000 UTC m=+107.859017818"
Sep 17 22:00:06.448865 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:06.448824    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:06.678513 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:06.678475    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:10.378690 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:10.378659    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:10.432117 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:10.432088    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:12.357968 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:12.357934    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7jzhx"
Sep 17 22:00:12.610790 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:12.610697    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:12.675026 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:12.674988    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:15.128632 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:15.128595    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:15.193069 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:15.193038    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:15.619374 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:15.619279    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:15.720701 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:15.720666    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:15.819641 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:15.819612    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:15.920442 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:15.920409    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:21.028315 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:21.028282    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:21.116347 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:21.116308    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:21.660679 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:21.660620    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:21.722716 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:21.722688    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:22.351893 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:22.351866    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6975d6b4b9-m69t4"
Sep 17 22:00:26.359245 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:26.359199    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6694775df6-dlnxh" podUID="4b232899-c172-4e88-8bea-3c2835941d74" containerName="console" containerID="cri-o://82ec84332f90a7da879da67b4b88e2c5c61873e4b68f6dc1d047c6a1ccf2074c" gracePeriod=15
Sep 17 22:00:26.596724 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:26.596701    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6694775df6-dlnxh"
Sep 17 22:00:26.701103 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:26.701075    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4b232899-c172-4e88-8bea-3c2835941d74-oauth-serving-cert\") pod \"4b232899-c172-4e88-8bea-3c2835941d74\" (UID: \"4b232899-c172-4e88-8bea-3c2835941d74\") "
Sep 17 22:00:26.701291 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:26.701128    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4b232899-c172-4e88-8bea-3c2835941d74-console-oauth-config\") pod \"4b232899-c172-4e88-8bea-3c2835941d74\" (UID: \"4b232899-c172-4e88-8bea-3c2835941d74\") "
Sep 17 22:00:26.701291 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:26.701159    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jggxd\" (UniqueName: \"kubernetes.io/projected/4b232899-c172-4e88-8bea-3c2835941d74-kube-api-access-jggxd\") pod \"4b232899-c172-4e88-8bea-3c2835941d74\" (UID: \"4b232899-c172-4e88-8bea-3c2835941d74\") "
Sep 17 22:00:26.701291 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:26.701207    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b232899-c172-4e88-8bea-3c2835941d74-console-serving-cert\") pod \"4b232899-c172-4e88-8bea-3c2835941d74\" (UID: \"4b232899-c172-4e88-8bea-3c2835941d74\") "
Sep 17 22:00:26.701291 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:26.701263    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4b232899-c172-4e88-8bea-3c2835941d74-console-config\") pod \"4b232899-c172-4e88-8bea-3c2835941d74\" (UID: \"4b232899-c172-4e88-8bea-3c2835941d74\") "
Sep 17 22:00:26.701505 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:26.701299    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4b232899-c172-4e88-8bea-3c2835941d74-service-ca\") pod \"4b232899-c172-4e88-8bea-3c2835941d74\" (UID: \"4b232899-c172-4e88-8bea-3c2835941d74\") "
Sep 17 22:00:26.701630 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:26.701582    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b232899-c172-4e88-8bea-3c2835941d74-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4b232899-c172-4e88-8bea-3c2835941d74" (UID: "4b232899-c172-4e88-8bea-3c2835941d74"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:00:26.701733 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:26.701628    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b232899-c172-4e88-8bea-3c2835941d74-console-config" (OuterVolumeSpecName: "console-config") pod "4b232899-c172-4e88-8bea-3c2835941d74" (UID: "4b232899-c172-4e88-8bea-3c2835941d74"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:00:26.701733 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:26.701703    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b232899-c172-4e88-8bea-3c2835941d74-service-ca" (OuterVolumeSpecName: "service-ca") pod "4b232899-c172-4e88-8bea-3c2835941d74" (UID: "4b232899-c172-4e88-8bea-3c2835941d74"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:00:26.703562 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:26.703538    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b232899-c172-4e88-8bea-3c2835941d74-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4b232899-c172-4e88-8bea-3c2835941d74" (UID: "4b232899-c172-4e88-8bea-3c2835941d74"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:00:26.703784 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:26.703766    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b232899-c172-4e88-8bea-3c2835941d74-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4b232899-c172-4e88-8bea-3c2835941d74" (UID: "4b232899-c172-4e88-8bea-3c2835941d74"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:00:26.703859 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:26.703798    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b232899-c172-4e88-8bea-3c2835941d74-kube-api-access-jggxd" (OuterVolumeSpecName: "kube-api-access-jggxd") pod "4b232899-c172-4e88-8bea-3c2835941d74" (UID: "4b232899-c172-4e88-8bea-3c2835941d74"). InnerVolumeSpecName "kube-api-access-jggxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:00:26.802360 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:26.802329    2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b232899-c172-4e88-8bea-3c2835941d74-console-serving-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:00:26.802360 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:26.802355    2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4b232899-c172-4e88-8bea-3c2835941d74-console-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:00:26.802360 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:26.802363    2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4b232899-c172-4e88-8bea-3c2835941d74-service-ca\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:00:26.802552 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:26.802371    2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4b232899-c172-4e88-8bea-3c2835941d74-oauth-serving-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:00:26.802552 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:26.802380    2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4b232899-c172-4e88-8bea-3c2835941d74-console-oauth-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:00:26.802552 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:26.802388    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jggxd\" (UniqueName: \"kubernetes.io/projected/4b232899-c172-4e88-8bea-3c2835941d74-kube-api-access-jggxd\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:00:27.373050 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:27.372998    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6975d6b4b9-m69t4" podUID="8f7439f6-b83c-437d-9d4b-536775fee0d2" containerName="registry" containerID="cri-o://fbb9118a647cea9bd85c7361e36f4567f41b65dd7bdd36df1fdb4ea9ed10e001" gracePeriod=30
Sep 17 22:00:27.458099 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:27.458062    2555 generic.go:356] "Generic (PLEG): container finished" podID="4b232899-c172-4e88-8bea-3c2835941d74" containerID="82ec84332f90a7da879da67b4b88e2c5c61873e4b68f6dc1d047c6a1ccf2074c" exitCode=0
Sep 17 22:00:27.458268 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:27.458108    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6694775df6-dlnxh"
Sep 17 22:00:27.458268 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:27.458141    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6694775df6-dlnxh" event={"ID":"4b232899-c172-4e88-8bea-3c2835941d74","Type":"ContainerDied","Data":"82ec84332f90a7da879da67b4b88e2c5c61873e4b68f6dc1d047c6a1ccf2074c"}
Sep 17 22:00:27.458268 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:27.458207    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6694775df6-dlnxh" event={"ID":"4b232899-c172-4e88-8bea-3c2835941d74","Type":"ContainerDied","Data":"0f893748c8d0567657c31de1dc6e2315a6a7684f158ceb4b845eaa8b77a3de48"}
Sep 17 22:00:27.458268 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:27.458231    2555 scope.go:117] "RemoveContainer" containerID="82ec84332f90a7da879da67b4b88e2c5c61873e4b68f6dc1d047c6a1ccf2074c"
Sep 17 22:00:27.479458 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:27.479434    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6694775df6-dlnxh"]
Sep 17 22:00:27.482233 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:27.482210    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6694775df6-dlnxh"]
Sep 17 22:00:27.483414 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:27.483392    2555 scope.go:117] "RemoveContainer" containerID="82ec84332f90a7da879da67b4b88e2c5c61873e4b68f6dc1d047c6a1ccf2074c"
Sep 17 22:00:27.483725 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:00:27.483699    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82ec84332f90a7da879da67b4b88e2c5c61873e4b68f6dc1d047c6a1ccf2074c\": container with ID starting with 82ec84332f90a7da879da67b4b88e2c5c61873e4b68f6dc1d047c6a1ccf2074c not found: ID does not exist" containerID="82ec84332f90a7da879da67b4b88e2c5c61873e4b68f6dc1d047c6a1ccf2074c"
Sep 17 22:00:27.483797 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:27.483738    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82ec84332f90a7da879da67b4b88e2c5c61873e4b68f6dc1d047c6a1ccf2074c"} err="failed to get container status \"82ec84332f90a7da879da67b4b88e2c5c61873e4b68f6dc1d047c6a1ccf2074c\": rpc error: code = NotFound desc = could not find container \"82ec84332f90a7da879da67b4b88e2c5c61873e4b68f6dc1d047c6a1ccf2074c\": container with ID starting with 82ec84332f90a7da879da67b4b88e2c5c61873e4b68f6dc1d047c6a1ccf2074c not found: ID does not exist"
Sep 17 22:00:27.616129 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:27.616101    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6975d6b4b9-m69t4"
Sep 17 22:00:27.709143 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:27.709119    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f7439f6-b83c-437d-9d4b-536775fee0d2-bound-sa-token\") pod \"8f7439f6-b83c-437d-9d4b-536775fee0d2\" (UID: \"8f7439f6-b83c-437d-9d4b-536775fee0d2\") "
Sep 17 22:00:27.709294 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:27.709158    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8f7439f6-b83c-437d-9d4b-536775fee0d2-ca-trust-extracted\") pod \"8f7439f6-b83c-437d-9d4b-536775fee0d2\" (UID: \"8f7439f6-b83c-437d-9d4b-536775fee0d2\") "
Sep 17 22:00:27.709294 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:27.709200    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qn7x5\" (UniqueName: \"kubernetes.io/projected/8f7439f6-b83c-437d-9d4b-536775fee0d2-kube-api-access-qn7x5\") pod \"8f7439f6-b83c-437d-9d4b-536775fee0d2\" (UID: \"8f7439f6-b83c-437d-9d4b-536775fee0d2\") "
Sep 17 22:00:27.709294 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:27.709244    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8f7439f6-b83c-437d-9d4b-536775fee0d2-registry-certificates\") pod \"8f7439f6-b83c-437d-9d4b-536775fee0d2\" (UID: \"8f7439f6-b83c-437d-9d4b-536775fee0d2\") "
Sep 17 22:00:27.709294 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:27.709276    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f7439f6-b83c-437d-9d4b-536775fee0d2-registry-tls\") pod \"8f7439f6-b83c-437d-9d4b-536775fee0d2\" (UID: \"8f7439f6-b83c-437d-9d4b-536775fee0d2\") "
Sep 17 22:00:27.709480 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:27.709299    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8f7439f6-b83c-437d-9d4b-536775fee0d2-image-registry-private-configuration\") pod \"8f7439f6-b83c-437d-9d4b-536775fee0d2\" (UID: \"8f7439f6-b83c-437d-9d4b-536775fee0d2\") "
Sep 17 22:00:27.709480 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:27.709323    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8f7439f6-b83c-437d-9d4b-536775fee0d2-trusted-ca\") pod \"8f7439f6-b83c-437d-9d4b-536775fee0d2\" (UID: \"8f7439f6-b83c-437d-9d4b-536775fee0d2\") "
Sep 17 22:00:27.709480 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:27.709362    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8f7439f6-b83c-437d-9d4b-536775fee0d2-installation-pull-secrets\") pod \"8f7439f6-b83c-437d-9d4b-536775fee0d2\" (UID: \"8f7439f6-b83c-437d-9d4b-536775fee0d2\") "
Sep 17 22:00:27.709749 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:27.709723    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f7439f6-b83c-437d-9d4b-536775fee0d2-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "8f7439f6-b83c-437d-9d4b-536775fee0d2" (UID: "8f7439f6-b83c-437d-9d4b-536775fee0d2"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:00:27.709822 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:27.709802    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f7439f6-b83c-437d-9d4b-536775fee0d2-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "8f7439f6-b83c-437d-9d4b-536775fee0d2" (UID: "8f7439f6-b83c-437d-9d4b-536775fee0d2"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:00:27.711927 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:27.711892    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f7439f6-b83c-437d-9d4b-536775fee0d2-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "8f7439f6-b83c-437d-9d4b-536775fee0d2" (UID: "8f7439f6-b83c-437d-9d4b-536775fee0d2"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:00:27.712055 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:27.712032    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f7439f6-b83c-437d-9d4b-536775fee0d2-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "8f7439f6-b83c-437d-9d4b-536775fee0d2" (UID: "8f7439f6-b83c-437d-9d4b-536775fee0d2"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:00:27.712055 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:27.712042    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f7439f6-b83c-437d-9d4b-536775fee0d2-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "8f7439f6-b83c-437d-9d4b-536775fee0d2" (UID: "8f7439f6-b83c-437d-9d4b-536775fee0d2"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:00:27.712162 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:27.712084    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f7439f6-b83c-437d-9d4b-536775fee0d2-kube-api-access-qn7x5" (OuterVolumeSpecName: "kube-api-access-qn7x5") pod "8f7439f6-b83c-437d-9d4b-536775fee0d2" (UID: "8f7439f6-b83c-437d-9d4b-536775fee0d2"). InnerVolumeSpecName "kube-api-access-qn7x5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:00:27.712162 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:27.712104    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f7439f6-b83c-437d-9d4b-536775fee0d2-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8f7439f6-b83c-437d-9d4b-536775fee0d2" (UID: "8f7439f6-b83c-437d-9d4b-536775fee0d2"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:00:27.717301 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:27.717280    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f7439f6-b83c-437d-9d4b-536775fee0d2-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "8f7439f6-b83c-437d-9d4b-536775fee0d2" (UID: "8f7439f6-b83c-437d-9d4b-536775fee0d2"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:00:27.810507 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:27.810483    2555 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f7439f6-b83c-437d-9d4b-536775fee0d2-registry-tls\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:00:27.810507 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:27.810504    2555 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8f7439f6-b83c-437d-9d4b-536775fee0d2-image-registry-private-configuration\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:00:27.810619 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:27.810514    2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8f7439f6-b83c-437d-9d4b-536775fee0d2-trusted-ca\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:00:27.810619 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:27.810522    2555 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8f7439f6-b83c-437d-9d4b-536775fee0d2-installation-pull-secrets\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:00:27.810619 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:27.810531    2555 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f7439f6-b83c-437d-9d4b-536775fee0d2-bound-sa-token\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:00:27.810619 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:27.810539    2555 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8f7439f6-b83c-437d-9d4b-536775fee0d2-ca-trust-extracted\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:00:27.810619 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:27.810547    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn7x5\" (UniqueName: \"kubernetes.io/projected/8f7439f6-b83c-437d-9d4b-536775fee0d2-kube-api-access-qn7x5\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:00:27.810619 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:27.810556    2555 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8f7439f6-b83c-437d-9d4b-536775fee0d2-registry-certificates\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:00:28.463859 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:28.463832    2555 generic.go:356] "Generic (PLEG): container finished" podID="b90aafb6-01ac-4c30-a30a-530c91d1c944" containerID="fab6e903227d51cc327ddda970a490f8ae43fbc7534ec8c0d606f8d7e46d98c8" exitCode=0
Sep 17 22:00:28.464253 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:28.463904    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-2kws6" event={"ID":"b90aafb6-01ac-4c30-a30a-530c91d1c944","Type":"ContainerDied","Data":"fab6e903227d51cc327ddda970a490f8ae43fbc7534ec8c0d606f8d7e46d98c8"}
Sep 17 22:00:28.464253 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:28.464244    2555 scope.go:117] "RemoveContainer" containerID="fab6e903227d51cc327ddda970a490f8ae43fbc7534ec8c0d606f8d7e46d98c8"
Sep 17 22:00:28.465612 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:28.465589    2555 generic.go:356] "Generic (PLEG): container finished" podID="6b79292b-9dd7-4560-8a09-80ee28ec9a4b" containerID="bb554b77f4bb72d527e84bd5d3678050df2308d6fef49673fab4f21941449544" exitCode=0
Sep 17 22:00:28.465669 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:28.465655    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-db9vd" event={"ID":"6b79292b-9dd7-4560-8a09-80ee28ec9a4b","Type":"ContainerDied","Data":"bb554b77f4bb72d527e84bd5d3678050df2308d6fef49673fab4f21941449544"}
Sep 17 22:00:28.465984 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:28.465968    2555 scope.go:117] "RemoveContainer" containerID="bb554b77f4bb72d527e84bd5d3678050df2308d6fef49673fab4f21941449544"
Sep 17 22:00:28.467936 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:28.467917    2555 generic.go:356] "Generic (PLEG): container finished" podID="8f7439f6-b83c-437d-9d4b-536775fee0d2" containerID="fbb9118a647cea9bd85c7361e36f4567f41b65dd7bdd36df1fdb4ea9ed10e001" exitCode=0
Sep 17 22:00:28.467985 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:28.467949    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6975d6b4b9-m69t4"
Sep 17 22:00:28.467985 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:28.467956    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6975d6b4b9-m69t4" event={"ID":"8f7439f6-b83c-437d-9d4b-536775fee0d2","Type":"ContainerDied","Data":"fbb9118a647cea9bd85c7361e36f4567f41b65dd7bdd36df1fdb4ea9ed10e001"}
Sep 17 22:00:28.468157 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:28.467996    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6975d6b4b9-m69t4" event={"ID":"8f7439f6-b83c-437d-9d4b-536775fee0d2","Type":"ContainerDied","Data":"6e136b4567883ba78ee01802f0f9e85b46e99307ba376b9e6a0966dff2bcaafc"}
Sep 17 22:00:28.468157 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:28.468023    2555 scope.go:117] "RemoveContainer" containerID="fbb9118a647cea9bd85c7361e36f4567f41b65dd7bdd36df1fdb4ea9ed10e001"
Sep 17 22:00:28.478295 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:28.478252    2555 scope.go:117] "RemoveContainer" containerID="fbb9118a647cea9bd85c7361e36f4567f41b65dd7bdd36df1fdb4ea9ed10e001"
Sep 17 22:00:28.478580 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:00:28.478562    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbb9118a647cea9bd85c7361e36f4567f41b65dd7bdd36df1fdb4ea9ed10e001\": container with ID starting with fbb9118a647cea9bd85c7361e36f4567f41b65dd7bdd36df1fdb4ea9ed10e001 not found: ID does not exist" containerID="fbb9118a647cea9bd85c7361e36f4567f41b65dd7bdd36df1fdb4ea9ed10e001"
Sep 17 22:00:28.478640 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:28.478587    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbb9118a647cea9bd85c7361e36f4567f41b65dd7bdd36df1fdb4ea9ed10e001"} err="failed to get container status \"fbb9118a647cea9bd85c7361e36f4567f41b65dd7bdd36df1fdb4ea9ed10e001\": rpc error: code = NotFound desc = could not find container \"fbb9118a647cea9bd85c7361e36f4567f41b65dd7bdd36df1fdb4ea9ed10e001\": container with ID starting with fbb9118a647cea9bd85c7361e36f4567f41b65dd7bdd36df1fdb4ea9ed10e001 not found: ID does not exist"
Sep 17 22:00:28.509603 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:28.509575    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6975d6b4b9-m69t4"]
Sep 17 22:00:28.515272 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:28.515240    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6975d6b4b9-m69t4"]
Sep 17 22:00:28.678971 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:28.678891    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b232899-c172-4e88-8bea-3c2835941d74" path="/var/lib/kubelet/pods/4b232899-c172-4e88-8bea-3c2835941d74/volumes"
Sep 17 22:00:28.679327 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:28.679312    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f7439f6-b83c-437d-9d4b-536775fee0d2" path="/var/lib/kubelet/pods/8f7439f6-b83c-437d-9d4b-536775fee0d2/volumes"
Sep 17 22:00:29.070887 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:29.070796    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-775b7455c8-brwr9_845db971-2b15-4713-8f19-3e9e851b0d37/router/0.log"
Sep 17 22:00:29.083908 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:29.083882    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-f87z5_a53be88f-7ad0-4542-9c47-4ec5728fed7b/serve-healthcheck-canary/0.log"
Sep 17 22:00:29.473567 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:29.473522    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-db9vd" event={"ID":"6b79292b-9dd7-4560-8a09-80ee28ec9a4b","Type":"ContainerStarted","Data":"def1971edee52eff7b32362f1d575ec4ceee08ce169a123b8bcf520305c7110c"}
Sep 17 22:00:29.476117 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:29.476094    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-2kws6" event={"ID":"b90aafb6-01ac-4c30-a30a-530c91d1c944","Type":"ContainerStarted","Data":"ad6f66a1f8aad327ba0f8d3987cc1a6a0acda40f02d5217ca8d744e4a57d4b28"}
Sep 17 22:00:29.525785 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:29.525751    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a236d0c6-0efa-47e2-94c1-f4ca507442c4-metrics-certs\") pod \"network-metrics-daemon-n7n4d\" (UID: \"a236d0c6-0efa-47e2-94c1-f4ca507442c4\") " pod="openshift-multus/network-metrics-daemon-n7n4d"
Sep 17 22:00:29.528242 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:29.528218    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a236d0c6-0efa-47e2-94c1-f4ca507442c4-metrics-certs\") pod \"network-metrics-daemon-n7n4d\" (UID: \"a236d0c6-0efa-47e2-94c1-f4ca507442c4\") " pod="openshift-multus/network-metrics-daemon-n7n4d"
Sep 17 22:00:29.593365 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:29.593340    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9ks7q\""
Sep 17 22:00:29.601153 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:29.601133    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7n4d"
Sep 17 22:00:29.898627 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:29.898453    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n7n4d"]
Sep 17 22:00:29.901073 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:00:29.901047    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda236d0c6_0efa_47e2_94c1_f4ca507442c4.slice/crio-f4a356f957062a118931d69cf061005db2d7b8ec48f815c83ca80861c8c77daf WatchSource:0}: Error finding container f4a356f957062a118931d69cf061005db2d7b8ec48f815c83ca80861c8c77daf: Status 404 returned error can't find the container with id f4a356f957062a118931d69cf061005db2d7b8ec48f815c83ca80861c8c77daf
Sep 17 22:00:30.481873 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:30.481837    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n7n4d" event={"ID":"a236d0c6-0efa-47e2-94c1-f4ca507442c4","Type":"ContainerStarted","Data":"f4a356f957062a118931d69cf061005db2d7b8ec48f815c83ca80861c8c77daf"}
Sep 17 22:00:32.491700 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:32.491666    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n7n4d" event={"ID":"a236d0c6-0efa-47e2-94c1-f4ca507442c4","Type":"ContainerStarted","Data":"d4f06b117a7416b06988649c98a496a82443d444f27a957a811de75844dbc048"}
Sep 17 22:00:32.492061 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:32.491717    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n7n4d" event={"ID":"a236d0c6-0efa-47e2-94c1-f4ca507442c4","Type":"ContainerStarted","Data":"f82248d7e4e27dea9df8b937e114c129a74284f3dec58930985df84714db5886"}
Sep 17 22:00:32.508386 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:32.508324    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-n7n4d" podStartSLOduration=131.04167061 podStartE2EDuration="2m12.508310525s" totalImagesPullingTime="1.466639915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:00:32.507568127 +0000 UTC m=+133.953613409" watchObservedRunningTime="2026-09-17 22:00:32.508310525 +0000 UTC m=+133.954355797"
Sep 17 22:00:32.593541 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:32.593511    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:32.694663 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:32.694631    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:33.128097 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:33.127785    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:34.502689 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:34.502659    2555 generic.go:356] "Generic (PLEG): container finished" podID="c62743be-af33-4043-abb5-a7afcc097a8b" containerID="04c768b896d81a90de4091e47a446930115cf4bf69e6ba59d382a0bff43663ef" exitCode=0
Sep 17 22:00:34.503151 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:34.502739    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-pw29d" event={"ID":"c62743be-af33-4043-abb5-a7afcc097a8b","Type":"ContainerDied","Data":"04c768b896d81a90de4091e47a446930115cf4bf69e6ba59d382a0bff43663ef"}
Sep 17 22:00:34.503262 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:34.503151    2555 scope.go:117] "RemoveContainer" containerID="04c768b896d81a90de4091e47a446930115cf4bf69e6ba59d382a0bff43663ef"
Sep 17 22:00:35.508285 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:35.508250    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-pw29d" event={"ID":"c62743be-af33-4043-abb5-a7afcc097a8b","Type":"ContainerStarted","Data":"5b44532317baf2abd06015272939040c4b9c7e35ed20281d1daa3141e43de9da"}
Sep 17 22:00:42.601465 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:42.601429    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:42.701776 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:42.701743    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:42.757842 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:00:42.757808    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:01:19.625986 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:19.625894    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:01:20.008685 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:20.008658    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:01:24.696983 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:24.696942    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-b64656858-jwsq5"]
Sep 17 22:01:24.697371 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:24.697255    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f7439f6-b83c-437d-9d4b-536775fee0d2" containerName="registry"
Sep 17 22:01:24.697371 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:24.697268    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f7439f6-b83c-437d-9d4b-536775fee0d2" containerName="registry"
Sep 17 22:01:24.697371 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:24.697283    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b232899-c172-4e88-8bea-3c2835941d74" containerName="console"
Sep 17 22:01:24.697371 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:24.697289    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b232899-c172-4e88-8bea-3c2835941d74" containerName="console"
Sep 17 22:01:24.697371 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:24.697344    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b232899-c172-4e88-8bea-3c2835941d74" containerName="console"
Sep 17 22:01:24.697371 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:24.697355    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f7439f6-b83c-437d-9d4b-536775fee0d2" containerName="registry"
Sep 17 22:01:24.702447 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:24.702423    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-b64656858-jwsq5"
Sep 17 22:01:24.704685 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:24.704662    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 17 22:01:24.704871 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:24.704845    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 17 22:01:24.704996 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:24.704675    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 17 22:01:24.704996 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:24.704823    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 17 22:01:24.704996 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:24.704852    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-64hkg\""
Sep 17 22:01:24.704996 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:24.704668    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 17 22:01:24.710697 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:24.710675    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 17 22:01:24.715433 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:24.715412    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-b64656858-jwsq5"]
Sep 17 22:01:24.821157 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:24.821126    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86c7e467-08dd-4134-8557-3748db54ab9d-serving-certs-ca-bundle\") pod \"telemeter-client-b64656858-jwsq5\" (UID: \"86c7e467-08dd-4134-8557-3748db54ab9d\") " pod="openshift-monitoring/telemeter-client-b64656858-jwsq5"
Sep 17 22:01:24.821329 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:24.821167    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86c7e467-08dd-4134-8557-3748db54ab9d-metrics-client-ca\") pod \"telemeter-client-b64656858-jwsq5\" (UID: \"86c7e467-08dd-4134-8557-3748db54ab9d\") " pod="openshift-monitoring/telemeter-client-b64656858-jwsq5"
Sep 17 22:01:24.821329 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:24.821210    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/86c7e467-08dd-4134-8557-3748db54ab9d-federate-client-tls\") pod \"telemeter-client-b64656858-jwsq5\" (UID: \"86c7e467-08dd-4134-8557-3748db54ab9d\") " pod="openshift-monitoring/telemeter-client-b64656858-jwsq5"
Sep 17 22:01:24.821329 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:24.821262    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/86c7e467-08dd-4134-8557-3748db54ab9d-secret-telemeter-client\") pod \"telemeter-client-b64656858-jwsq5\" (UID: \"86c7e467-08dd-4134-8557-3748db54ab9d\") " pod="openshift-monitoring/telemeter-client-b64656858-jwsq5"
Sep 17 22:01:24.821329 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:24.821282    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p48f\" (UniqueName: \"kubernetes.io/projected/86c7e467-08dd-4134-8557-3748db54ab9d-kube-api-access-5p48f\") pod \"telemeter-client-b64656858-jwsq5\" (UID: \"86c7e467-08dd-4134-8557-3748db54ab9d\") " pod="openshift-monitoring/telemeter-client-b64656858-jwsq5"
Sep 17 22:01:24.821329 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:24.821318    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/86c7e467-08dd-4134-8557-3748db54ab9d-telemeter-client-tls\") pod \"telemeter-client-b64656858-jwsq5\" (UID: \"86c7e467-08dd-4134-8557-3748db54ab9d\") " pod="openshift-monitoring/telemeter-client-b64656858-jwsq5"
Sep 17 22:01:24.821515 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:24.821338    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/86c7e467-08dd-4134-8557-3748db54ab9d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b64656858-jwsq5\" (UID: \"86c7e467-08dd-4134-8557-3748db54ab9d\") " pod="openshift-monitoring/telemeter-client-b64656858-jwsq5"
Sep 17 22:01:24.821515 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:24.821362    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86c7e467-08dd-4134-8557-3748db54ab9d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b64656858-jwsq5\" (UID: \"86c7e467-08dd-4134-8557-3748db54ab9d\") " pod="openshift-monitoring/telemeter-client-b64656858-jwsq5"
Sep 17 22:01:24.922400 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:24.922374    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/86c7e467-08dd-4134-8557-3748db54ab9d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b64656858-jwsq5\" (UID: \"86c7e467-08dd-4134-8557-3748db54ab9d\") " pod="openshift-monitoring/telemeter-client-b64656858-jwsq5"
Sep 17 22:01:24.922492 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:24.922407    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86c7e467-08dd-4134-8557-3748db54ab9d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b64656858-jwsq5\" (UID: \"86c7e467-08dd-4134-8557-3748db54ab9d\") " pod="openshift-monitoring/telemeter-client-b64656858-jwsq5"
Sep 17 22:01:24.922492 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:24.922459    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86c7e467-08dd-4134-8557-3748db54ab9d-serving-certs-ca-bundle\") pod \"telemeter-client-b64656858-jwsq5\" (UID: \"86c7e467-08dd-4134-8557-3748db54ab9d\") " pod="openshift-monitoring/telemeter-client-b64656858-jwsq5"
Sep 17 22:01:24.922588 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:24.922490    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86c7e467-08dd-4134-8557-3748db54ab9d-metrics-client-ca\") pod \"telemeter-client-b64656858-jwsq5\" (UID: \"86c7e467-08dd-4134-8557-3748db54ab9d\") " pod="openshift-monitoring/telemeter-client-b64656858-jwsq5"
Sep 17 22:01:24.922588 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:24.922513    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/86c7e467-08dd-4134-8557-3748db54ab9d-federate-client-tls\") pod \"telemeter-client-b64656858-jwsq5\" (UID: \"86c7e467-08dd-4134-8557-3748db54ab9d\") " pod="openshift-monitoring/telemeter-client-b64656858-jwsq5"
Sep 17 22:01:24.922752 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:24.922725    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/86c7e467-08dd-4134-8557-3748db54ab9d-secret-telemeter-client\") pod \"telemeter-client-b64656858-jwsq5\" (UID: \"86c7e467-08dd-4134-8557-3748db54ab9d\") " pod="openshift-monitoring/telemeter-client-b64656858-jwsq5"
Sep 17 22:01:24.922802 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:24.922774    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p48f\" (UniqueName: \"kubernetes.io/projected/86c7e467-08dd-4134-8557-3748db54ab9d-kube-api-access-5p48f\") pod \"telemeter-client-b64656858-jwsq5\" (UID: \"86c7e467-08dd-4134-8557-3748db54ab9d\") " pod="openshift-monitoring/telemeter-client-b64656858-jwsq5"
Sep 17 22:01:24.922854 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:24.922805    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/86c7e467-08dd-4134-8557-3748db54ab9d-telemeter-client-tls\") pod \"telemeter-client-b64656858-jwsq5\" (UID: \"86c7e467-08dd-4134-8557-3748db54ab9d\") " pod="openshift-monitoring/telemeter-client-b64656858-jwsq5"
Sep 17 22:01:24.923452 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:24.923390    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86c7e467-08dd-4134-8557-3748db54ab9d-serving-certs-ca-bundle\") pod \"telemeter-client-b64656858-jwsq5\" (UID: \"86c7e467-08dd-4134-8557-3748db54ab9d\") " pod="openshift-monitoring/telemeter-client-b64656858-jwsq5"
Sep 17 22:01:24.923588 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:24.923466    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86c7e467-08dd-4134-8557-3748db54ab9d-metrics-client-ca\") pod \"telemeter-client-b64656858-jwsq5\" (UID: \"86c7e467-08dd-4134-8557-3748db54ab9d\") " pod="openshift-monitoring/telemeter-client-b64656858-jwsq5"
Sep 17 22:01:24.923661 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:24.923604    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86c7e467-08dd-4134-8557-3748db54ab9d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b64656858-jwsq5\" (UID: \"86c7e467-08dd-4134-8557-3748db54ab9d\") " pod="openshift-monitoring/telemeter-client-b64656858-jwsq5"
Sep 17 22:01:24.925121 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:24.925094    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/86c7e467-08dd-4134-8557-3748db54ab9d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b64656858-jwsq5\" (UID: \"86c7e467-08dd-4134-8557-3748db54ab9d\") " pod="openshift-monitoring/telemeter-client-b64656858-jwsq5"
Sep 17 22:01:24.925299 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:24.925283    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/86c7e467-08dd-4134-8557-3748db54ab9d-secret-telemeter-client\") pod \"telemeter-client-b64656858-jwsq5\" (UID: \"86c7e467-08dd-4134-8557-3748db54ab9d\") " pod="openshift-monitoring/telemeter-client-b64656858-jwsq5"
Sep 17 22:01:24.925611 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:24.925589    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/86c7e467-08dd-4134-8557-3748db54ab9d-federate-client-tls\") pod \"telemeter-client-b64656858-jwsq5\" (UID: \"86c7e467-08dd-4134-8557-3748db54ab9d\") " pod="openshift-monitoring/telemeter-client-b64656858-jwsq5"
Sep 17 22:01:24.925668 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:24.925592    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/86c7e467-08dd-4134-8557-3748db54ab9d-telemeter-client-tls\") pod \"telemeter-client-b64656858-jwsq5\" (UID: \"86c7e467-08dd-4134-8557-3748db54ab9d\") " pod="openshift-monitoring/telemeter-client-b64656858-jwsq5"
Sep 17 22:01:24.930411 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:24.930391    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p48f\" (UniqueName: \"kubernetes.io/projected/86c7e467-08dd-4134-8557-3748db54ab9d-kube-api-access-5p48f\") pod \"telemeter-client-b64656858-jwsq5\" (UID: \"86c7e467-08dd-4134-8557-3748db54ab9d\") " pod="openshift-monitoring/telemeter-client-b64656858-jwsq5"
Sep 17 22:01:25.013443 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:25.013395    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-b64656858-jwsq5"
Sep 17 22:01:25.289047 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:25.289020    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-b64656858-jwsq5"]
Sep 17 22:01:25.291597 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:01:25.291568    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86c7e467_08dd_4134_8557_3748db54ab9d.slice/crio-4432864f3a2a7e683eee9cb353986bb357e6f92f915b7e8ea78ea8018174090e WatchSource:0}: Error finding container 4432864f3a2a7e683eee9cb353986bb357e6f92f915b7e8ea78ea8018174090e: Status 404 returned error can't find the container with id 4432864f3a2a7e683eee9cb353986bb357e6f92f915b7e8ea78ea8018174090e
Sep 17 22:01:25.707672 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:25.707636    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b64656858-jwsq5" event={"ID":"86c7e467-08dd-4134-8557-3748db54ab9d","Type":"ContainerStarted","Data":"4432864f3a2a7e683eee9cb353986bb357e6f92f915b7e8ea78ea8018174090e"}
Sep 17 22:01:28.720840 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:28.720803    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b64656858-jwsq5" event={"ID":"86c7e467-08dd-4134-8557-3748db54ab9d","Type":"ContainerStarted","Data":"d4f8315e896a55215b4beb35c726a662e60f060eb4e50ae331ddbbfcbd250139"}
Sep 17 22:01:29.727098 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:29.727059    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b64656858-jwsq5" event={"ID":"86c7e467-08dd-4134-8557-3748db54ab9d","Type":"ContainerStarted","Data":"fb67b313b61d3f69eeda11a09e4c120217f49e32c567dbb9a33f668ae06e4000"}
Sep 17 22:01:29.727573 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:29.727117    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b64656858-jwsq5" event={"ID":"86c7e467-08dd-4134-8557-3748db54ab9d","Type":"ContainerStarted","Data":"a0c81cc1b31d9b3c501c1cf5f3ddb5c83a4997775c77ed8d46149b2239db943a"}
Sep 17 22:01:29.751265 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:29.751218    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-b64656858-jwsq5" podStartSLOduration=2.150073764 podStartE2EDuration="5.751204099s" totalImagesPullingTime="3.601130335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:01:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:01:29.74947321 +0000 UTC m=+191.195518476" watchObservedRunningTime="2026-09-17 22:01:29.751204099 +0000 UTC m=+191.197249372"
Sep 17 22:01:29.830919 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:29.830882    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f9998f667-c5bgg"]
Sep 17 22:01:29.834473 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:29.834443    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f9998f667-c5bgg"
Sep 17 22:01:29.838800 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:29.838732    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 22:01:29.841342 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:29.840549    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 22:01:29.841342 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:29.840680    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 22:01:29.841342 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:29.840826    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 22:01:29.841342 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:29.840962    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-qjgsp\""
Sep 17 22:01:29.841342 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:29.841347    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 22:01:29.844727 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:29.844708    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 22:01:29.848697 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:29.848677    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f9998f667-c5bgg"]
Sep 17 22:01:29.858000 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:29.857977    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3a58d38-c284-44d6-a772-428c137d7e8b-oauth-serving-cert\") pod \"console-7f9998f667-c5bgg\" (UID: \"c3a58d38-c284-44d6-a772-428c137d7e8b\") " pod="openshift-console/console-7f9998f667-c5bgg"
Sep 17 22:01:29.858100 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:29.858015    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3a58d38-c284-44d6-a772-428c137d7e8b-trusted-ca-bundle\") pod \"console-7f9998f667-c5bgg\" (UID: \"c3a58d38-c284-44d6-a772-428c137d7e8b\") " pod="openshift-console/console-7f9998f667-c5bgg"
Sep 17 22:01:29.858100 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:29.858032    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdckc\" (UniqueName: \"kubernetes.io/projected/c3a58d38-c284-44d6-a772-428c137d7e8b-kube-api-access-wdckc\") pod \"console-7f9998f667-c5bgg\" (UID: \"c3a58d38-c284-44d6-a772-428c137d7e8b\") " pod="openshift-console/console-7f9998f667-c5bgg"
Sep 17 22:01:29.858206 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:29.858127    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3a58d38-c284-44d6-a772-428c137d7e8b-console-config\") pod \"console-7f9998f667-c5bgg\" (UID: \"c3a58d38-c284-44d6-a772-428c137d7e8b\") " pod="openshift-console/console-7f9998f667-c5bgg"
Sep 17 22:01:29.858206 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:29.858166    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3a58d38-c284-44d6-a772-428c137d7e8b-console-oauth-config\") pod \"console-7f9998f667-c5bgg\" (UID: \"c3a58d38-c284-44d6-a772-428c137d7e8b\") " pod="openshift-console/console-7f9998f667-c5bgg"
Sep 17 22:01:29.858280 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:29.858215    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3a58d38-c284-44d6-a772-428c137d7e8b-console-serving-cert\") pod \"console-7f9998f667-c5bgg\" (UID: \"c3a58d38-c284-44d6-a772-428c137d7e8b\") " pod="openshift-console/console-7f9998f667-c5bgg"
Sep 17 22:01:29.858280 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:29.858248    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3a58d38-c284-44d6-a772-428c137d7e8b-service-ca\") pod \"console-7f9998f667-c5bgg\" (UID: \"c3a58d38-c284-44d6-a772-428c137d7e8b\") " pod="openshift-console/console-7f9998f667-c5bgg"
Sep 17 22:01:29.915211 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:29.915151    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:01:29.958723 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:29.958699    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3a58d38-c284-44d6-a772-428c137d7e8b-trusted-ca-bundle\") pod \"console-7f9998f667-c5bgg\" (UID: \"c3a58d38-c284-44d6-a772-428c137d7e8b\") " pod="openshift-console/console-7f9998f667-c5bgg"
Sep 17 22:01:29.958827 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:29.958727    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdckc\" (UniqueName: \"kubernetes.io/projected/c3a58d38-c284-44d6-a772-428c137d7e8b-kube-api-access-wdckc\") pod \"console-7f9998f667-c5bgg\" (UID: \"c3a58d38-c284-44d6-a772-428c137d7e8b\") " pod="openshift-console/console-7f9998f667-c5bgg"
Sep 17 22:01:29.958827 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:29.958774    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3a58d38-c284-44d6-a772-428c137d7e8b-console-config\") pod \"console-7f9998f667-c5bgg\" (UID: \"c3a58d38-c284-44d6-a772-428c137d7e8b\") " pod="openshift-console/console-7f9998f667-c5bgg"
Sep 17 22:01:29.958827 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:29.958814    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3a58d38-c284-44d6-a772-428c137d7e8b-console-oauth-config\") pod \"console-7f9998f667-c5bgg\" (UID: \"c3a58d38-c284-44d6-a772-428c137d7e8b\") " pod="openshift-console/console-7f9998f667-c5bgg"
Sep 17 22:01:29.958927 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:29.958844    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3a58d38-c284-44d6-a772-428c137d7e8b-console-serving-cert\") pod \"console-7f9998f667-c5bgg\" (UID: \"c3a58d38-c284-44d6-a772-428c137d7e8b\") " pod="openshift-console/console-7f9998f667-c5bgg"
Sep 17 22:01:29.958927 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:29.958869    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3a58d38-c284-44d6-a772-428c137d7e8b-service-ca\") pod \"console-7f9998f667-c5bgg\" (UID: \"c3a58d38-c284-44d6-a772-428c137d7e8b\") " pod="openshift-console/console-7f9998f667-c5bgg"
Sep 17 22:01:29.958927 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:29.958913    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3a58d38-c284-44d6-a772-428c137d7e8b-oauth-serving-cert\") pod \"console-7f9998f667-c5bgg\" (UID: \"c3a58d38-c284-44d6-a772-428c137d7e8b\") " pod="openshift-console/console-7f9998f667-c5bgg"
Sep 17 22:01:29.959557 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:29.959525    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3a58d38-c284-44d6-a772-428c137d7e8b-console-config\") pod \"console-7f9998f667-c5bgg\" (UID: \"c3a58d38-c284-44d6-a772-428c137d7e8b\") " pod="openshift-console/console-7f9998f667-c5bgg"
Sep 17 22:01:29.959557 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:29.959542    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3a58d38-c284-44d6-a772-428c137d7e8b-oauth-serving-cert\") pod \"console-7f9998f667-c5bgg\" (UID: \"c3a58d38-c284-44d6-a772-428c137d7e8b\") " pod="openshift-console/console-7f9998f667-c5bgg"
Sep 17 22:01:29.959733 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:29.959712    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3a58d38-c284-44d6-a772-428c137d7e8b-service-ca\") pod \"console-7f9998f667-c5bgg\" (UID: \"c3a58d38-c284-44d6-a772-428c137d7e8b\") " pod="openshift-console/console-7f9998f667-c5bgg"
Sep 17 22:01:29.959916 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:29.959893    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3a58d38-c284-44d6-a772-428c137d7e8b-trusted-ca-bundle\") pod \"console-7f9998f667-c5bgg\" (UID: \"c3a58d38-c284-44d6-a772-428c137d7e8b\") " pod="openshift-console/console-7f9998f667-c5bgg"
Sep 17 22:01:29.961522 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:29.961500    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3a58d38-c284-44d6-a772-428c137d7e8b-console-serving-cert\") pod \"console-7f9998f667-c5bgg\" (UID: \"c3a58d38-c284-44d6-a772-428c137d7e8b\") " pod="openshift-console/console-7f9998f667-c5bgg"
Sep 17 22:01:29.961608 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:29.961506    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3a58d38-c284-44d6-a772-428c137d7e8b-console-oauth-config\") pod \"console-7f9998f667-c5bgg\" (UID: \"c3a58d38-c284-44d6-a772-428c137d7e8b\") " pod="openshift-console/console-7f9998f667-c5bgg"
Sep 17 22:01:29.966471 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:29.966451    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdckc\" (UniqueName: \"kubernetes.io/projected/c3a58d38-c284-44d6-a772-428c137d7e8b-kube-api-access-wdckc\") pod \"console-7f9998f667-c5bgg\" (UID: \"c3a58d38-c284-44d6-a772-428c137d7e8b\") " pod="openshift-console/console-7f9998f667-c5bgg"
Sep 17 22:01:29.978342 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:29.978285    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:01:30.046850 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:30.046819    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:01:30.149674 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:30.149648    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f9998f667-c5bgg"
Sep 17 22:01:30.421225 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:30.421203    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f9998f667-c5bgg"]
Sep 17 22:01:30.423145 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:01:30.423118    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3a58d38_c284_44d6_a772_428c137d7e8b.slice/crio-ad3ac857876afa3456bf98d29e77363406c138b325d879c6da863a8794bd6cf0 WatchSource:0}: Error finding container ad3ac857876afa3456bf98d29e77363406c138b325d879c6da863a8794bd6cf0: Status 404 returned error can't find the container with id ad3ac857876afa3456bf98d29e77363406c138b325d879c6da863a8794bd6cf0
Sep 17 22:01:30.680175 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:30.680070    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:01:30.732510 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:30.732470    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f9998f667-c5bgg" event={"ID":"c3a58d38-c284-44d6-a772-428c137d7e8b","Type":"ContainerStarted","Data":"c1155aa161fe6c2dda2ebbc1e81e260c73072a205108fccfecb6748dfd6cd5f6"}
Sep 17 22:01:30.732883 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:30.732525    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f9998f667-c5bgg" event={"ID":"c3a58d38-c284-44d6-a772-428c137d7e8b","Type":"ContainerStarted","Data":"ad3ac857876afa3456bf98d29e77363406c138b325d879c6da863a8794bd6cf0"}
Sep 17 22:01:30.748897 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:30.748853    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7f9998f667-c5bgg" podStartSLOduration=1.74883776 podStartE2EDuration="1.74883776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:01:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:01:30.74754713 +0000 UTC m=+192.193592402" watchObservedRunningTime="2026-09-17 22:01:30.74883776 +0000 UTC m=+192.194883032"
Sep 17 22:01:30.793785 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:30.793753    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:01:34.840623 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:34.840580    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:01:34.912247 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:34.912205    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:01:40.150150 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:40.150108    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7f9998f667-c5bgg"
Sep 17 22:01:40.150566 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:40.150242    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f9998f667-c5bgg"
Sep 17 22:01:40.154883 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:40.154858    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7f9998f667-c5bgg"
Sep 17 22:01:40.774858 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:40.774833    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7f9998f667-c5bgg"
Sep 17 22:01:40.932607 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:40.932576    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-79df74fcd5-r7ncf"]
Sep 17 22:01:40.936252 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:40.936226    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79df74fcd5-r7ncf"
Sep 17 22:01:40.946651 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:40.946628    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79df74fcd5-r7ncf"]
Sep 17 22:01:40.950255 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:40.950171    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:01:41.026436 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:41.026369    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:01:41.036697 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:41.036672    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4197ebae-9c48-4dde-b941-0406df61ce88-trusted-ca-bundle\") pod \"console-79df74fcd5-r7ncf\" (UID: \"4197ebae-9c48-4dde-b941-0406df61ce88\") " pod="openshift-console/console-79df74fcd5-r7ncf"
Sep 17 22:01:41.036834 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:41.036705    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4197ebae-9c48-4dde-b941-0406df61ce88-service-ca\") pod \"console-79df74fcd5-r7ncf\" (UID: \"4197ebae-9c48-4dde-b941-0406df61ce88\") " pod="openshift-console/console-79df74fcd5-r7ncf"
Sep 17 22:01:41.036834 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:41.036738    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4197ebae-9c48-4dde-b941-0406df61ce88-oauth-serving-cert\") pod \"console-79df74fcd5-r7ncf\" (UID: \"4197ebae-9c48-4dde-b941-0406df61ce88\") " pod="openshift-console/console-79df74fcd5-r7ncf"
Sep 17 22:01:41.036834 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:41.036775    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4197ebae-9c48-4dde-b941-0406df61ce88-console-oauth-config\") pod \"console-79df74fcd5-r7ncf\" (UID: \"4197ebae-9c48-4dde-b941-0406df61ce88\") " pod="openshift-console/console-79df74fcd5-r7ncf"
Sep 17 22:01:41.036834 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:41.036813    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4197ebae-9c48-4dde-b941-0406df61ce88-console-config\") pod \"console-79df74fcd5-r7ncf\" (UID: \"4197ebae-9c48-4dde-b941-0406df61ce88\") " pod="openshift-console/console-79df74fcd5-r7ncf"
Sep 17 22:01:41.036971 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:41.036835    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m2kw\" (UniqueName: \"kubernetes.io/projected/4197ebae-9c48-4dde-b941-0406df61ce88-kube-api-access-2m2kw\") pod \"console-79df74fcd5-r7ncf\" (UID: \"4197ebae-9c48-4dde-b941-0406df61ce88\") " pod="openshift-console/console-79df74fcd5-r7ncf"
Sep 17 22:01:41.036971 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:41.036862    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4197ebae-9c48-4dde-b941-0406df61ce88-console-serving-cert\") pod \"console-79df74fcd5-r7ncf\" (UID: \"4197ebae-9c48-4dde-b941-0406df61ce88\") " pod="openshift-console/console-79df74fcd5-r7ncf"
Sep 17 22:01:41.137651 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:41.137621    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4197ebae-9c48-4dde-b941-0406df61ce88-oauth-serving-cert\") pod \"console-79df74fcd5-r7ncf\" (UID: \"4197ebae-9c48-4dde-b941-0406df61ce88\") " pod="openshift-console/console-79df74fcd5-r7ncf"
Sep 17 22:01:41.137783 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:41.137660    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4197ebae-9c48-4dde-b941-0406df61ce88-console-oauth-config\") pod \"console-79df74fcd5-r7ncf\" (UID: \"4197ebae-9c48-4dde-b941-0406df61ce88\") " pod="openshift-console/console-79df74fcd5-r7ncf"
Sep 17 22:01:41.137783 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:41.137683    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4197ebae-9c48-4dde-b941-0406df61ce88-console-config\") pod \"console-79df74fcd5-r7ncf\" (UID: \"4197ebae-9c48-4dde-b941-0406df61ce88\") " pod="openshift-console/console-79df74fcd5-r7ncf"
Sep 17 22:01:41.137783 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:41.137710    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m2kw\" (UniqueName: \"kubernetes.io/projected/4197ebae-9c48-4dde-b941-0406df61ce88-kube-api-access-2m2kw\") pod \"console-79df74fcd5-r7ncf\" (UID: \"4197ebae-9c48-4dde-b941-0406df61ce88\") " pod="openshift-console/console-79df74fcd5-r7ncf"
Sep 17 22:01:41.137783 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:41.137739    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4197ebae-9c48-4dde-b941-0406df61ce88-console-serving-cert\") pod \"console-79df74fcd5-r7ncf\" (UID: \"4197ebae-9c48-4dde-b941-0406df61ce88\") " pod="openshift-console/console-79df74fcd5-r7ncf"
Sep 17 22:01:41.137989 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:41.137810    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4197ebae-9c48-4dde-b941-0406df61ce88-trusted-ca-bundle\") pod \"console-79df74fcd5-r7ncf\" (UID: \"4197ebae-9c48-4dde-b941-0406df61ce88\") " pod="openshift-console/console-79df74fcd5-r7ncf"
Sep 17 22:01:41.137989 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:41.137839    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4197ebae-9c48-4dde-b941-0406df61ce88-service-ca\") pod \"console-79df74fcd5-r7ncf\" (UID: \"4197ebae-9c48-4dde-b941-0406df61ce88\") " pod="openshift-console/console-79df74fcd5-r7ncf"
Sep 17 22:01:41.138518 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:41.138496    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4197ebae-9c48-4dde-b941-0406df61ce88-service-ca\") pod \"console-79df74fcd5-r7ncf\" (UID: \"4197ebae-9c48-4dde-b941-0406df61ce88\") " pod="openshift-console/console-79df74fcd5-r7ncf"
Sep 17 22:01:41.138606 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:41.138524    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4197ebae-9c48-4dde-b941-0406df61ce88-oauth-serving-cert\") pod \"console-79df74fcd5-r7ncf\" (UID: \"4197ebae-9c48-4dde-b941-0406df61ce88\") " pod="openshift-console/console-79df74fcd5-r7ncf"
Sep 17 22:01:41.138606 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:41.138543    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4197ebae-9c48-4dde-b941-0406df61ce88-console-config\") pod \"console-79df74fcd5-r7ncf\" (UID: \"4197ebae-9c48-4dde-b941-0406df61ce88\") " pod="openshift-console/console-79df74fcd5-r7ncf"
Sep 17 22:01:41.138769 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:41.138752    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4197ebae-9c48-4dde-b941-0406df61ce88-trusted-ca-bundle\") pod \"console-79df74fcd5-r7ncf\" (UID: \"4197ebae-9c48-4dde-b941-0406df61ce88\") " pod="openshift-console/console-79df74fcd5-r7ncf"
Sep 17 22:01:41.140143 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:41.140122    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4197ebae-9c48-4dde-b941-0406df61ce88-console-serving-cert\") pod \"console-79df74fcd5-r7ncf\" (UID: \"4197ebae-9c48-4dde-b941-0406df61ce88\") " pod="openshift-console/console-79df74fcd5-r7ncf"
Sep 17 22:01:41.140270 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:41.140161    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4197ebae-9c48-4dde-b941-0406df61ce88-console-oauth-config\") pod \"console-79df74fcd5-r7ncf\" (UID: \"4197ebae-9c48-4dde-b941-0406df61ce88\") " pod="openshift-console/console-79df74fcd5-r7ncf"
Sep 17 22:01:41.145464 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:41.145441    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m2kw\" (UniqueName: \"kubernetes.io/projected/4197ebae-9c48-4dde-b941-0406df61ce88-kube-api-access-2m2kw\") pod \"console-79df74fcd5-r7ncf\" (UID: \"4197ebae-9c48-4dde-b941-0406df61ce88\") " pod="openshift-console/console-79df74fcd5-r7ncf"
Sep 17 22:01:41.247599 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:41.247570    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79df74fcd5-r7ncf"
Sep 17 22:01:41.470024 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:41.469990    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:01:41.522134 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:41.522106    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79df74fcd5-r7ncf"]
Sep 17 22:01:41.524841 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:01:41.524817    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4197ebae_9c48_4dde_b941_0406df61ce88.slice/crio-f2e61d89ac359103a696ec6cff1a2a6d569b9b414cf864ec55c41ad6319cdf45 WatchSource:0}: Error finding container f2e61d89ac359103a696ec6cff1a2a6d569b9b414cf864ec55c41ad6319cdf45: Status 404 returned error can't find the container with id f2e61d89ac359103a696ec6cff1a2a6d569b9b414cf864ec55c41ad6319cdf45
Sep 17 22:01:41.527601 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:41.527482    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:01:41.776557 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:41.776519    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79df74fcd5-r7ncf" event={"ID":"4197ebae-9c48-4dde-b941-0406df61ce88","Type":"ContainerStarted","Data":"a7060025825ee26c670af6fde0f84d1077c60dd1d3f435ad89cc8133bc93c8de"}
Sep 17 22:01:41.776716 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:41.776571    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79df74fcd5-r7ncf" event={"ID":"4197ebae-9c48-4dde-b941-0406df61ce88","Type":"ContainerStarted","Data":"f2e61d89ac359103a696ec6cff1a2a6d569b9b414cf864ec55c41ad6319cdf45"}
Sep 17 22:01:41.794871 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:41.794827    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-79df74fcd5-r7ncf" podStartSLOduration=1.7948124079999999 podStartE2EDuration="1.794812408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:01:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:01:41.793378325 +0000 UTC m=+203.239423597" watchObservedRunningTime="2026-09-17 22:01:41.794812408 +0000 UTC m=+203.240857680"
Sep 17 22:01:46.692811 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:46.692763    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:01:46.767341 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:46.767306    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:01:51.019832 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:51.019799    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:01:51.120043 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:51.120012    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:01:51.248073 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:51.248045    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-79df74fcd5-r7ncf"
Sep 17 22:01:51.248222 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:51.248102    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-79df74fcd5-r7ncf"
Sep 17 22:01:51.252884 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:51.252863    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-79df74fcd5-r7ncf"
Sep 17 22:01:51.817471 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:51.817444    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-79df74fcd5-r7ncf"
Sep 17 22:01:51.862717 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:51.862686    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f9998f667-c5bgg"]
Sep 17 22:01:51.963294 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:51.963257    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:01:52.306085 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:52.306049    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:01:52.365673 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:52.365641    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:01:59.179600 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:59.179569    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:01:59.230526 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:01:59.230490    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:02:01.508434 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:02:01.508400    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:02:16.886145 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:02:16.886098    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7f9998f667-c5bgg" podUID="c3a58d38-c284-44d6-a772-428c137d7e8b" containerName="console" containerID="cri-o://c1155aa161fe6c2dda2ebbc1e81e260c73072a205108fccfecb6748dfd6cd5f6" gracePeriod=15
Sep 17 22:02:17.119661 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:02:17.119639    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f9998f667-c5bgg"
Sep 17 22:02:17.183977 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:02:17.183904    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:02:17.210724 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:02:17.210703    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3a58d38-c284-44d6-a772-428c137d7e8b-service-ca\") pod \"c3a58d38-c284-44d6-a772-428c137d7e8b\" (UID: \"c3a58d38-c284-44d6-a772-428c137d7e8b\") "
Sep 17 22:02:17.210820 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:02:17.210747    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3a58d38-c284-44d6-a772-428c137d7e8b-trusted-ca-bundle\") pod \"c3a58d38-c284-44d6-a772-428c137d7e8b\" (UID: \"c3a58d38-c284-44d6-a772-428c137d7e8b\") "
Sep 17 22:02:17.210820 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:02:17.210773    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdckc\" (UniqueName: \"kubernetes.io/projected/c3a58d38-c284-44d6-a772-428c137d7e8b-kube-api-access-wdckc\") pod \"c3a58d38-c284-44d6-a772-428c137d7e8b\" (UID: \"c3a58d38-c284-44d6-a772-428c137d7e8b\") "
Sep 17 22:02:17.210923 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:02:17.210905    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3a58d38-c284-44d6-a772-428c137d7e8b-console-serving-cert\") pod \"c3a58d38-c284-44d6-a772-428c137d7e8b\" (UID: \"c3a58d38-c284-44d6-a772-428c137d7e8b\") "
Sep 17 22:02:17.210989 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:02:17.210976    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3a58d38-c284-44d6-a772-428c137d7e8b-console-oauth-config\") pod \"c3a58d38-c284-44d6-a772-428c137d7e8b\" (UID: \"c3a58d38-c284-44d6-a772-428c137d7e8b\") "
Sep 17 22:02:17.211048 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:02:17.211012    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3a58d38-c284-44d6-a772-428c137d7e8b-console-config\") pod \"c3a58d38-c284-44d6-a772-428c137d7e8b\" (UID: \"c3a58d38-c284-44d6-a772-428c137d7e8b\") "
Sep 17 22:02:17.211099 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:02:17.211074    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3a58d38-c284-44d6-a772-428c137d7e8b-oauth-serving-cert\") pod \"c3a58d38-c284-44d6-a772-428c137d7e8b\" (UID: \"c3a58d38-c284-44d6-a772-428c137d7e8b\") "
Sep 17 22:02:17.211150 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:02:17.211096    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3a58d38-c284-44d6-a772-428c137d7e8b-service-ca" (OuterVolumeSpecName: "service-ca") pod "c3a58d38-c284-44d6-a772-428c137d7e8b" (UID: "c3a58d38-c284-44d6-a772-428c137d7e8b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:02:17.211273 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:02:17.211147    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3a58d38-c284-44d6-a772-428c137d7e8b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c3a58d38-c284-44d6-a772-428c137d7e8b" (UID: "c3a58d38-c284-44d6-a772-428c137d7e8b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:02:17.211460 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:02:17.211441    2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3a58d38-c284-44d6-a772-428c137d7e8b-service-ca\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:02:17.211460 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:02:17.211459    2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3a58d38-c284-44d6-a772-428c137d7e8b-trusted-ca-bundle\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:02:17.211639 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:02:17.211504    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3a58d38-c284-44d6-a772-428c137d7e8b-console-config" (OuterVolumeSpecName: "console-config") pod "c3a58d38-c284-44d6-a772-428c137d7e8b" (UID: "c3a58d38-c284-44d6-a772-428c137d7e8b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:02:17.211639 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:02:17.211529    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3a58d38-c284-44d6-a772-428c137d7e8b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c3a58d38-c284-44d6-a772-428c137d7e8b" (UID: "c3a58d38-c284-44d6-a772-428c137d7e8b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:02:17.213046 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:02:17.213017    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3a58d38-c284-44d6-a772-428c137d7e8b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c3a58d38-c284-44d6-a772-428c137d7e8b" (UID: "c3a58d38-c284-44d6-a772-428c137d7e8b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:02:17.213121 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:02:17.213044    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3a58d38-c284-44d6-a772-428c137d7e8b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c3a58d38-c284-44d6-a772-428c137d7e8b" (UID: "c3a58d38-c284-44d6-a772-428c137d7e8b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:02:17.213121 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:02:17.213102    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3a58d38-c284-44d6-a772-428c137d7e8b-kube-api-access-wdckc" (OuterVolumeSpecName: "kube-api-access-wdckc") pod "c3a58d38-c284-44d6-a772-428c137d7e8b" (UID: "c3a58d38-c284-44d6-a772-428c137d7e8b"). InnerVolumeSpecName "kube-api-access-wdckc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:02:17.312140 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:02:17.312117    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdckc\" (UniqueName: \"kubernetes.io/projected/c3a58d38-c284-44d6-a772-428c137d7e8b-kube-api-access-wdckc\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:02:17.312140 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:02:17.312137    2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3a58d38-c284-44d6-a772-428c137d7e8b-console-serving-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:02:17.312292 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:02:17.312147    2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3a58d38-c284-44d6-a772-428c137d7e8b-console-oauth-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:02:17.312292 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:02:17.312156    2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3a58d38-c284-44d6-a772-428c137d7e8b-console-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:02:17.312292 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:02:17.312164    2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3a58d38-c284-44d6-a772-428c137d7e8b-oauth-serving-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:02:17.916890 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:02:17.916859    2555 generic.go:356] "Generic (PLEG): container finished" podID="c3a58d38-c284-44d6-a772-428c137d7e8b" containerID="c1155aa161fe6c2dda2ebbc1e81e260c73072a205108fccfecb6748dfd6cd5f6" exitCode=0
Sep 17 22:02:17.917381 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:02:17.916903    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f9998f667-c5bgg"
Sep 17 22:02:17.917381 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:02:17.916911    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f9998f667-c5bgg" event={"ID":"c3a58d38-c284-44d6-a772-428c137d7e8b","Type":"ContainerDied","Data":"c1155aa161fe6c2dda2ebbc1e81e260c73072a205108fccfecb6748dfd6cd5f6"}
Sep 17 22:02:17.917381 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:02:17.916950    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f9998f667-c5bgg" event={"ID":"c3a58d38-c284-44d6-a772-428c137d7e8b","Type":"ContainerDied","Data":"ad3ac857876afa3456bf98d29e77363406c138b325d879c6da863a8794bd6cf0"}
Sep 17 22:02:17.917381 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:02:17.916968    2555 scope.go:117] "RemoveContainer" containerID="c1155aa161fe6c2dda2ebbc1e81e260c73072a205108fccfecb6748dfd6cd5f6"
Sep 17 22:02:17.925332 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:02:17.925315    2555 scope.go:117] "RemoveContainer" containerID="c1155aa161fe6c2dda2ebbc1e81e260c73072a205108fccfecb6748dfd6cd5f6"
Sep 17 22:02:17.925598 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:02:17.925581    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1155aa161fe6c2dda2ebbc1e81e260c73072a205108fccfecb6748dfd6cd5f6\": container with ID starting with c1155aa161fe6c2dda2ebbc1e81e260c73072a205108fccfecb6748dfd6cd5f6 not found: ID does not exist" containerID="c1155aa161fe6c2dda2ebbc1e81e260c73072a205108fccfecb6748dfd6cd5f6"
Sep 17 22:02:17.925642 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:02:17.925605    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1155aa161fe6c2dda2ebbc1e81e260c73072a205108fccfecb6748dfd6cd5f6"} err="failed to get container status \"c1155aa161fe6c2dda2ebbc1e81e260c73072a205108fccfecb6748dfd6cd5f6\": rpc error: code = NotFound desc = could not find container \"c1155aa161fe6c2dda2ebbc1e81e260c73072a205108fccfecb6748dfd6cd5f6\": container with ID starting with c1155aa161fe6c2dda2ebbc1e81e260c73072a205108fccfecb6748dfd6cd5f6 not found: ID does not exist"
Sep 17 22:02:17.937305 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:02:17.937271    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f9998f667-c5bgg"]
Sep 17 22:02:17.943161 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:02:17.943140    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7f9998f667-c5bgg"]
Sep 17 22:02:18.677949 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:02:18.677916    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3a58d38-c284-44d6-a772-428c137d7e8b" path="/var/lib/kubelet/pods/c3a58d38-c284-44d6-a772-428c137d7e8b/volumes"
Sep 17 22:02:32.124692 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:02:32.124655    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:03:06.660451 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:03:06.660413    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:03:06.731442 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:03:06.731402    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:03:20.561511 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:03:20.561481    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-7njvj_a324c9ab-5d42-4d31-8ff7-449f619be774/console-operator/1.log"
Sep 17 22:03:20.565806 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:03:20.565781    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s7f5_4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf/ovn-acl-logging/0.log"
Sep 17 22:03:20.566069 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:03:20.566054    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-7njvj_a324c9ab-5d42-4d31-8ff7-449f619be774/console-operator/1.log"
Sep 17 22:03:20.569935 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:03:20.569915    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s7f5_4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf/ovn-acl-logging/0.log"
Sep 17 22:03:20.572529 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:03:20.572513    2555 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 22:04:38.711894 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:04:38.711861    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:04:39.860289 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:04:39.860256    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avcspc"]
Sep 17 22:04:39.860684 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:04:39.860555    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3a58d38-c284-44d6-a772-428c137d7e8b" containerName="console"
Sep 17 22:04:39.860684 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:04:39.860565    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3a58d38-c284-44d6-a772-428c137d7e8b" containerName="console"
Sep 17 22:04:39.860684 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:04:39.860625    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3a58d38-c284-44d6-a772-428c137d7e8b" containerName="console"
Sep 17 22:04:39.863645 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:04:39.863629    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avcspc"
Sep 17 22:04:39.867113 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:04:39.867082    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 22:04:39.867274 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:04:39.867141    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-h4r5v\""
Sep 17 22:04:39.870128 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:04:39.870111    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 22:04:39.882651 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:04:39.882626    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avcspc"]
Sep 17 22:04:40.045620 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:04:40.045593    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e4ddf34-8bfe-4a02-9bbf-4a0843742f02-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avcspc\" (UID: \"8e4ddf34-8bfe-4a02-9bbf-4a0843742f02\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avcspc"
Sep 17 22:04:40.045780 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:04:40.045633    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7snqv\" (UniqueName: \"kubernetes.io/projected/8e4ddf34-8bfe-4a02-9bbf-4a0843742f02-kube-api-access-7snqv\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avcspc\" (UID: \"8e4ddf34-8bfe-4a02-9bbf-4a0843742f02\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avcspc"
Sep 17 22:04:40.045780 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:04:40.045660    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e4ddf34-8bfe-4a02-9bbf-4a0843742f02-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avcspc\" (UID: \"8e4ddf34-8bfe-4a02-9bbf-4a0843742f02\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avcspc"
Sep 17 22:04:40.146927 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:04:40.146891    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e4ddf34-8bfe-4a02-9bbf-4a0843742f02-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avcspc\" (UID: \"8e4ddf34-8bfe-4a02-9bbf-4a0843742f02\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avcspc"
Sep 17 22:04:40.147079 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:04:40.146970    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e4ddf34-8bfe-4a02-9bbf-4a0843742f02-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avcspc\" (UID: \"8e4ddf34-8bfe-4a02-9bbf-4a0843742f02\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avcspc"
Sep 17 22:04:40.147079 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:04:40.147021    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7snqv\" (UniqueName: \"kubernetes.io/projected/8e4ddf34-8bfe-4a02-9bbf-4a0843742f02-kube-api-access-7snqv\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avcspc\" (UID: \"8e4ddf34-8bfe-4a02-9bbf-4a0843742f02\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avcspc"
Sep 17 22:04:40.147298 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:04:40.147277    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e4ddf34-8bfe-4a02-9bbf-4a0843742f02-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avcspc\" (UID: \"8e4ddf34-8bfe-4a02-9bbf-4a0843742f02\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avcspc"
Sep 17 22:04:40.147396 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:04:40.147342    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e4ddf34-8bfe-4a02-9bbf-4a0843742f02-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avcspc\" (UID: \"8e4ddf34-8bfe-4a02-9bbf-4a0843742f02\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avcspc"
Sep 17 22:04:40.155428 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:04:40.155404    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7snqv\" (UniqueName: \"kubernetes.io/projected/8e4ddf34-8bfe-4a02-9bbf-4a0843742f02-kube-api-access-7snqv\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avcspc\" (UID: \"8e4ddf34-8bfe-4a02-9bbf-4a0843742f02\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avcspc"
Sep 17 22:04:40.172194 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:04:40.172162    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avcspc"
Sep 17 22:04:40.379371 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:04:40.379338    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:04:40.442545 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:04:40.442469    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avcspc"]
Sep 17 22:04:40.445266 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:04:40.445240    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e4ddf34_8bfe_4a02_9bbf_4a0843742f02.slice/crio-4f43fa6a6d814cbcd62ed86d8587a2be5321d691fd29fe5241e43efe56993a00 WatchSource:0}: Error finding container 4f43fa6a6d814cbcd62ed86d8587a2be5321d691fd29fe5241e43efe56993a00: Status 404 returned error can't find the container with id 4f43fa6a6d814cbcd62ed86d8587a2be5321d691fd29fe5241e43efe56993a00
Sep 17 22:04:40.447227 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:04:40.447172    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 22:04:40.469748 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:04:40.469722    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avcspc" event={"ID":"8e4ddf34-8bfe-4a02-9bbf-4a0843742f02","Type":"ContainerStarted","Data":"4f43fa6a6d814cbcd62ed86d8587a2be5321d691fd29fe5241e43efe56993a00"}
Sep 17 22:04:46.500747 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:04:46.500717    2555 generic.go:356] "Generic (PLEG): container finished" podID="8e4ddf34-8bfe-4a02-9bbf-4a0843742f02" containerID="c6c826dae9b4b58c509ccc03decafb640ec0b265b5b34b64189e977d033424e3" exitCode=0
Sep 17 22:04:46.501067 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:04:46.500785    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avcspc" event={"ID":"8e4ddf34-8bfe-4a02-9bbf-4a0843742f02","Type":"ContainerDied","Data":"c6c826dae9b4b58c509ccc03decafb640ec0b265b5b34b64189e977d033424e3"}
Sep 17 22:04:49.513628 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:04:49.513594    2555 generic.go:356] "Generic (PLEG): container finished" podID="8e4ddf34-8bfe-4a02-9bbf-4a0843742f02" containerID="98adfaf3fd0753d4c147eabdeac2a774afac4c43e19389c84558fe3e3e50408d" exitCode=0
Sep 17 22:04:49.513916 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:04:49.513656    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avcspc" event={"ID":"8e4ddf34-8bfe-4a02-9bbf-4a0843742f02","Type":"ContainerDied","Data":"98adfaf3fd0753d4c147eabdeac2a774afac4c43e19389c84558fe3e3e50408d"}
Sep 17 22:04:56.547054 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:04:56.547020    2555 generic.go:356] "Generic (PLEG): container finished" podID="8e4ddf34-8bfe-4a02-9bbf-4a0843742f02" containerID="e5adcfef1208cdc4b489a61166f91598662a1ed77594044c6e9030c8cac61f0c" exitCode=0
Sep 17 22:04:56.547444 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:04:56.547065    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avcspc" event={"ID":"8e4ddf34-8bfe-4a02-9bbf-4a0843742f02","Type":"ContainerDied","Data":"e5adcfef1208cdc4b489a61166f91598662a1ed77594044c6e9030c8cac61f0c"}
Sep 17 22:04:57.674583 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:04:57.674562    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avcspc"
Sep 17 22:04:57.791063 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:04:57.791033    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e4ddf34-8bfe-4a02-9bbf-4a0843742f02-util\") pod \"8e4ddf34-8bfe-4a02-9bbf-4a0843742f02\" (UID: \"8e4ddf34-8bfe-4a02-9bbf-4a0843742f02\") "
Sep 17 22:04:57.791231 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:04:57.791077    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e4ddf34-8bfe-4a02-9bbf-4a0843742f02-bundle\") pod \"8e4ddf34-8bfe-4a02-9bbf-4a0843742f02\" (UID: \"8e4ddf34-8bfe-4a02-9bbf-4a0843742f02\") "
Sep 17 22:04:57.791231 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:04:57.791150    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7snqv\" (UniqueName: \"kubernetes.io/projected/8e4ddf34-8bfe-4a02-9bbf-4a0843742f02-kube-api-access-7snqv\") pod \"8e4ddf34-8bfe-4a02-9bbf-4a0843742f02\" (UID: \"8e4ddf34-8bfe-4a02-9bbf-4a0843742f02\") "
Sep 17 22:04:57.791832 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:04:57.791803    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e4ddf34-8bfe-4a02-9bbf-4a0843742f02-bundle" (OuterVolumeSpecName: "bundle") pod "8e4ddf34-8bfe-4a02-9bbf-4a0843742f02" (UID: "8e4ddf34-8bfe-4a02-9bbf-4a0843742f02"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:04:57.793298 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:04:57.793276    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e4ddf34-8bfe-4a02-9bbf-4a0843742f02-kube-api-access-7snqv" (OuterVolumeSpecName: "kube-api-access-7snqv") pod "8e4ddf34-8bfe-4a02-9bbf-4a0843742f02" (UID: "8e4ddf34-8bfe-4a02-9bbf-4a0843742f02"). InnerVolumeSpecName "kube-api-access-7snqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:04:57.797072 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:04:57.797048    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e4ddf34-8bfe-4a02-9bbf-4a0843742f02-util" (OuterVolumeSpecName: "util") pod "8e4ddf34-8bfe-4a02-9bbf-4a0843742f02" (UID: "8e4ddf34-8bfe-4a02-9bbf-4a0843742f02"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:04:57.892472 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:04:57.892445    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7snqv\" (UniqueName: \"kubernetes.io/projected/8e4ddf34-8bfe-4a02-9bbf-4a0843742f02-kube-api-access-7snqv\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:04:57.892472 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:04:57.892469    2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e4ddf34-8bfe-4a02-9bbf-4a0843742f02-util\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:04:57.892625 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:04:57.892479    2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e4ddf34-8bfe-4a02-9bbf-4a0843742f02-bundle\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:04:58.556484 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:04:58.556451    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avcspc" event={"ID":"8e4ddf34-8bfe-4a02-9bbf-4a0843742f02","Type":"ContainerDied","Data":"4f43fa6a6d814cbcd62ed86d8587a2be5321d691fd29fe5241e43efe56993a00"}
Sep 17 22:04:58.556484 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:04:58.556480    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avcspc"
Sep 17 22:04:58.556693 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:04:58.556520    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f43fa6a6d814cbcd62ed86d8587a2be5321d691fd29fe5241e43efe56993a00"
Sep 17 22:05:01.139392 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:01.139360    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:05:01.234537 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:01.234507    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:05:02.247988 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:02.247956    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fgx5l"]
Sep 17 22:05:02.248385 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:02.248268    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e4ddf34-8bfe-4a02-9bbf-4a0843742f02" containerName="pull"
Sep 17 22:05:02.248385 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:02.248279    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e4ddf34-8bfe-4a02-9bbf-4a0843742f02" containerName="pull"
Sep 17 22:05:02.248385 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:02.248288    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e4ddf34-8bfe-4a02-9bbf-4a0843742f02" containerName="extract"
Sep 17 22:05:02.248385 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:02.248293    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e4ddf34-8bfe-4a02-9bbf-4a0843742f02" containerName="extract"
Sep 17 22:05:02.248385 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:02.248308    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e4ddf34-8bfe-4a02-9bbf-4a0843742f02" containerName="util"
Sep 17 22:05:02.248385 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:02.248313    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e4ddf34-8bfe-4a02-9bbf-4a0843742f02" containerName="util"
Sep 17 22:05:02.248385 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:02.248357    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e4ddf34-8bfe-4a02-9bbf-4a0843742f02" containerName="extract"
Sep 17 22:05:02.297664 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:02.297632    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fgx5l"]
Sep 17 22:05:02.297802 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:02.297756    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fgx5l"
Sep 17 22:05:02.304374 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:02.304349    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 17 22:05:02.304481 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:02.304466    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-zcb24\""
Sep 17 22:05:02.304629 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:02.304614    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 22:05:02.305043 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:02.305025    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 22:05:02.327992 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:02.327959    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/c6d93bde-06e4-4ff9-b125-ff6f31ca235d-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-fgx5l\" (UID: \"c6d93bde-06e4-4ff9-b125-ff6f31ca235d\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fgx5l"
Sep 17 22:05:02.328157 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:02.328012    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g74hv\" (UniqueName: \"kubernetes.io/projected/c6d93bde-06e4-4ff9-b125-ff6f31ca235d-kube-api-access-g74hv\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-fgx5l\" (UID: \"c6d93bde-06e4-4ff9-b125-ff6f31ca235d\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fgx5l"
Sep 17 22:05:02.428924 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:02.428879    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g74hv\" (UniqueName: \"kubernetes.io/projected/c6d93bde-06e4-4ff9-b125-ff6f31ca235d-kube-api-access-g74hv\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-fgx5l\" (UID: \"c6d93bde-06e4-4ff9-b125-ff6f31ca235d\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fgx5l"
Sep 17 22:05:02.429132 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:02.428973    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/c6d93bde-06e4-4ff9-b125-ff6f31ca235d-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-fgx5l\" (UID: \"c6d93bde-06e4-4ff9-b125-ff6f31ca235d\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fgx5l"
Sep 17 22:05:02.431343 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:02.431324    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/c6d93bde-06e4-4ff9-b125-ff6f31ca235d-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-fgx5l\" (UID: \"c6d93bde-06e4-4ff9-b125-ff6f31ca235d\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fgx5l"
Sep 17 22:05:02.438896 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:02.438875    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g74hv\" (UniqueName: \"kubernetes.io/projected/c6d93bde-06e4-4ff9-b125-ff6f31ca235d-kube-api-access-g74hv\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-fgx5l\" (UID: \"c6d93bde-06e4-4ff9-b125-ff6f31ca235d\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fgx5l"
Sep 17 22:05:02.608163 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:02.608078    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fgx5l"
Sep 17 22:05:02.902442 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:02.902413    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fgx5l"]
Sep 17 22:05:02.905391 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:05:02.905360    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6d93bde_06e4_4ff9_b125_ff6f31ca235d.slice/crio-39ba5b5ac707e43a96cf27013bae94d46eb4cd36a0ef6f1a2d3ec2ea64f0b9b1 WatchSource:0}: Error finding container 39ba5b5ac707e43a96cf27013bae94d46eb4cd36a0ef6f1a2d3ec2ea64f0b9b1: Status 404 returned error can't find the container with id 39ba5b5ac707e43a96cf27013bae94d46eb4cd36a0ef6f1a2d3ec2ea64f0b9b1
Sep 17 22:05:03.577210 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:03.577158    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fgx5l" event={"ID":"c6d93bde-06e4-4ff9-b125-ff6f31ca235d","Type":"ContainerStarted","Data":"39ba5b5ac707e43a96cf27013bae94d46eb4cd36a0ef6f1a2d3ec2ea64f0b9b1"}
Sep 17 22:05:07.599363 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:07.599326    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fgx5l" event={"ID":"c6d93bde-06e4-4ff9-b125-ff6f31ca235d","Type":"ContainerStarted","Data":"d34bcd5b0aa382ce2ad0a089888c8509af6f8c7c39aae99d246bdb790cbf16a8"}
Sep 17 22:05:07.599782 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:07.599415    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fgx5l"
Sep 17 22:05:07.618309 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:07.618258    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fgx5l" podStartSLOduration=1.094193986 podStartE2EDuration="5.618243974s" totalImagesPullingTime="4.524049988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:05:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:05:07.617409766 +0000 UTC m=+409.063455039" watchObservedRunningTime="2026-09-17 22:05:07.618243974 +0000 UTC m=+409.064289244"
Sep 17 22:05:07.889272 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:07.889242    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:05:08.159389 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.159318    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:05:08.167881 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.167855    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-6k696"]
Sep 17 22:05:08.190274 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.190250    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-6k696"]
Sep 17 22:05:08.190391 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.190337    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-6k696"
Sep 17 22:05:08.192807 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.192777    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 17 22:05:08.193025 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.193007    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 22:05:08.193118 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.193030    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-9sx48\""
Sep 17 22:05:08.282133 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.282102    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/c1af53e7-e5fa-424a-85e1-de723d539c7f-cabundle0\") pod \"keda-operator-579c8958f7-6k696\" (UID: \"c1af53e7-e5fa-424a-85e1-de723d539c7f\") " pod="openshift-keda/keda-operator-579c8958f7-6k696"
Sep 17 22:05:08.282278 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.282228    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c1af53e7-e5fa-424a-85e1-de723d539c7f-certificates\") pod \"keda-operator-579c8958f7-6k696\" (UID: \"c1af53e7-e5fa-424a-85e1-de723d539c7f\") " pod="openshift-keda/keda-operator-579c8958f7-6k696"
Sep 17 22:05:08.282278 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.282268    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj5gm\" (UniqueName: \"kubernetes.io/projected/c1af53e7-e5fa-424a-85e1-de723d539c7f-kube-api-access-rj5gm\") pod \"keda-operator-579c8958f7-6k696\" (UID: \"c1af53e7-e5fa-424a-85e1-de723d539c7f\") " pod="openshift-keda/keda-operator-579c8958f7-6k696"
Sep 17 22:05:08.302806 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.302780    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:05:08.364429 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.364398    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-hn7qz"]
Sep 17 22:05:08.378535 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.378507    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:05:08.381330 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.381309    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-hn7qz"]
Sep 17 22:05:08.381466 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.381413    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-hn7qz"
Sep 17 22:05:08.382730 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.382704    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c1af53e7-e5fa-424a-85e1-de723d539c7f-certificates\") pod \"keda-operator-579c8958f7-6k696\" (UID: \"c1af53e7-e5fa-424a-85e1-de723d539c7f\") " pod="openshift-keda/keda-operator-579c8958f7-6k696"
Sep 17 22:05:08.382831 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.382759    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj5gm\" (UniqueName: \"kubernetes.io/projected/c1af53e7-e5fa-424a-85e1-de723d539c7f-kube-api-access-rj5gm\") pod \"keda-operator-579c8958f7-6k696\" (UID: \"c1af53e7-e5fa-424a-85e1-de723d539c7f\") " pod="openshift-keda/keda-operator-579c8958f7-6k696"
Sep 17 22:05:08.382831 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.382811    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/c1af53e7-e5fa-424a-85e1-de723d539c7f-cabundle0\") pod \"keda-operator-579c8958f7-6k696\" (UID: \"c1af53e7-e5fa-424a-85e1-de723d539c7f\") " pod="openshift-keda/keda-operator-579c8958f7-6k696"
Sep 17 22:05:08.382938 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:05:08.382856    2555 secret.go:281] references non-existent secret key: ca.crt
Sep 17 22:05:08.382938 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:05:08.382876    2555 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 17 22:05:08.382938 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:05:08.382886    2555 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-6k696: references non-existent secret key: ca.crt
Sep 17 22:05:08.383087 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:05:08.382949    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1af53e7-e5fa-424a-85e1-de723d539c7f-certificates podName:c1af53e7-e5fa-424a-85e1-de723d539c7f nodeName:}" failed. No retries permitted until 2026-09-17 22:05:08.882930535 +0000 UTC m=+410.328975804 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/c1af53e7-e5fa-424a-85e1-de723d539c7f-certificates") pod "keda-operator-579c8958f7-6k696" (UID: "c1af53e7-e5fa-424a-85e1-de723d539c7f") : references non-existent secret key: ca.crt
Sep 17 22:05:08.383736 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.383715    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/c1af53e7-e5fa-424a-85e1-de723d539c7f-cabundle0\") pod \"keda-operator-579c8958f7-6k696\" (UID: \"c1af53e7-e5fa-424a-85e1-de723d539c7f\") " pod="openshift-keda/keda-operator-579c8958f7-6k696"
Sep 17 22:05:08.383967 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.383951    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 17 22:05:08.401892 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.401869    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj5gm\" (UniqueName: \"kubernetes.io/projected/c1af53e7-e5fa-424a-85e1-de723d539c7f-kube-api-access-rj5gm\") pod \"keda-operator-579c8958f7-6k696\" (UID: \"c1af53e7-e5fa-424a-85e1-de723d539c7f\") " pod="openshift-keda/keda-operator-579c8958f7-6k696"
Sep 17 22:05:08.475354 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.475273    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-84b584ccc8-th4zj"]
Sep 17 22:05:08.484204 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.484156    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0a49cf89-12af-4e84-b970-b99dab40fa28-certificates\") pod \"keda-metrics-apiserver-77857b7b98-hn7qz\" (UID: \"0a49cf89-12af-4e84-b970-b99dab40fa28\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-hn7qz"
Sep 17 22:05:08.484351 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.484326    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/0a49cf89-12af-4e84-b970-b99dab40fa28-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-hn7qz\" (UID: \"0a49cf89-12af-4e84-b970-b99dab40fa28\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-hn7qz"
Sep 17 22:05:08.484418 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.484366    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f65s\" (UniqueName: \"kubernetes.io/projected/0a49cf89-12af-4e84-b970-b99dab40fa28-kube-api-access-7f65s\") pod \"keda-metrics-apiserver-77857b7b98-hn7qz\" (UID: \"0a49cf89-12af-4e84-b970-b99dab40fa28\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-hn7qz"
Sep 17 22:05:08.497902 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.497387    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84b584ccc8-th4zj"]
Sep 17 22:05:08.497902 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.497501    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84b584ccc8-th4zj"
Sep 17 22:05:08.563597 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.563572    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:05:08.584774 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.584752    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f-console-oauth-config\") pod \"console-84b584ccc8-th4zj\" (UID: \"8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f\") " pod="openshift-console/console-84b584ccc8-th4zj"
Sep 17 22:05:08.584906 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.584787    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f-console-serving-cert\") pod \"console-84b584ccc8-th4zj\" (UID: \"8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f\") " pod="openshift-console/console-84b584ccc8-th4zj"
Sep 17 22:05:08.584906 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.584824    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f-trusted-ca-bundle\") pod \"console-84b584ccc8-th4zj\" (UID: \"8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f\") " pod="openshift-console/console-84b584ccc8-th4zj"
Sep 17 22:05:08.584906 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.584887    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f-oauth-serving-cert\") pod \"console-84b584ccc8-th4zj\" (UID: \"8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f\") " pod="openshift-console/console-84b584ccc8-th4zj"
Sep 17 22:05:08.585022 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.584954    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/0a49cf89-12af-4e84-b970-b99dab40fa28-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-hn7qz\" (UID: \"0a49cf89-12af-4e84-b970-b99dab40fa28\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-hn7qz"
Sep 17 22:05:08.585022 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.584984    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f65s\" (UniqueName: \"kubernetes.io/projected/0a49cf89-12af-4e84-b970-b99dab40fa28-kube-api-access-7f65s\") pod \"keda-metrics-apiserver-77857b7b98-hn7qz\" (UID: \"0a49cf89-12af-4e84-b970-b99dab40fa28\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-hn7qz"
Sep 17 22:05:08.585113 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.585035    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0a49cf89-12af-4e84-b970-b99dab40fa28-certificates\") pod \"keda-metrics-apiserver-77857b7b98-hn7qz\" (UID: \"0a49cf89-12af-4e84-b970-b99dab40fa28\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-hn7qz"
Sep 17 22:05:08.585113 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.585073    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f-console-config\") pod \"console-84b584ccc8-th4zj\" (UID: \"8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f\") " pod="openshift-console/console-84b584ccc8-th4zj"
Sep 17 22:05:08.585113 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.585100    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f-service-ca\") pod \"console-84b584ccc8-th4zj\" (UID: \"8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f\") " pod="openshift-console/console-84b584ccc8-th4zj"
Sep 17 22:05:08.585267 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:05:08.585161    2555 secret.go:281] references non-existent secret key: tls.crt
Sep 17 22:05:08.585267 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:05:08.585198    2555 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Sep 17 22:05:08.585267 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:05:08.585221    2555 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Sep 17 22:05:08.585267 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:05:08.585242    2555 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-hn7qz: [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Sep 17 22:05:08.585267 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.585241    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5dkl\" (UniqueName: \"kubernetes.io/projected/8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f-kube-api-access-m5dkl\") pod \"console-84b584ccc8-th4zj\" (UID: \"8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f\") " pod="openshift-console/console-84b584ccc8-th4zj"
Sep 17 22:05:08.585541 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:05:08.585297    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a49cf89-12af-4e84-b970-b99dab40fa28-certificates podName:0a49cf89-12af-4e84-b970-b99dab40fa28 nodeName:}" failed. No retries permitted until 2026-09-17 22:05:09.085279703 +0000 UTC m=+410.531324972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/0a49cf89-12af-4e84-b970-b99dab40fa28-certificates") pod "keda-metrics-apiserver-77857b7b98-hn7qz" (UID: "0a49cf89-12af-4e84-b970-b99dab40fa28") : [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Sep 17 22:05:08.585541 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.585340    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/0a49cf89-12af-4e84-b970-b99dab40fa28-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-hn7qz\" (UID: \"0a49cf89-12af-4e84-b970-b99dab40fa28\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-hn7qz"
Sep 17 22:05:08.593971 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.593949    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f65s\" (UniqueName: \"kubernetes.io/projected/0a49cf89-12af-4e84-b970-b99dab40fa28-kube-api-access-7f65s\") pod \"keda-metrics-apiserver-77857b7b98-hn7qz\" (UID: \"0a49cf89-12af-4e84-b970-b99dab40fa28\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-hn7qz"
Sep 17 22:05:08.664777 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.664745    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:05:08.686777 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.686744    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f-console-config\") pod \"console-84b584ccc8-th4zj\" (UID: \"8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f\") " pod="openshift-console/console-84b584ccc8-th4zj"
Sep 17 22:05:08.686943 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.686788    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f-service-ca\") pod \"console-84b584ccc8-th4zj\" (UID: \"8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f\") " pod="openshift-console/console-84b584ccc8-th4zj"
Sep 17 22:05:08.686943 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.686861    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5dkl\" (UniqueName: \"kubernetes.io/projected/8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f-kube-api-access-m5dkl\") pod \"console-84b584ccc8-th4zj\" (UID: \"8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f\") " pod="openshift-console/console-84b584ccc8-th4zj"
Sep 17 22:05:08.687053 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.686945    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f-console-oauth-config\") pod \"console-84b584ccc8-th4zj\" (UID: \"8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f\") " pod="openshift-console/console-84b584ccc8-th4zj"
Sep 17 22:05:08.687053 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.686981    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f-console-serving-cert\") pod \"console-84b584ccc8-th4zj\" (UID: \"8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f\") " pod="openshift-console/console-84b584ccc8-th4zj"
Sep 17 22:05:08.687053 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.687027    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f-trusted-ca-bundle\") pod \"console-84b584ccc8-th4zj\" (UID: \"8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f\") " pod="openshift-console/console-84b584ccc8-th4zj"
Sep 17 22:05:08.687226 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.687056    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f-oauth-serving-cert\") pod \"console-84b584ccc8-th4zj\" (UID: \"8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f\") " pod="openshift-console/console-84b584ccc8-th4zj"
Sep 17 22:05:08.687583 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.687554    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f-console-config\") pod \"console-84b584ccc8-th4zj\" (UID: \"8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f\") " pod="openshift-console/console-84b584ccc8-th4zj"
Sep 17 22:05:08.687693 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.687647    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f-service-ca\") pod \"console-84b584ccc8-th4zj\" (UID: \"8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f\") " pod="openshift-console/console-84b584ccc8-th4zj"
Sep 17 22:05:08.687763 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.687724    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f-oauth-serving-cert\") pod \"console-84b584ccc8-th4zj\" (UID: \"8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f\") " pod="openshift-console/console-84b584ccc8-th4zj"
Sep 17 22:05:08.688505 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.688479    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f-trusted-ca-bundle\") pod \"console-84b584ccc8-th4zj\" (UID: \"8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f\") " pod="openshift-console/console-84b584ccc8-th4zj"
Sep 17 22:05:08.689793 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.689772    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f-console-serving-cert\") pod \"console-84b584ccc8-th4zj\" (UID: \"8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f\") " pod="openshift-console/console-84b584ccc8-th4zj"
Sep 17 22:05:08.689897 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.689858    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f-console-oauth-config\") pod \"console-84b584ccc8-th4zj\" (UID: \"8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f\") " pod="openshift-console/console-84b584ccc8-th4zj"
Sep 17 22:05:08.694891 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.694871    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5dkl\" (UniqueName: \"kubernetes.io/projected/8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f-kube-api-access-m5dkl\") pod \"console-84b584ccc8-th4zj\" (UID: \"8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f\") " pod="openshift-console/console-84b584ccc8-th4zj"
Sep 17 22:05:08.764602 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.764529    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:05:08.807785 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.807752    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84b584ccc8-th4zj"
Sep 17 22:05:08.889261 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.889077    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c1af53e7-e5fa-424a-85e1-de723d539c7f-certificates\") pod \"keda-operator-579c8958f7-6k696\" (UID: \"c1af53e7-e5fa-424a-85e1-de723d539c7f\") " pod="openshift-keda/keda-operator-579c8958f7-6k696"
Sep 17 22:05:08.891536 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.891516    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c1af53e7-e5fa-424a-85e1-de723d539c7f-certificates\") pod \"keda-operator-579c8958f7-6k696\" (UID: \"c1af53e7-e5fa-424a-85e1-de723d539c7f\") " pod="openshift-keda/keda-operator-579c8958f7-6k696"
Sep 17 22:05:08.943643 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:08.943618    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:05:09.025508 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:09.025432    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:05:09.090057 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:09.090031    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84b584ccc8-th4zj"]
Sep 17 22:05:09.090402 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:09.090382    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0a49cf89-12af-4e84-b970-b99dab40fa28-certificates\") pod \"keda-metrics-apiserver-77857b7b98-hn7qz\" (UID: \"0a49cf89-12af-4e84-b970-b99dab40fa28\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-hn7qz"
Sep 17 22:05:09.091902 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:05:09.091877    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ecc6ad2_d7e3_4b5f_8bc8_f1eacb39e42f.slice/crio-73d23da1d2537126583a6b85575819c6a6edaf81b89ece9c58221f314fba3da0 WatchSource:0}: Error finding container 73d23da1d2537126583a6b85575819c6a6edaf81b89ece9c58221f314fba3da0: Status 404 returned error can't find the container with id 73d23da1d2537126583a6b85575819c6a6edaf81b89ece9c58221f314fba3da0
Sep 17 22:05:09.092989 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:09.092968    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0a49cf89-12af-4e84-b970-b99dab40fa28-certificates\") pod \"keda-metrics-apiserver-77857b7b98-hn7qz\" (UID: \"0a49cf89-12af-4e84-b970-b99dab40fa28\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-hn7qz"
Sep 17 22:05:09.093394 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:09.093373    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:05:09.100168 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:09.100152    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-6k696"
Sep 17 22:05:09.209949 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:09.209918    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:05:09.294130 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:09.294051    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-hn7qz"
Sep 17 22:05:09.337723 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:09.337694    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:05:09.373236 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:09.373213    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-6k696"]
Sep 17 22:05:09.375948 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:05:09.375921    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1af53e7_e5fa_424a_85e1_de723d539c7f.slice/crio-99f607914216c7c2b8b5b51d6edd2617db31b0f2313818dc3237dc9153879e34 WatchSource:0}: Error finding container 99f607914216c7c2b8b5b51d6edd2617db31b0f2313818dc3237dc9153879e34: Status 404 returned error can't find the container with id 99f607914216c7c2b8b5b51d6edd2617db31b0f2313818dc3237dc9153879e34
Sep 17 22:05:09.512636 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:09.512602    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:05:09.564684 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:09.564659    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-hn7qz"]
Sep 17 22:05:09.566798 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:05:09.566770    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a49cf89_12af_4e84_b970_b99dab40fa28.slice/crio-020c703f464db095abaf265776b075f832041d38033b3e628998e2bdf50e68c8 WatchSource:0}: Error finding container 020c703f464db095abaf265776b075f832041d38033b3e628998e2bdf50e68c8: Status 404 returned error can't find the container with id 020c703f464db095abaf265776b075f832041d38033b3e628998e2bdf50e68c8
Sep 17 22:05:09.610171 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:09.610138    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-hn7qz" event={"ID":"0a49cf89-12af-4e84-b970-b99dab40fa28","Type":"ContainerStarted","Data":"020c703f464db095abaf265776b075f832041d38033b3e628998e2bdf50e68c8"}
Sep 17 22:05:09.611311 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:09.611291    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-6k696" event={"ID":"c1af53e7-e5fa-424a-85e1-de723d539c7f","Type":"ContainerStarted","Data":"99f607914216c7c2b8b5b51d6edd2617db31b0f2313818dc3237dc9153879e34"}
Sep 17 22:05:09.612891 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:09.612866    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84b584ccc8-th4zj" event={"ID":"8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f","Type":"ContainerStarted","Data":"ef5b1e62eacc8101024039dda219970b46bdfba598c9648f5a7099f1ff0c29ff"}
Sep 17 22:05:09.612985 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:09.612907    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84b584ccc8-th4zj" event={"ID":"8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f","Type":"ContainerStarted","Data":"73d23da1d2537126583a6b85575819c6a6edaf81b89ece9c58221f314fba3da0"}
Sep 17 22:05:09.631796 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:09.631768    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:05:09.639018 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:09.638976    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-84b584ccc8-th4zj" podStartSLOduration=1.638963784 podStartE2EDuration="1.638963784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:05:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:05:09.638108603 +0000 UTC m=+411.084153877" watchObservedRunningTime="2026-09-17 22:05:09.638963784 +0000 UTC m=+411.085009055"
Sep 17 22:05:09.724998 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:09.724957    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:05:09.829573 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:09.829499    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:05:15.643337 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:15.643301    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-6k696" event={"ID":"c1af53e7-e5fa-424a-85e1-de723d539c7f","Type":"ContainerStarted","Data":"e74e1223bba8ae8b1a09e19c35e2b2591ae1affdf124f8978e51e74aab3dd625"}
Sep 17 22:05:15.643726 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:15.643444    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-6k696"
Sep 17 22:05:15.660235 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:15.660170    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-6k696" podStartSLOduration=1.6111249029999999 podStartE2EDuration="7.66015563s" totalImagesPullingTime="6.049030727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:05:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:05:15.658436529 +0000 UTC m=+417.104481793" watchObservedRunningTime="2026-09-17 22:05:15.66015563 +0000 UTC m=+417.106200901"
Sep 17 22:05:18.659107 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:18.659071    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-hn7qz" event={"ID":"0a49cf89-12af-4e84-b970-b99dab40fa28","Type":"ContainerStarted","Data":"d6a7b8a6bde1ddf7233f19da9ee9586ea28b5b49d64ec61680bf76de61c32b3c"}
Sep 17 22:05:18.659483 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:18.659136    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-hn7qz"
Sep 17 22:05:18.674537 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:18.674490    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-hn7qz" podStartSLOduration=2.4456214210000002 podStartE2EDuration="10.674475007s" totalImagesPullingTime="8.228853586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:05:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:05:18.673977681 +0000 UTC m=+420.120022955" watchObservedRunningTime="2026-09-17 22:05:18.674475007 +0000 UTC m=+420.120520279"
Sep 17 22:05:18.808757 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:18.808725    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84b584ccc8-th4zj"
Sep 17 22:05:18.809046 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:18.809022    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84b584ccc8-th4zj"
Sep 17 22:05:18.814659 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:18.814622    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84b584ccc8-th4zj"
Sep 17 22:05:19.668589 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:19.668556    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84b584ccc8-th4zj"
Sep 17 22:05:19.714099 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:19.714071    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79df74fcd5-r7ncf"]
Sep 17 22:05:19.811365 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:19.811290    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:05:19.871626 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:19.871595    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:05:19.925804 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:19.925731    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:05:28.606480 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:28.606451    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fgx5l"
Sep 17 22:05:29.668743 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:29.668715    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-hn7qz"
Sep 17 22:05:36.650964 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:36.650935    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-6k696"
Sep 17 22:05:44.737555 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:44.737515    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-79df74fcd5-r7ncf" podUID="4197ebae-9c48-4dde-b941-0406df61ce88" containerName="console" containerID="cri-o://a7060025825ee26c670af6fde0f84d1077c60dd1d3f435ad89cc8133bc93c8de" gracePeriod=15
Sep 17 22:05:44.982596 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:44.982572    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79df74fcd5-r7ncf"
Sep 17 22:05:44.985730 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:44.985708    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4197ebae-9c48-4dde-b941-0406df61ce88-service-ca\") pod \"4197ebae-9c48-4dde-b941-0406df61ce88\" (UID: \"4197ebae-9c48-4dde-b941-0406df61ce88\") "
Sep 17 22:05:44.985803 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:44.985757    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4197ebae-9c48-4dde-b941-0406df61ce88-oauth-serving-cert\") pod \"4197ebae-9c48-4dde-b941-0406df61ce88\" (UID: \"4197ebae-9c48-4dde-b941-0406df61ce88\") "
Sep 17 22:05:44.985803 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:44.985773    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4197ebae-9c48-4dde-b941-0406df61ce88-console-config\") pod \"4197ebae-9c48-4dde-b941-0406df61ce88\" (UID: \"4197ebae-9c48-4dde-b941-0406df61ce88\") "
Sep 17 22:05:44.985921 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:44.985901    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4197ebae-9c48-4dde-b941-0406df61ce88-console-oauth-config\") pod \"4197ebae-9c48-4dde-b941-0406df61ce88\" (UID: \"4197ebae-9c48-4dde-b941-0406df61ce88\") "
Sep 17 22:05:44.985984 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:44.985949    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4197ebae-9c48-4dde-b941-0406df61ce88-trusted-ca-bundle\") pod \"4197ebae-9c48-4dde-b941-0406df61ce88\" (UID: \"4197ebae-9c48-4dde-b941-0406df61ce88\") "
Sep 17 22:05:44.986036 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:44.985994    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4197ebae-9c48-4dde-b941-0406df61ce88-console-serving-cert\") pod \"4197ebae-9c48-4dde-b941-0406df61ce88\" (UID: \"4197ebae-9c48-4dde-b941-0406df61ce88\") "
Sep 17 22:05:44.986088 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:44.986042    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2m2kw\" (UniqueName: \"kubernetes.io/projected/4197ebae-9c48-4dde-b941-0406df61ce88-kube-api-access-2m2kw\") pod \"4197ebae-9c48-4dde-b941-0406df61ce88\" (UID: \"4197ebae-9c48-4dde-b941-0406df61ce88\") "
Sep 17 22:05:44.986138 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:44.986079    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4197ebae-9c48-4dde-b941-0406df61ce88-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4197ebae-9c48-4dde-b941-0406df61ce88" (UID: "4197ebae-9c48-4dde-b941-0406df61ce88"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:05:44.986138 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:44.986097    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4197ebae-9c48-4dde-b941-0406df61ce88-console-config" (OuterVolumeSpecName: "console-config") pod "4197ebae-9c48-4dde-b941-0406df61ce88" (UID: "4197ebae-9c48-4dde-b941-0406df61ce88"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:05:44.986138 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:44.986099    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4197ebae-9c48-4dde-b941-0406df61ce88-service-ca" (OuterVolumeSpecName: "service-ca") pod "4197ebae-9c48-4dde-b941-0406df61ce88" (UID: "4197ebae-9c48-4dde-b941-0406df61ce88"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:05:44.986647 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:44.986604    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4197ebae-9c48-4dde-b941-0406df61ce88-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4197ebae-9c48-4dde-b941-0406df61ce88" (UID: "4197ebae-9c48-4dde-b941-0406df61ce88"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:05:44.986959 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:44.986939    2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4197ebae-9c48-4dde-b941-0406df61ce88-oauth-serving-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:05:44.986959 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:44.986961    2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4197ebae-9c48-4dde-b941-0406df61ce88-console-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:05:44.987105 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:44.986970    2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4197ebae-9c48-4dde-b941-0406df61ce88-trusted-ca-bundle\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:05:44.987105 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:44.986978    2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4197ebae-9c48-4dde-b941-0406df61ce88-service-ca\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:05:44.988129 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:44.988070    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4197ebae-9c48-4dde-b941-0406df61ce88-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4197ebae-9c48-4dde-b941-0406df61ce88" (UID: "4197ebae-9c48-4dde-b941-0406df61ce88"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:05:44.988129 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:44.988125    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4197ebae-9c48-4dde-b941-0406df61ce88-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4197ebae-9c48-4dde-b941-0406df61ce88" (UID: "4197ebae-9c48-4dde-b941-0406df61ce88"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:05:44.988345 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:44.988325    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4197ebae-9c48-4dde-b941-0406df61ce88-kube-api-access-2m2kw" (OuterVolumeSpecName: "kube-api-access-2m2kw") pod "4197ebae-9c48-4dde-b941-0406df61ce88" (UID: "4197ebae-9c48-4dde-b941-0406df61ce88"). InnerVolumeSpecName "kube-api-access-2m2kw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:05:45.091900 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:45.091869    2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4197ebae-9c48-4dde-b941-0406df61ce88-console-oauth-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:05:45.091900 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:45.091894    2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4197ebae-9c48-4dde-b941-0406df61ce88-console-serving-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:05:45.091900 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:45.091906    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m2kw\" (UniqueName: \"kubernetes.io/projected/4197ebae-9c48-4dde-b941-0406df61ce88-kube-api-access-2m2kw\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:05:45.784378 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:45.784342    2555 generic.go:356] "Generic (PLEG): container finished" podID="4197ebae-9c48-4dde-b941-0406df61ce88" containerID="a7060025825ee26c670af6fde0f84d1077c60dd1d3f435ad89cc8133bc93c8de" exitCode=0
Sep 17 22:05:45.784759 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:45.784404    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79df74fcd5-r7ncf"
Sep 17 22:05:45.784759 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:45.784429    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79df74fcd5-r7ncf" event={"ID":"4197ebae-9c48-4dde-b941-0406df61ce88","Type":"ContainerDied","Data":"a7060025825ee26c670af6fde0f84d1077c60dd1d3f435ad89cc8133bc93c8de"}
Sep 17 22:05:45.784759 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:45.784480    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79df74fcd5-r7ncf" event={"ID":"4197ebae-9c48-4dde-b941-0406df61ce88","Type":"ContainerDied","Data":"f2e61d89ac359103a696ec6cff1a2a6d569b9b414cf864ec55c41ad6319cdf45"}
Sep 17 22:05:45.784759 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:45.784500    2555 scope.go:117] "RemoveContainer" containerID="a7060025825ee26c670af6fde0f84d1077c60dd1d3f435ad89cc8133bc93c8de"
Sep 17 22:05:45.793900 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:45.793885    2555 scope.go:117] "RemoveContainer" containerID="a7060025825ee26c670af6fde0f84d1077c60dd1d3f435ad89cc8133bc93c8de"
Sep 17 22:05:45.794152 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:05:45.794134    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7060025825ee26c670af6fde0f84d1077c60dd1d3f435ad89cc8133bc93c8de\": container with ID starting with a7060025825ee26c670af6fde0f84d1077c60dd1d3f435ad89cc8133bc93c8de not found: ID does not exist" containerID="a7060025825ee26c670af6fde0f84d1077c60dd1d3f435ad89cc8133bc93c8de"
Sep 17 22:05:45.794214 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:45.794160    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7060025825ee26c670af6fde0f84d1077c60dd1d3f435ad89cc8133bc93c8de"} err="failed to get container status \"a7060025825ee26c670af6fde0f84d1077c60dd1d3f435ad89cc8133bc93c8de\": rpc error: code = NotFound desc = could not find container \"a7060025825ee26c670af6fde0f84d1077c60dd1d3f435ad89cc8133bc93c8de\": container with ID starting with a7060025825ee26c670af6fde0f84d1077c60dd1d3f435ad89cc8133bc93c8de not found: ID does not exist"
Sep 17 22:05:45.805111 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:45.805091    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79df74fcd5-r7ncf"]
Sep 17 22:05:45.809021 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:45.809002    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-79df74fcd5-r7ncf"]
Sep 17 22:05:46.678059 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:46.678026    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4197ebae-9c48-4dde-b941-0406df61ce88" path="/var/lib/kubelet/pods/4197ebae-9c48-4dde-b941-0406df61ce88/volumes"
Sep 17 22:05:53.724469 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:53.724438    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:05:59.552282 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:59.552249    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:05:59.651996 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:59.651968    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:05:59.793939 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:59.793903    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rv4j"]
Sep 17 22:05:59.794226 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:59.794214    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4197ebae-9c48-4dde-b941-0406df61ce88" containerName="console"
Sep 17 22:05:59.794284 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:59.794227    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4197ebae-9c48-4dde-b941-0406df61ce88" containerName="console"
Sep 17 22:05:59.794326 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:59.794288    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4197ebae-9c48-4dde-b941-0406df61ce88" containerName="console"
Sep 17 22:05:59.802867 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:59.802796    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rv4j"
Sep 17 22:05:59.804532 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:59.804498    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rv4j"]
Sep 17 22:05:59.805291 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:59.805268    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-h4r5v\""
Sep 17 22:05:59.805395 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:59.805270    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 22:05:59.806012 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:59.805994    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 22:05:59.902162 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:59.902130    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f806330-23a6-489b-946d-675e3f9ee619-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rv4j\" (UID: \"4f806330-23a6-489b-946d-675e3f9ee619\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rv4j"
Sep 17 22:05:59.902343 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:59.902229    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f806330-23a6-489b-946d-675e3f9ee619-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rv4j\" (UID: \"4f806330-23a6-489b-946d-675e3f9ee619\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rv4j"
Sep 17 22:05:59.902343 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:05:59.902258    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45zdp\" (UniqueName: \"kubernetes.io/projected/4f806330-23a6-489b-946d-675e3f9ee619-kube-api-access-45zdp\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rv4j\" (UID: \"4f806330-23a6-489b-946d-675e3f9ee619\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rv4j"
Sep 17 22:06:00.003231 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:00.003176    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f806330-23a6-489b-946d-675e3f9ee619-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rv4j\" (UID: \"4f806330-23a6-489b-946d-675e3f9ee619\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rv4j"
Sep 17 22:06:00.003344 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:00.003243    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45zdp\" (UniqueName: \"kubernetes.io/projected/4f806330-23a6-489b-946d-675e3f9ee619-kube-api-access-45zdp\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rv4j\" (UID: \"4f806330-23a6-489b-946d-675e3f9ee619\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rv4j"
Sep 17 22:06:00.003344 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:00.003287    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f806330-23a6-489b-946d-675e3f9ee619-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rv4j\" (UID: \"4f806330-23a6-489b-946d-675e3f9ee619\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rv4j"
Sep 17 22:06:00.003607 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:00.003588    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f806330-23a6-489b-946d-675e3f9ee619-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rv4j\" (UID: \"4f806330-23a6-489b-946d-675e3f9ee619\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rv4j"
Sep 17 22:06:00.003642 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:00.003619    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f806330-23a6-489b-946d-675e3f9ee619-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rv4j\" (UID: \"4f806330-23a6-489b-946d-675e3f9ee619\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rv4j"
Sep 17 22:06:00.011378 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:00.011353    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45zdp\" (UniqueName: \"kubernetes.io/projected/4f806330-23a6-489b-946d-675e3f9ee619-kube-api-access-45zdp\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rv4j\" (UID: \"4f806330-23a6-489b-946d-675e3f9ee619\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rv4j"
Sep 17 22:06:00.113607 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:00.113539    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rv4j"
Sep 17 22:06:00.382131 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:00.382059    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rv4j"]
Sep 17 22:06:00.384463 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:06:00.384431    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f806330_23a6_489b_946d_675e3f9ee619.slice/crio-49077cba07e1b3019578cd8971f16f7feda71b52ddfd5a590286a758b4943709 WatchSource:0}: Error finding container 49077cba07e1b3019578cd8971f16f7feda71b52ddfd5a590286a758b4943709: Status 404 returned error can't find the container with id 49077cba07e1b3019578cd8971f16f7feda71b52ddfd5a590286a758b4943709
Sep 17 22:06:00.855368 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:00.855275    2555 generic.go:356] "Generic (PLEG): container finished" podID="4f806330-23a6-489b-946d-675e3f9ee619" containerID="f143df5fb630f2d4d549d86bb65f12948bea6108a8190175193e4973219323ff" exitCode=0
Sep 17 22:06:00.855368 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:00.855351    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rv4j" event={"ID":"4f806330-23a6-489b-946d-675e3f9ee619","Type":"ContainerDied","Data":"f143df5fb630f2d4d549d86bb65f12948bea6108a8190175193e4973219323ff"}
Sep 17 22:06:00.855753 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:00.855393    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rv4j" event={"ID":"4f806330-23a6-489b-946d-675e3f9ee619","Type":"ContainerStarted","Data":"49077cba07e1b3019578cd8971f16f7feda71b52ddfd5a590286a758b4943709"}
Sep 17 22:06:03.872119 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:03.872079    2555 generic.go:356] "Generic (PLEG): container finished" podID="4f806330-23a6-489b-946d-675e3f9ee619" containerID="20b8dbc2b7755b03edffc539657acaf514c0489ba19841502eb4bbdd3f29dc64" exitCode=0
Sep 17 22:06:03.872512 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:03.872194    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rv4j" event={"ID":"4f806330-23a6-489b-946d-675e3f9ee619","Type":"ContainerDied","Data":"20b8dbc2b7755b03edffc539657acaf514c0489ba19841502eb4bbdd3f29dc64"}
Sep 17 22:06:04.879585 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:04.879550    2555 generic.go:356] "Generic (PLEG): container finished" podID="4f806330-23a6-489b-946d-675e3f9ee619" containerID="6945b0421244226561983a9d6c44b95a9eb5bbb8d24b4859d402d2b022e2ba6f" exitCode=0
Sep 17 22:06:04.879956 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:04.879615    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rv4j" event={"ID":"4f806330-23a6-489b-946d-675e3f9ee619","Type":"ContainerDied","Data":"6945b0421244226561983a9d6c44b95a9eb5bbb8d24b4859d402d2b022e2ba6f"}
Sep 17 22:06:06.018786 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:06.018760    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rv4j"
Sep 17 22:06:06.152540 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:06.152462    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45zdp\" (UniqueName: \"kubernetes.io/projected/4f806330-23a6-489b-946d-675e3f9ee619-kube-api-access-45zdp\") pod \"4f806330-23a6-489b-946d-675e3f9ee619\" (UID: \"4f806330-23a6-489b-946d-675e3f9ee619\") "
Sep 17 22:06:06.152540 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:06.152504    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f806330-23a6-489b-946d-675e3f9ee619-util\") pod \"4f806330-23a6-489b-946d-675e3f9ee619\" (UID: \"4f806330-23a6-489b-946d-675e3f9ee619\") "
Sep 17 22:06:06.152760 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:06.152551    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f806330-23a6-489b-946d-675e3f9ee619-bundle\") pod \"4f806330-23a6-489b-946d-675e3f9ee619\" (UID: \"4f806330-23a6-489b-946d-675e3f9ee619\") "
Sep 17 22:06:06.153346 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:06.153318    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f806330-23a6-489b-946d-675e3f9ee619-bundle" (OuterVolumeSpecName: "bundle") pod "4f806330-23a6-489b-946d-675e3f9ee619" (UID: "4f806330-23a6-489b-946d-675e3f9ee619"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:06:06.154567 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:06.154548    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f806330-23a6-489b-946d-675e3f9ee619-kube-api-access-45zdp" (OuterVolumeSpecName: "kube-api-access-45zdp") pod "4f806330-23a6-489b-946d-675e3f9ee619" (UID: "4f806330-23a6-489b-946d-675e3f9ee619"). InnerVolumeSpecName "kube-api-access-45zdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:06:06.157132 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:06.157103    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f806330-23a6-489b-946d-675e3f9ee619-util" (OuterVolumeSpecName: "util") pod "4f806330-23a6-489b-946d-675e3f9ee619" (UID: "4f806330-23a6-489b-946d-675e3f9ee619"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:06:06.253035 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:06.253006    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45zdp\" (UniqueName: \"kubernetes.io/projected/4f806330-23a6-489b-946d-675e3f9ee619-kube-api-access-45zdp\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:06:06.253035 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:06.253032    2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f806330-23a6-489b-946d-675e3f9ee619-util\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:06:06.253196 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:06.253046    2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f806330-23a6-489b-946d-675e3f9ee619-bundle\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:06:06.889892 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:06.889858    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rv4j" event={"ID":"4f806330-23a6-489b-946d-675e3f9ee619","Type":"ContainerDied","Data":"49077cba07e1b3019578cd8971f16f7feda71b52ddfd5a590286a758b4943709"}
Sep 17 22:06:06.889892 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:06.889900    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49077cba07e1b3019578cd8971f16f7feda71b52ddfd5a590286a758b4943709"
Sep 17 22:06:06.890097 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:06.889898    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rv4j"
Sep 17 22:06:10.903506 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:10.903476    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:06:12.214992 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:12.214959    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:06:12.857687 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:12.857657    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-q892m"]
Sep 17 22:06:12.857958 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:12.857947    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f806330-23a6-489b-946d-675e3f9ee619" containerName="util"
Sep 17 22:06:12.858002 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:12.857960    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f806330-23a6-489b-946d-675e3f9ee619" containerName="util"
Sep 17 22:06:12.858002 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:12.857967    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f806330-23a6-489b-946d-675e3f9ee619" containerName="pull"
Sep 17 22:06:12.858002 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:12.857973    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f806330-23a6-489b-946d-675e3f9ee619" containerName="pull"
Sep 17 22:06:12.858002 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:12.857985    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f806330-23a6-489b-946d-675e3f9ee619" containerName="extract"
Sep 17 22:06:12.858002 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:12.857990    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f806330-23a6-489b-946d-675e3f9ee619" containerName="extract"
Sep 17 22:06:12.858145 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:12.858045    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f806330-23a6-489b-946d-675e3f9ee619" containerName="extract"
Sep 17 22:06:12.862022 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:12.862005    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-q892m"
Sep 17 22:06:12.864460 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:12.864438    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-7plxk\""
Sep 17 22:06:12.865224 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:12.864987    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 17 22:06:12.865224 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:12.865050    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 17 22:06:12.873924 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:12.873902    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-q892m"]
Sep 17 22:06:13.007803 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:13.007776    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgq8k\" (UniqueName: \"kubernetes.io/projected/ddad629a-4eac-41e2-9d17-e7bbc1d9d777-kube-api-access-mgq8k\") pod \"cert-manager-operator-controller-manager-684f79c9b9-q892m\" (UID: \"ddad629a-4eac-41e2-9d17-e7bbc1d9d777\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-q892m"
Sep 17 22:06:13.007940 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:13.007837    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ddad629a-4eac-41e2-9d17-e7bbc1d9d777-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-q892m\" (UID: \"ddad629a-4eac-41e2-9d17-e7bbc1d9d777\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-q892m"
Sep 17 22:06:13.108818 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:13.108741    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ddad629a-4eac-41e2-9d17-e7bbc1d9d777-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-q892m\" (UID: \"ddad629a-4eac-41e2-9d17-e7bbc1d9d777\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-q892m"
Sep 17 22:06:13.108818 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:13.108805    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgq8k\" (UniqueName: \"kubernetes.io/projected/ddad629a-4eac-41e2-9d17-e7bbc1d9d777-kube-api-access-mgq8k\") pod \"cert-manager-operator-controller-manager-684f79c9b9-q892m\" (UID: \"ddad629a-4eac-41e2-9d17-e7bbc1d9d777\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-q892m"
Sep 17 22:06:13.109124 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:13.109104    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ddad629a-4eac-41e2-9d17-e7bbc1d9d777-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-q892m\" (UID: \"ddad629a-4eac-41e2-9d17-e7bbc1d9d777\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-q892m"
Sep 17 22:06:13.118099 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:13.118076    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgq8k\" (UniqueName: \"kubernetes.io/projected/ddad629a-4eac-41e2-9d17-e7bbc1d9d777-kube-api-access-mgq8k\") pod \"cert-manager-operator-controller-manager-684f79c9b9-q892m\" (UID: \"ddad629a-4eac-41e2-9d17-e7bbc1d9d777\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-q892m"
Sep 17 22:06:13.174652 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:13.174630    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-q892m"
Sep 17 22:06:13.455063 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:13.455029    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-q892m"]
Sep 17 22:06:13.459112 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:06:13.459088    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddad629a_4eac_41e2_9d17_e7bbc1d9d777.slice/crio-cbfac09440d8e4afe0ed797096baa6ab7ad8518abde305113b9bd5fc8f0a4823 WatchSource:0}: Error finding container cbfac09440d8e4afe0ed797096baa6ab7ad8518abde305113b9bd5fc8f0a4823: Status 404 returned error can't find the container with id cbfac09440d8e4afe0ed797096baa6ab7ad8518abde305113b9bd5fc8f0a4823
Sep 17 22:06:13.923126 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:13.923088    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-q892m" event={"ID":"ddad629a-4eac-41e2-9d17-e7bbc1d9d777","Type":"ContainerStarted","Data":"cbfac09440d8e4afe0ed797096baa6ab7ad8518abde305113b9bd5fc8f0a4823"}
Sep 17 22:06:14.511045 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:14.511014    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:06:14.617536 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:14.617499    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:06:15.934334 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:15.934299    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-q892m" event={"ID":"ddad629a-4eac-41e2-9d17-e7bbc1d9d777","Type":"ContainerStarted","Data":"9dd04dfc120e70232cf87473b5f2698b0e2c655ae1681d6b1e1d491c21f1d01c"}
Sep 17 22:06:15.955067 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:15.955023    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-q892m" podStartSLOduration=2.246295152 podStartE2EDuration="3.95500918s" totalImagesPullingTime="1.708714028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:06:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:06:15.9539083 +0000 UTC m=+477.399953573" watchObservedRunningTime="2026-09-17 22:06:15.95500918 +0000 UTC m=+477.401054453"
Sep 17 22:06:16.777147 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:16.777112    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:06:16.881741 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:16.881710    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:06:18.978098 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:18.978071    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:06:19.135312 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:19.135279    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:06:19.211629 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:19.211600    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:06:19.312442 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:19.312364    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:06:20.970129 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:20.970094    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:06:21.082341 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:21.082311    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:06:21.178721 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:21.178692    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:06:22.120571 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:22.120535    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq4bk"]
Sep 17 22:06:22.125238 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:22.125220    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq4bk"
Sep 17 22:06:22.127402 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:22.127370    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 22:06:22.127402 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:22.127392    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-h4r5v\""
Sep 17 22:06:22.127582 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:22.127370    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 22:06:22.132012 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:22.131990    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq4bk"]
Sep 17 22:06:22.280674 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:22.280646    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/adb5df7d-ada3-4f42-ab23-bcce9b987845-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq4bk\" (UID: \"adb5df7d-ada3-4f42-ab23-bcce9b987845\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq4bk"
Sep 17 22:06:22.280843 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:22.280706    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dsd9\" (UniqueName: \"kubernetes.io/projected/adb5df7d-ada3-4f42-ab23-bcce9b987845-kube-api-access-5dsd9\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq4bk\" (UID: \"adb5df7d-ada3-4f42-ab23-bcce9b987845\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq4bk"
Sep 17 22:06:22.280843 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:22.280820    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/adb5df7d-ada3-4f42-ab23-bcce9b987845-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq4bk\" (UID: \"adb5df7d-ada3-4f42-ab23-bcce9b987845\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq4bk"
Sep 17 22:06:22.381858 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:22.381833    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/adb5df7d-ada3-4f42-ab23-bcce9b987845-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq4bk\" (UID: \"adb5df7d-ada3-4f42-ab23-bcce9b987845\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq4bk"
Sep 17 22:06:22.381985 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:22.381875    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/adb5df7d-ada3-4f42-ab23-bcce9b987845-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq4bk\" (UID: \"adb5df7d-ada3-4f42-ab23-bcce9b987845\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq4bk"
Sep 17 22:06:22.381985 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:22.381914    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dsd9\" (UniqueName: \"kubernetes.io/projected/adb5df7d-ada3-4f42-ab23-bcce9b987845-kube-api-access-5dsd9\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq4bk\" (UID: \"adb5df7d-ada3-4f42-ab23-bcce9b987845\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq4bk"
Sep 17 22:06:22.382322 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:22.382299    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/adb5df7d-ada3-4f42-ab23-bcce9b987845-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq4bk\" (UID: \"adb5df7d-ada3-4f42-ab23-bcce9b987845\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq4bk"
Sep 17 22:06:22.382322 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:22.382315    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/adb5df7d-ada3-4f42-ab23-bcce9b987845-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq4bk\" (UID: \"adb5df7d-ada3-4f42-ab23-bcce9b987845\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq4bk"
Sep 17 22:06:22.390495 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:22.390478    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dsd9\" (UniqueName: \"kubernetes.io/projected/adb5df7d-ada3-4f42-ab23-bcce9b987845-kube-api-access-5dsd9\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq4bk\" (UID: \"adb5df7d-ada3-4f42-ab23-bcce9b987845\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq4bk"
Sep 17 22:06:22.435392 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:22.435364    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq4bk"
Sep 17 22:06:22.706130 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:22.706106    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq4bk"]
Sep 17 22:06:22.708360 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:06:22.708326    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadb5df7d_ada3_4f42_ab23_bcce9b987845.slice/crio-804e25b758c1fd94ffc30247c59ff064eb23ff0fbaf78e2f74cbac16bc6de225 WatchSource:0}: Error finding container 804e25b758c1fd94ffc30247c59ff064eb23ff0fbaf78e2f74cbac16bc6de225: Status 404 returned error can't find the container with id 804e25b758c1fd94ffc30247c59ff064eb23ff0fbaf78e2f74cbac16bc6de225
Sep 17 22:06:22.967487 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:22.967409    2555 generic.go:356] "Generic (PLEG): container finished" podID="adb5df7d-ada3-4f42-ab23-bcce9b987845" containerID="1598302ae7f28e397238855270f9fb5ac4aceb22ed3916c3bea53c90e4f63125" exitCode=0
Sep 17 22:06:22.967620 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:22.967495    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq4bk" event={"ID":"adb5df7d-ada3-4f42-ab23-bcce9b987845","Type":"ContainerDied","Data":"1598302ae7f28e397238855270f9fb5ac4aceb22ed3916c3bea53c90e4f63125"}
Sep 17 22:06:22.967620 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:22.967553    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq4bk" event={"ID":"adb5df7d-ada3-4f42-ab23-bcce9b987845","Type":"ContainerStarted","Data":"804e25b758c1fd94ffc30247c59ff064eb23ff0fbaf78e2f74cbac16bc6de225"}
Sep 17 22:06:24.363861 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:24.363812    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:06:24.425429 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:24.425394    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:06:24.979007 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:24.978976    2555 generic.go:356] "Generic (PLEG): container finished" podID="adb5df7d-ada3-4f42-ab23-bcce9b987845" containerID="7205fea73ce18bf9a7b5ea665bc53b12cd9c16af65738206dfa3fe4f99e5694d" exitCode=0
Sep 17 22:06:24.979140 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:24.979065    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq4bk" event={"ID":"adb5df7d-ada3-4f42-ab23-bcce9b987845","Type":"ContainerDied","Data":"7205fea73ce18bf9a7b5ea665bc53b12cd9c16af65738206dfa3fe4f99e5694d"}
Sep 17 22:06:25.986074 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:25.986040    2555 generic.go:356] "Generic (PLEG): container finished" podID="adb5df7d-ada3-4f42-ab23-bcce9b987845" containerID="8e0a3f5f6868a53aa5d556b36dffca55b1c82b4a1e4b14fa6913b956b2036e0e" exitCode=0
Sep 17 22:06:25.986468 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:25.986137    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq4bk" event={"ID":"adb5df7d-ada3-4f42-ab23-bcce9b987845","Type":"ContainerDied","Data":"8e0a3f5f6868a53aa5d556b36dffca55b1c82b4a1e4b14fa6913b956b2036e0e"}
Sep 17 22:06:27.117271 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:27.117248    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq4bk"
Sep 17 22:06:27.182296 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:27.182265    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:06:27.223340 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:27.223311    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/adb5df7d-ada3-4f42-ab23-bcce9b987845-util\") pod \"adb5df7d-ada3-4f42-ab23-bcce9b987845\" (UID: \"adb5df7d-ada3-4f42-ab23-bcce9b987845\") "
Sep 17 22:06:27.223468 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:27.223356    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/adb5df7d-ada3-4f42-ab23-bcce9b987845-bundle\") pod \"adb5df7d-ada3-4f42-ab23-bcce9b987845\" (UID: \"adb5df7d-ada3-4f42-ab23-bcce9b987845\") "
Sep 17 22:06:27.223468 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:27.223400    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dsd9\" (UniqueName: \"kubernetes.io/projected/adb5df7d-ada3-4f42-ab23-bcce9b987845-kube-api-access-5dsd9\") pod \"adb5df7d-ada3-4f42-ab23-bcce9b987845\" (UID: \"adb5df7d-ada3-4f42-ab23-bcce9b987845\") "
Sep 17 22:06:27.223778 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:27.223750    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb5df7d-ada3-4f42-ab23-bcce9b987845-bundle" (OuterVolumeSpecName: "bundle") pod "adb5df7d-ada3-4f42-ab23-bcce9b987845" (UID: "adb5df7d-ada3-4f42-ab23-bcce9b987845"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:06:27.225519 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:27.225498    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adb5df7d-ada3-4f42-ab23-bcce9b987845-kube-api-access-5dsd9" (OuterVolumeSpecName: "kube-api-access-5dsd9") pod "adb5df7d-ada3-4f42-ab23-bcce9b987845" (UID: "adb5df7d-ada3-4f42-ab23-bcce9b987845"). InnerVolumeSpecName "kube-api-access-5dsd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:06:27.228948 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:27.228912    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb5df7d-ada3-4f42-ab23-bcce9b987845-util" (OuterVolumeSpecName: "util") pod "adb5df7d-ada3-4f42-ab23-bcce9b987845" (UID: "adb5df7d-ada3-4f42-ab23-bcce9b987845"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:06:27.324002 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:27.323941    2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/adb5df7d-ada3-4f42-ab23-bcce9b987845-util\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:06:27.324002 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:27.323964    2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/adb5df7d-ada3-4f42-ab23-bcce9b987845-bundle\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:06:27.324002 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:27.323977    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dsd9\" (UniqueName: \"kubernetes.io/projected/adb5df7d-ada3-4f42-ab23-bcce9b987845-kube-api-access-5dsd9\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:06:27.997365 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:27.997329    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq4bk" event={"ID":"adb5df7d-ada3-4f42-ab23-bcce9b987845","Type":"ContainerDied","Data":"804e25b758c1fd94ffc30247c59ff064eb23ff0fbaf78e2f74cbac16bc6de225"}
Sep 17 22:06:27.997538 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:27.997375    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="804e25b758c1fd94ffc30247c59ff064eb23ff0fbaf78e2f74cbac16bc6de225"
Sep 17 22:06:27.997538 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:27.997343    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq4bk"
Sep 17 22:06:33.315852 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:33.315817    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:06:33.412345 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:33.412313    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:06:44.539939 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:44.539903    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:06:48.847252 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:48.847223    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:06:48.951567 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:48.951527    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:06:49.163573 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:49.163544    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kdwl9"]
Sep 17 22:06:49.163876 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:49.163864    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adb5df7d-ada3-4f42-ab23-bcce9b987845" containerName="extract"
Sep 17 22:06:49.163922 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:49.163877    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="adb5df7d-ada3-4f42-ab23-bcce9b987845" containerName="extract"
Sep 17 22:06:49.163922 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:49.163888    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adb5df7d-ada3-4f42-ab23-bcce9b987845" containerName="pull"
Sep 17 22:06:49.163922 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:49.163893    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="adb5df7d-ada3-4f42-ab23-bcce9b987845" containerName="pull"
Sep 17 22:06:49.163922 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:49.163909    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adb5df7d-ada3-4f42-ab23-bcce9b987845" containerName="util"
Sep 17 22:06:49.163922 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:49.163914    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="adb5df7d-ada3-4f42-ab23-bcce9b987845" containerName="util"
Sep 17 22:06:49.164070 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:49.163965    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="adb5df7d-ada3-4f42-ab23-bcce9b987845" containerName="extract"
Sep 17 22:06:49.167364 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:49.167345    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kdwl9"
Sep 17 22:06:49.169717 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:49.169694    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 17 22:06:49.169717 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:49.169707    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 17 22:06:49.169897 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:49.169726    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 17 22:06:49.169897 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:49.169705    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 17 22:06:49.170023 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:49.169969    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-q7nnf\""
Sep 17 22:06:49.171584 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:49.171562    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:06:49.179748 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:49.179727    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kdwl9"]
Sep 17 22:06:49.283530 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:49.283497    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:06:49.291648 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:49.291627    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/83ff6a8e-b9d4-42d3-9891-728ac2eed63a-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-kdwl9\" (UID: \"83ff6a8e-b9d4-42d3-9891-728ac2eed63a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kdwl9"
Sep 17 22:06:49.291713 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:49.291656    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/83ff6a8e-b9d4-42d3-9891-728ac2eed63a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-kdwl9\" (UID: \"83ff6a8e-b9d4-42d3-9891-728ac2eed63a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kdwl9"
Sep 17 22:06:49.291713 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:49.291681    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/83ff6a8e-b9d4-42d3-9891-728ac2eed63a-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-kdwl9\" (UID: \"83ff6a8e-b9d4-42d3-9891-728ac2eed63a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kdwl9"
Sep 17 22:06:49.291780 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:49.291726    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/83ff6a8e-b9d4-42d3-9891-728ac2eed63a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-kdwl9\" (UID: \"83ff6a8e-b9d4-42d3-9891-728ac2eed63a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kdwl9"
Sep 17 22:06:49.291780 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:49.291772    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sdvp\" (UniqueName: \"kubernetes.io/projected/83ff6a8e-b9d4-42d3-9891-728ac2eed63a-kube-api-access-9sdvp\") pod \"istiod-openshift-gateway-7b7995d5cd-kdwl9\" (UID: \"83ff6a8e-b9d4-42d3-9891-728ac2eed63a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kdwl9"
Sep 17 22:06:49.291872 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:49.291856    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83ff6a8e-b9d4-42d3-9891-728ac2eed63a-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-kdwl9\" (UID: \"83ff6a8e-b9d4-42d3-9891-728ac2eed63a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kdwl9"
Sep 17 22:06:49.291907 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:49.291883    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/83ff6a8e-b9d4-42d3-9891-728ac2eed63a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-kdwl9\" (UID: \"83ff6a8e-b9d4-42d3-9891-728ac2eed63a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kdwl9"
Sep 17 22:06:49.392797 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:49.392762    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83ff6a8e-b9d4-42d3-9891-728ac2eed63a-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-kdwl9\" (UID: \"83ff6a8e-b9d4-42d3-9891-728ac2eed63a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kdwl9"
Sep 17 22:06:49.392939 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:49.392805    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/83ff6a8e-b9d4-42d3-9891-728ac2eed63a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-kdwl9\" (UID: \"83ff6a8e-b9d4-42d3-9891-728ac2eed63a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kdwl9"
Sep 17 22:06:49.392939 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:49.392850    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/83ff6a8e-b9d4-42d3-9891-728ac2eed63a-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-kdwl9\" (UID: \"83ff6a8e-b9d4-42d3-9891-728ac2eed63a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kdwl9"
Sep 17 22:06:49.392939 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:49.392882    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/83ff6a8e-b9d4-42d3-9891-728ac2eed63a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-kdwl9\" (UID: \"83ff6a8e-b9d4-42d3-9891-728ac2eed63a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kdwl9"
Sep 17 22:06:49.392939 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:49.392920    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/83ff6a8e-b9d4-42d3-9891-728ac2eed63a-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-kdwl9\" (UID: \"83ff6a8e-b9d4-42d3-9891-728ac2eed63a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kdwl9"
Sep 17 22:06:49.393144 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:49.392950    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/83ff6a8e-b9d4-42d3-9891-728ac2eed63a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-kdwl9\" (UID: \"83ff6a8e-b9d4-42d3-9891-728ac2eed63a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kdwl9"
Sep 17 22:06:49.393144 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:49.393001    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sdvp\" (UniqueName: \"kubernetes.io/projected/83ff6a8e-b9d4-42d3-9891-728ac2eed63a-kube-api-access-9sdvp\") pod \"istiod-openshift-gateway-7b7995d5cd-kdwl9\" (UID: \"83ff6a8e-b9d4-42d3-9891-728ac2eed63a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kdwl9"
Sep 17 22:06:49.393590 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:49.393560    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/83ff6a8e-b9d4-42d3-9891-728ac2eed63a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-kdwl9\" (UID: \"83ff6a8e-b9d4-42d3-9891-728ac2eed63a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kdwl9"
Sep 17 22:06:49.395366 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:49.395336    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/83ff6a8e-b9d4-42d3-9891-728ac2eed63a-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-kdwl9\" (UID: \"83ff6a8e-b9d4-42d3-9891-728ac2eed63a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kdwl9"
Sep 17 22:06:49.395481 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:49.395445    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/83ff6a8e-b9d4-42d3-9891-728ac2eed63a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-kdwl9\" (UID: \"83ff6a8e-b9d4-42d3-9891-728ac2eed63a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kdwl9"
Sep 17 22:06:49.395553 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:49.395501    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/83ff6a8e-b9d4-42d3-9891-728ac2eed63a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-kdwl9\" (UID: \"83ff6a8e-b9d4-42d3-9891-728ac2eed63a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kdwl9"
Sep 17 22:06:49.395611 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:49.395575    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/83ff6a8e-b9d4-42d3-9891-728ac2eed63a-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-kdwl9\" (UID: \"83ff6a8e-b9d4-42d3-9891-728ac2eed63a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kdwl9"
Sep 17 22:06:49.400366 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:49.400344    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83ff6a8e-b9d4-42d3-9891-728ac2eed63a-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-kdwl9\" (UID: \"83ff6a8e-b9d4-42d3-9891-728ac2eed63a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kdwl9"
Sep 17 22:06:49.400558 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:49.400537    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sdvp\" (UniqueName: \"kubernetes.io/projected/83ff6a8e-b9d4-42d3-9891-728ac2eed63a-kube-api-access-9sdvp\") pod \"istiod-openshift-gateway-7b7995d5cd-kdwl9\" (UID: \"83ff6a8e-b9d4-42d3-9891-728ac2eed63a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kdwl9"
Sep 17 22:06:49.476974 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:49.476922    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kdwl9"
Sep 17 22:06:49.754917 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:49.754839    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kdwl9"]
Sep 17 22:06:49.759297 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:06:49.759263    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83ff6a8e_b9d4_42d3_9891_728ac2eed63a.slice/crio-e16b520ee03cca21fe2e47e8ea1bfad079b6840eaa068df9130dc7c3392f5ef4 WatchSource:0}: Error finding container e16b520ee03cca21fe2e47e8ea1bfad079b6840eaa068df9130dc7c3392f5ef4: Status 404 returned error can't find the container with id e16b520ee03cca21fe2e47e8ea1bfad079b6840eaa068df9130dc7c3392f5ef4
Sep 17 22:06:50.099393 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:50.099297    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kdwl9" event={"ID":"83ff6a8e-b9d4-42d3-9891-728ac2eed63a","Type":"ContainerStarted","Data":"e16b520ee03cca21fe2e47e8ea1bfad079b6840eaa068df9130dc7c3392f5ef4"}
Sep 17 22:06:50.221206 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:50.221155    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:06:51.179587 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:51.179552    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:06:52.440193 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:52.440142    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 22:06:52.440457 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:52.440244    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 22:06:53.117773 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:53.117739    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kdwl9" event={"ID":"83ff6a8e-b9d4-42d3-9891-728ac2eed63a","Type":"ContainerStarted","Data":"5988b1e1fe9bec90a639eec14a6a157f84f2c093e86de21b22d426c1d9de9b4f"}
Sep 17 22:06:53.117977 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:53.117880    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kdwl9"
Sep 17 22:06:53.119637 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:53.119610    2555 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-kdwl9 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 22:06:53.119752 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:53.119657    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kdwl9" podUID="83ff6a8e-b9d4-42d3-9891-728ac2eed63a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:06:53.137098 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:53.137045    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kdwl9" podStartSLOduration=1.458563013 podStartE2EDuration="4.137031781s" totalImagesPullingTime="2.678468768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:06:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:06:53.135698687 +0000 UTC m=+514.581743961" watchObservedRunningTime="2026-09-17 22:06:53.137031781 +0000 UTC m=+514.583077052"
Sep 17 22:06:53.464423 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:53.464365    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:06:53.563346 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:53.563303    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:06:54.123351 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:54.123324    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kdwl9"
Sep 17 22:06:55.898483 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:55.898452    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8"]
Sep 17 22:06:55.906352 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:55.906329    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8"
Sep 17 22:06:55.909105 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:55.909083    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-4pfg8\""
Sep 17 22:06:55.915805 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:55.915786    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8"]
Sep 17 22:06:55.934652 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:55.934614    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:06:55.938939 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:55.938914    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/296255f9-9ebf-49bb-8fcb-be7c8b392d12-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-2q5j8\" (UID: \"296255f9-9ebf-49bb-8fcb-be7c8b392d12\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8"
Sep 17 22:06:55.939048 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:55.938948    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk4wg\" (UniqueName: \"kubernetes.io/projected/296255f9-9ebf-49bb-8fcb-be7c8b392d12-kube-api-access-bk4wg\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-2q5j8\" (UID: \"296255f9-9ebf-49bb-8fcb-be7c8b392d12\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8"
Sep 17 22:06:55.939048 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:55.939012    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/296255f9-9ebf-49bb-8fcb-be7c8b392d12-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-2q5j8\" (UID: \"296255f9-9ebf-49bb-8fcb-be7c8b392d12\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8"
Sep 17 22:06:55.939048 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:55.939042    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/296255f9-9ebf-49bb-8fcb-be7c8b392d12-workload-certs\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-2q5j8\" (UID: \"296255f9-9ebf-49bb-8fcb-be7c8b392d12\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8"
Sep 17 22:06:55.939223 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:55.939102    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/296255f9-9ebf-49bb-8fcb-be7c8b392d12-credential-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-2q5j8\" (UID: \"296255f9-9ebf-49bb-8fcb-be7c8b392d12\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8"
Sep 17 22:06:55.939223 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:55.939130    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/296255f9-9ebf-49bb-8fcb-be7c8b392d12-istio-data\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-2q5j8\" (UID: \"296255f9-9ebf-49bb-8fcb-be7c8b392d12\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8"
Sep 17 22:06:55.939223 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:55.939157    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/296255f9-9ebf-49bb-8fcb-be7c8b392d12-istio-envoy\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-2q5j8\" (UID: \"296255f9-9ebf-49bb-8fcb-be7c8b392d12\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8"
Sep 17 22:06:55.939223 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:55.939172    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/296255f9-9ebf-49bb-8fcb-be7c8b392d12-workload-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-2q5j8\" (UID: \"296255f9-9ebf-49bb-8fcb-be7c8b392d12\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8"
Sep 17 22:06:55.939355 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:55.939258    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/296255f9-9ebf-49bb-8fcb-be7c8b392d12-istio-token\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-2q5j8\" (UID: \"296255f9-9ebf-49bb-8fcb-be7c8b392d12\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8"
Sep 17 22:06:55.999811 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:55.999783    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:06:56.039942 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:56.039916    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/296255f9-9ebf-49bb-8fcb-be7c8b392d12-credential-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-2q5j8\" (UID: \"296255f9-9ebf-49bb-8fcb-be7c8b392d12\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8"
Sep 17 22:06:56.039942 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:56.039945    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/296255f9-9ebf-49bb-8fcb-be7c8b392d12-istio-data\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-2q5j8\" (UID: \"296255f9-9ebf-49bb-8fcb-be7c8b392d12\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8"
Sep 17 22:06:56.040152 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:56.039964    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/296255f9-9ebf-49bb-8fcb-be7c8b392d12-istio-envoy\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-2q5j8\" (UID: \"296255f9-9ebf-49bb-8fcb-be7c8b392d12\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8"
Sep 17 22:06:56.040152 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:56.039980    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/296255f9-9ebf-49bb-8fcb-be7c8b392d12-workload-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-2q5j8\" (UID: \"296255f9-9ebf-49bb-8fcb-be7c8b392d12\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8"
Sep 17 22:06:56.040152 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:56.040008    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/296255f9-9ebf-49bb-8fcb-be7c8b392d12-istio-token\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-2q5j8\" (UID: \"296255f9-9ebf-49bb-8fcb-be7c8b392d12\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8"
Sep 17 22:06:56.040152 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:56.040033    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/296255f9-9ebf-49bb-8fcb-be7c8b392d12-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-2q5j8\" (UID: \"296255f9-9ebf-49bb-8fcb-be7c8b392d12\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8"
Sep 17 22:06:56.040152 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:56.040096    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk4wg\" (UniqueName: \"kubernetes.io/projected/296255f9-9ebf-49bb-8fcb-be7c8b392d12-kube-api-access-bk4wg\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-2q5j8\" (UID: \"296255f9-9ebf-49bb-8fcb-be7c8b392d12\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8"
Sep 17 22:06:56.040373 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:56.040154    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/296255f9-9ebf-49bb-8fcb-be7c8b392d12-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-2q5j8\" (UID: \"296255f9-9ebf-49bb-8fcb-be7c8b392d12\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8"
Sep 17 22:06:56.040373 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:56.040214    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/296255f9-9ebf-49bb-8fcb-be7c8b392d12-workload-certs\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-2q5j8\" (UID: \"296255f9-9ebf-49bb-8fcb-be7c8b392d12\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8"
Sep 17 22:06:56.040466 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:56.040445    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/296255f9-9ebf-49bb-8fcb-be7c8b392d12-istio-data\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-2q5j8\" (UID: \"296255f9-9ebf-49bb-8fcb-be7c8b392d12\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8"
Sep 17 22:06:56.040583 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:56.040560    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/296255f9-9ebf-49bb-8fcb-be7c8b392d12-workload-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-2q5j8\" (UID: \"296255f9-9ebf-49bb-8fcb-be7c8b392d12\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8"
Sep 17 22:06:56.040647 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:56.040590    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/296255f9-9ebf-49bb-8fcb-be7c8b392d12-credential-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-2q5j8\" (UID: \"296255f9-9ebf-49bb-8fcb-be7c8b392d12\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8"
Sep 17 22:06:56.040756 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:56.040736    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/296255f9-9ebf-49bb-8fcb-be7c8b392d12-workload-certs\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-2q5j8\" (UID: \"296255f9-9ebf-49bb-8fcb-be7c8b392d12\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8"
Sep 17 22:06:56.040872 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:56.040853    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/296255f9-9ebf-49bb-8fcb-be7c8b392d12-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-2q5j8\" (UID: \"296255f9-9ebf-49bb-8fcb-be7c8b392d12\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8"
Sep 17 22:06:56.042193 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:56.042163    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/296255f9-9ebf-49bb-8fcb-be7c8b392d12-istio-envoy\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-2q5j8\" (UID: \"296255f9-9ebf-49bb-8fcb-be7c8b392d12\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8"
Sep 17 22:06:56.042580 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:56.042561    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/296255f9-9ebf-49bb-8fcb-be7c8b392d12-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-2q5j8\" (UID: \"296255f9-9ebf-49bb-8fcb-be7c8b392d12\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8"
Sep 17 22:06:56.047690 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:56.047667    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/296255f9-9ebf-49bb-8fcb-be7c8b392d12-istio-token\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-2q5j8\" (UID: \"296255f9-9ebf-49bb-8fcb-be7c8b392d12\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8"
Sep 17 22:06:56.047971 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:56.047952    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk4wg\" (UniqueName: \"kubernetes.io/projected/296255f9-9ebf-49bb-8fcb-be7c8b392d12-kube-api-access-bk4wg\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-2q5j8\" (UID: \"296255f9-9ebf-49bb-8fcb-be7c8b392d12\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8"
Sep 17 22:06:56.219046 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:56.218957    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8"
Sep 17 22:06:56.493460 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:56.493435    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8"]
Sep 17 22:06:56.495539 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:06:56.495508    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod296255f9_9ebf_49bb_8fcb_be7c8b392d12.slice/crio-d84e462e4873b4e2f35fad1c8b3cb65d817c63742221ee527ec54faaad81d18d WatchSource:0}: Error finding container d84e462e4873b4e2f35fad1c8b3cb65d817c63742221ee527ec54faaad81d18d: Status 404 returned error can't find the container with id d84e462e4873b4e2f35fad1c8b3cb65d817c63742221ee527ec54faaad81d18d
Sep 17 22:06:57.137236 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:57.137200    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8" event={"ID":"296255f9-9ebf-49bb-8fcb-be7c8b392d12","Type":"ContainerStarted","Data":"d84e462e4873b4e2f35fad1c8b3cb65d817c63742221ee527ec54faaad81d18d"}
Sep 17 22:06:58.940028 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:58.939986    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:06:59.180245 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:59.180211    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 22:06:59.180315 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:59.180290    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 22:06:59.180356 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:06:59.180324    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 22:07:00.152354 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:00.152316    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8" event={"ID":"296255f9-9ebf-49bb-8fcb-be7c8b392d12","Type":"ContainerStarted","Data":"a58e13e1433d9e6650dfdc7b28263fd7431fa01b2f65b7c57273df6c6e607102"}
Sep 17 22:07:00.175136 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:00.175087    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8" podStartSLOduration=2.492642946 podStartE2EDuration="5.175067965s" totalImagesPullingTime="2.682425019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:06:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:07:00.173707412 +0000 UTC m=+521.619752684" watchObservedRunningTime="2026-09-17 22:07:00.175067965 +0000 UTC m=+521.621113236"
Sep 17 22:07:00.219653 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:00.219626    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8"
Sep 17 22:07:00.224389 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:00.224370    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8"
Sep 17 22:07:01.157417 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:01.157387    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8"
Sep 17 22:07:01.158343 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:01.158325    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8"
Sep 17 22:07:01.414100 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:01.414026    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:01.516960 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:01.516934    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:04.648204 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:04.648114    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hmg4f"]
Sep 17 22:07:04.651548 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:04.651532    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hmg4f"
Sep 17 22:07:04.653750 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:04.653726    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 22:07:04.653865 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:04.653733    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 22:07:04.654518 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:04.654504    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-h4r5v\""
Sep 17 22:07:04.658984 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:04.658950    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hmg4f"]
Sep 17 22:07:04.718497 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:04.718469    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09c747bc-3b24-4434-ab0a-e69dd6f1ee08-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hmg4f\" (UID: \"09c747bc-3b24-4434-ab0a-e69dd6f1ee08\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hmg4f"
Sep 17 22:07:04.718655 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:04.718504    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09c747bc-3b24-4434-ab0a-e69dd6f1ee08-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hmg4f\" (UID: \"09c747bc-3b24-4434-ab0a-e69dd6f1ee08\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hmg4f"
Sep 17 22:07:04.718655 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:04.718532    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jldwr\" (UniqueName: \"kubernetes.io/projected/09c747bc-3b24-4434-ab0a-e69dd6f1ee08-kube-api-access-jldwr\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hmg4f\" (UID: \"09c747bc-3b24-4434-ab0a-e69dd6f1ee08\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hmg4f"
Sep 17 22:07:04.759243 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:04.759218    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5spng"]
Sep 17 22:07:04.762530 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:04.762514    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5spng"
Sep 17 22:07:04.774149 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:04.774123    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5spng"]
Sep 17 22:07:04.819447 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:04.819414    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7mzf\" (UniqueName: \"kubernetes.io/projected/efaa0f2e-64de-4ecb-8bc3-c057102180c2-kube-api-access-w7mzf\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5spng\" (UID: \"efaa0f2e-64de-4ecb-8bc3-c057102180c2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5spng"
Sep 17 22:07:04.819577 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:04.819457    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jldwr\" (UniqueName: \"kubernetes.io/projected/09c747bc-3b24-4434-ab0a-e69dd6f1ee08-kube-api-access-jldwr\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hmg4f\" (UID: \"09c747bc-3b24-4434-ab0a-e69dd6f1ee08\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hmg4f"
Sep 17 22:07:04.819577 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:04.819548    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/efaa0f2e-64de-4ecb-8bc3-c057102180c2-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5spng\" (UID: \"efaa0f2e-64de-4ecb-8bc3-c057102180c2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5spng"
Sep 17 22:07:04.819698 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:04.819579    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09c747bc-3b24-4434-ab0a-e69dd6f1ee08-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hmg4f\" (UID: \"09c747bc-3b24-4434-ab0a-e69dd6f1ee08\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hmg4f"
Sep 17 22:07:04.819698 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:04.819603    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09c747bc-3b24-4434-ab0a-e69dd6f1ee08-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hmg4f\" (UID: \"09c747bc-3b24-4434-ab0a-e69dd6f1ee08\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hmg4f"
Sep 17 22:07:04.819698 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:04.819618    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/efaa0f2e-64de-4ecb-8bc3-c057102180c2-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5spng\" (UID: \"efaa0f2e-64de-4ecb-8bc3-c057102180c2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5spng"
Sep 17 22:07:04.819904 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:04.819890    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09c747bc-3b24-4434-ab0a-e69dd6f1ee08-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hmg4f\" (UID: \"09c747bc-3b24-4434-ab0a-e69dd6f1ee08\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hmg4f"
Sep 17 22:07:04.819942 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:04.819927    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09c747bc-3b24-4434-ab0a-e69dd6f1ee08-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hmg4f\" (UID: \"09c747bc-3b24-4434-ab0a-e69dd6f1ee08\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hmg4f"
Sep 17 22:07:04.828474 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:04.828455    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jldwr\" (UniqueName: \"kubernetes.io/projected/09c747bc-3b24-4434-ab0a-e69dd6f1ee08-kube-api-access-jldwr\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hmg4f\" (UID: \"09c747bc-3b24-4434-ab0a-e69dd6f1ee08\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hmg4f"
Sep 17 22:07:04.844906 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:04.844883    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ztvtj"]
Sep 17 22:07:04.848281 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:04.848267    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ztvtj"
Sep 17 22:07:04.855736 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:04.855715    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ztvtj"]
Sep 17 22:07:04.920797 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:04.920715    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/efaa0f2e-64de-4ecb-8bc3-c057102180c2-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5spng\" (UID: \"efaa0f2e-64de-4ecb-8bc3-c057102180c2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5spng"
Sep 17 22:07:04.920797 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:04.920754    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7mzf\" (UniqueName: \"kubernetes.io/projected/efaa0f2e-64de-4ecb-8bc3-c057102180c2-kube-api-access-w7mzf\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5spng\" (UID: \"efaa0f2e-64de-4ecb-8bc3-c057102180c2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5spng"
Sep 17 22:07:04.920996 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:04.920804    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94271884-d40d-4f07-8680-b56912e0badd-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ztvtj\" (UID: \"94271884-d40d-4f07-8680-b56912e0badd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ztvtj"
Sep 17 22:07:04.920996 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:04.920842    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94271884-d40d-4f07-8680-b56912e0badd-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ztvtj\" (UID: \"94271884-d40d-4f07-8680-b56912e0badd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ztvtj"
Sep 17 22:07:04.920996 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:04.920869    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbwz8\" (UniqueName: \"kubernetes.io/projected/94271884-d40d-4f07-8680-b56912e0badd-kube-api-access-lbwz8\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ztvtj\" (UID: \"94271884-d40d-4f07-8680-b56912e0badd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ztvtj"
Sep 17 22:07:04.920996 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:04.920896    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/efaa0f2e-64de-4ecb-8bc3-c057102180c2-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5spng\" (UID: \"efaa0f2e-64de-4ecb-8bc3-c057102180c2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5spng"
Sep 17 22:07:04.921243 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:04.921094    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/efaa0f2e-64de-4ecb-8bc3-c057102180c2-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5spng\" (UID: \"efaa0f2e-64de-4ecb-8bc3-c057102180c2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5spng"
Sep 17 22:07:04.921243 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:04.921207    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/efaa0f2e-64de-4ecb-8bc3-c057102180c2-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5spng\" (UID: \"efaa0f2e-64de-4ecb-8bc3-c057102180c2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5spng"
Sep 17 22:07:04.928877 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:04.928860    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7mzf\" (UniqueName: \"kubernetes.io/projected/efaa0f2e-64de-4ecb-8bc3-c057102180c2-kube-api-access-w7mzf\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5spng\" (UID: \"efaa0f2e-64de-4ecb-8bc3-c057102180c2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5spng"
Sep 17 22:07:04.945085 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:04.945055    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crdw6x"]
Sep 17 22:07:04.948725 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:04.948708    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crdw6x"
Sep 17 22:07:04.956803 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:04.956769    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crdw6x"]
Sep 17 22:07:04.960917 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:04.960898    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hmg4f"
Sep 17 22:07:05.021901 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:05.021821    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ac5ed46-ef53-4750-b80c-de7d7209bfc4-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crdw6x\" (UID: \"8ac5ed46-ef53-4750-b80c-de7d7209bfc4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crdw6x"
Sep 17 22:07:05.021901 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:05.021874    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94271884-d40d-4f07-8680-b56912e0badd-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ztvtj\" (UID: \"94271884-d40d-4f07-8680-b56912e0badd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ztvtj"
Sep 17 22:07:05.022054 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:05.021939    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94271884-d40d-4f07-8680-b56912e0badd-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ztvtj\" (UID: \"94271884-d40d-4f07-8680-b56912e0badd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ztvtj"
Sep 17 22:07:05.022054 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:05.021971    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbwz8\" (UniqueName: \"kubernetes.io/projected/94271884-d40d-4f07-8680-b56912e0badd-kube-api-access-lbwz8\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ztvtj\" (UID: \"94271884-d40d-4f07-8680-b56912e0badd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ztvtj"
Sep 17 22:07:05.022054 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:05.022020    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ac5ed46-ef53-4750-b80c-de7d7209bfc4-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crdw6x\" (UID: \"8ac5ed46-ef53-4750-b80c-de7d7209bfc4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crdw6x"
Sep 17 22:07:05.022054 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:05.022050    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2mz4\" (UniqueName: \"kubernetes.io/projected/8ac5ed46-ef53-4750-b80c-de7d7209bfc4-kube-api-access-x2mz4\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crdw6x\" (UID: \"8ac5ed46-ef53-4750-b80c-de7d7209bfc4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crdw6x"
Sep 17 22:07:05.022558 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:05.022540    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94271884-d40d-4f07-8680-b56912e0badd-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ztvtj\" (UID: \"94271884-d40d-4f07-8680-b56912e0badd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ztvtj"
Sep 17 22:07:05.022778 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:05.022758    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94271884-d40d-4f07-8680-b56912e0badd-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ztvtj\" (UID: \"94271884-d40d-4f07-8680-b56912e0badd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ztvtj"
Sep 17 22:07:05.030561 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:05.030521    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbwz8\" (UniqueName: \"kubernetes.io/projected/94271884-d40d-4f07-8680-b56912e0badd-kube-api-access-lbwz8\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ztvtj\" (UID: \"94271884-d40d-4f07-8680-b56912e0badd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ztvtj"
Sep 17 22:07:05.073999 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:05.073971    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5spng"
Sep 17 22:07:05.123078 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:05.123025    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ac5ed46-ef53-4750-b80c-de7d7209bfc4-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crdw6x\" (UID: \"8ac5ed46-ef53-4750-b80c-de7d7209bfc4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crdw6x"
Sep 17 22:07:05.123214 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:05.123081    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2mz4\" (UniqueName: \"kubernetes.io/projected/8ac5ed46-ef53-4750-b80c-de7d7209bfc4-kube-api-access-x2mz4\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crdw6x\" (UID: \"8ac5ed46-ef53-4750-b80c-de7d7209bfc4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crdw6x"
Sep 17 22:07:05.123214 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:05.123176    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ac5ed46-ef53-4750-b80c-de7d7209bfc4-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crdw6x\" (UID: \"8ac5ed46-ef53-4750-b80c-de7d7209bfc4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crdw6x"
Sep 17 22:07:05.123700 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:05.123676    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ac5ed46-ef53-4750-b80c-de7d7209bfc4-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crdw6x\" (UID: \"8ac5ed46-ef53-4750-b80c-de7d7209bfc4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crdw6x"
Sep 17 22:07:05.124035 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:05.123925    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ac5ed46-ef53-4750-b80c-de7d7209bfc4-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crdw6x\" (UID: \"8ac5ed46-ef53-4750-b80c-de7d7209bfc4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crdw6x"
Sep 17 22:07:05.132540 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:05.132517    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2mz4\" (UniqueName: \"kubernetes.io/projected/8ac5ed46-ef53-4750-b80c-de7d7209bfc4-kube-api-access-x2mz4\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crdw6x\" (UID: \"8ac5ed46-ef53-4750-b80c-de7d7209bfc4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crdw6x"
Sep 17 22:07:05.157850 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:05.157832    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ztvtj"
Sep 17 22:07:05.238284 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:05.238261    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hmg4f"]
Sep 17 22:07:05.240072 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:07:05.240048    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09c747bc_3b24_4434_ab0a_e69dd6f1ee08.slice/crio-e424ffd842cfe22c020daa479d2ff325031e4e15b4df8a20cdeb8b138f79c8c0 WatchSource:0}: Error finding container e424ffd842cfe22c020daa479d2ff325031e4e15b4df8a20cdeb8b138f79c8c0: Status 404 returned error can't find the container with id e424ffd842cfe22c020daa479d2ff325031e4e15b4df8a20cdeb8b138f79c8c0
Sep 17 22:07:05.258276 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:05.258255    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crdw6x"
Sep 17 22:07:05.360107 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:05.360080    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5spng"]
Sep 17 22:07:05.409205 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:07:05.409153    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefaa0f2e_64de_4ecb_8bc3_c057102180c2.slice/crio-ac4e461ce5008021491076716ccc47027834b833b3d5175082a5a209ed6f8868 WatchSource:0}: Error finding container ac4e461ce5008021491076716ccc47027834b833b3d5175082a5a209ed6f8868: Status 404 returned error can't find the container with id ac4e461ce5008021491076716ccc47027834b833b3d5175082a5a209ed6f8868
Sep 17 22:07:05.430603 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:05.428833    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ztvtj"]
Sep 17 22:07:05.445361 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:07:05.445338    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94271884_d40d_4f07_8680_b56912e0badd.slice/crio-4903b26082065db45ba96f54ead4ccb1f209055fa1737499e2fc47bef2b4543e WatchSource:0}: Error finding container 4903b26082065db45ba96f54ead4ccb1f209055fa1737499e2fc47bef2b4543e: Status 404 returned error can't find the container with id 4903b26082065db45ba96f54ead4ccb1f209055fa1737499e2fc47bef2b4543e
Sep 17 22:07:05.541972 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:05.541948    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crdw6x"]
Sep 17 22:07:05.576083 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:07:05.576060    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ac5ed46_ef53_4750_b80c_de7d7209bfc4.slice/crio-9c98e99e4ff3bf6b6fec7bbe65e74db36b0cc92089f22e8e261b09a65e8ce340 WatchSource:0}: Error finding container 9c98e99e4ff3bf6b6fec7bbe65e74db36b0cc92089f22e8e261b09a65e8ce340: Status 404 returned error can't find the container with id 9c98e99e4ff3bf6b6fec7bbe65e74db36b0cc92089f22e8e261b09a65e8ce340
Sep 17 22:07:06.181622 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:06.181581    2555 generic.go:356] "Generic (PLEG): container finished" podID="8ac5ed46-ef53-4750-b80c-de7d7209bfc4" containerID="b0fe729d58bdd6f8570d90828c18efa61b03a4fa7cd4bc665200ebce97af6fdd" exitCode=0
Sep 17 22:07:06.182069 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:06.181660    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crdw6x" event={"ID":"8ac5ed46-ef53-4750-b80c-de7d7209bfc4","Type":"ContainerDied","Data":"b0fe729d58bdd6f8570d90828c18efa61b03a4fa7cd4bc665200ebce97af6fdd"}
Sep 17 22:07:06.182069 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:06.181708    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crdw6x" event={"ID":"8ac5ed46-ef53-4750-b80c-de7d7209bfc4","Type":"ContainerStarted","Data":"9c98e99e4ff3bf6b6fec7bbe65e74db36b0cc92089f22e8e261b09a65e8ce340"}
Sep 17 22:07:06.183305 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:06.183273    2555 generic.go:356] "Generic (PLEG): container finished" podID="efaa0f2e-64de-4ecb-8bc3-c057102180c2" containerID="21319ffac12cd4d75214f1ffdfad8bb2e69e68489d59429af1b6d5a15647235d" exitCode=0
Sep 17 22:07:06.183369 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:06.183354    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5spng" event={"ID":"efaa0f2e-64de-4ecb-8bc3-c057102180c2","Type":"ContainerDied","Data":"21319ffac12cd4d75214f1ffdfad8bb2e69e68489d59429af1b6d5a15647235d"}
Sep 17 22:07:06.183426 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:06.183391    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5spng" event={"ID":"efaa0f2e-64de-4ecb-8bc3-c057102180c2","Type":"ContainerStarted","Data":"ac4e461ce5008021491076716ccc47027834b833b3d5175082a5a209ed6f8868"}
Sep 17 22:07:06.184958 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:06.184937    2555 generic.go:356] "Generic (PLEG): container finished" podID="09c747bc-3b24-4434-ab0a-e69dd6f1ee08" containerID="550fa5f0c43c7ad87714d5c052a3f77d510a216adff4c16f126bcc9a67918f58" exitCode=0
Sep 17 22:07:06.185048 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:06.184997    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hmg4f" event={"ID":"09c747bc-3b24-4434-ab0a-e69dd6f1ee08","Type":"ContainerDied","Data":"550fa5f0c43c7ad87714d5c052a3f77d510a216adff4c16f126bcc9a67918f58"}
Sep 17 22:07:06.185106 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:06.185049    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hmg4f" event={"ID":"09c747bc-3b24-4434-ab0a-e69dd6f1ee08","Type":"ContainerStarted","Data":"e424ffd842cfe22c020daa479d2ff325031e4e15b4df8a20cdeb8b138f79c8c0"}
Sep 17 22:07:06.186866 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:06.186843    2555 generic.go:356] "Generic (PLEG): container finished" podID="94271884-d40d-4f07-8680-b56912e0badd" containerID="67db1d21f3efcbd5de6c7394d3d7bb6c4bb3ddab036f84ab70cea0b6a90a1f10" exitCode=0
Sep 17 22:07:06.186952 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:06.186878    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ztvtj" event={"ID":"94271884-d40d-4f07-8680-b56912e0badd","Type":"ContainerDied","Data":"67db1d21f3efcbd5de6c7394d3d7bb6c4bb3ddab036f84ab70cea0b6a90a1f10"}
Sep 17 22:07:06.186952 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:06.186917    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ztvtj" event={"ID":"94271884-d40d-4f07-8680-b56912e0badd","Type":"ContainerStarted","Data":"4903b26082065db45ba96f54ead4ccb1f209055fa1737499e2fc47bef2b4543e"}
Sep 17 22:07:07.195481 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:07.195453    2555 generic.go:356] "Generic (PLEG): container finished" podID="8ac5ed46-ef53-4750-b80c-de7d7209bfc4" containerID="bb172e3fdc7a32e65deecc82fa5710779c1fcc6c25843a7ec552c9af343e622b" exitCode=0
Sep 17 22:07:07.195817 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:07.195620    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crdw6x" event={"ID":"8ac5ed46-ef53-4750-b80c-de7d7209bfc4","Type":"ContainerDied","Data":"bb172e3fdc7a32e65deecc82fa5710779c1fcc6c25843a7ec552c9af343e622b"}
Sep 17 22:07:07.198081 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:07.198057    2555 generic.go:356] "Generic (PLEG): container finished" podID="94271884-d40d-4f07-8680-b56912e0badd" containerID="1a4004532ac8ace2b83eec508241370e9ce8494672ecc711e3ed51810dc36c12" exitCode=0
Sep 17 22:07:07.198163 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:07.198117    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ztvtj" event={"ID":"94271884-d40d-4f07-8680-b56912e0badd","Type":"ContainerDied","Data":"1a4004532ac8ace2b83eec508241370e9ce8494672ecc711e3ed51810dc36c12"}
Sep 17 22:07:08.205328 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:08.205288    2555 generic.go:356] "Generic (PLEG): container finished" podID="94271884-d40d-4f07-8680-b56912e0badd" containerID="a893b03e416a387be1c8fd2c8cc647ab09ec24f630531a09cc638eba2a69cc05" exitCode=0
Sep 17 22:07:08.205755 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:08.205375    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ztvtj" event={"ID":"94271884-d40d-4f07-8680-b56912e0badd","Type":"ContainerDied","Data":"a893b03e416a387be1c8fd2c8cc647ab09ec24f630531a09cc638eba2a69cc05"}
Sep 17 22:07:08.207460 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:08.207434    2555 generic.go:356] "Generic (PLEG): container finished" podID="8ac5ed46-ef53-4750-b80c-de7d7209bfc4" containerID="21ee72e31b9866318bde79feb8b796c8c4ffce40d05186b6d03ee70565b20f46" exitCode=0
Sep 17 22:07:08.207578 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:08.207506    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crdw6x" event={"ID":"8ac5ed46-ef53-4750-b80c-de7d7209bfc4","Type":"ContainerDied","Data":"21ee72e31b9866318bde79feb8b796c8c4ffce40d05186b6d03ee70565b20f46"}
Sep 17 22:07:08.209201 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:08.209170    2555 generic.go:356] "Generic (PLEG): container finished" podID="efaa0f2e-64de-4ecb-8bc3-c057102180c2" containerID="b925c9a00ce4079ad56561b4fed8afc29c4953e22e407c8d2f5a0b96054afeaf" exitCode=0
Sep 17 22:07:08.209558 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:08.209207    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5spng" event={"ID":"efaa0f2e-64de-4ecb-8bc3-c057102180c2","Type":"ContainerDied","Data":"b925c9a00ce4079ad56561b4fed8afc29c4953e22e407c8d2f5a0b96054afeaf"}
Sep 17 22:07:08.211090 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:08.211068    2555 generic.go:356] "Generic (PLEG): container finished" podID="09c747bc-3b24-4434-ab0a-e69dd6f1ee08" containerID="d233b03608d75aecdf8de3d93466b5d2c5d0eefbf60b3384697de74edc5cf2d3" exitCode=0
Sep 17 22:07:08.211169 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:08.211100    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hmg4f" event={"ID":"09c747bc-3b24-4434-ab0a-e69dd6f1ee08","Type":"ContainerDied","Data":"d233b03608d75aecdf8de3d93466b5d2c5d0eefbf60b3384697de74edc5cf2d3"}
Sep 17 22:07:09.134751 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:09.134725    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:09.194847 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:09.194820    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:09.218108 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:09.218083    2555 generic.go:356] "Generic (PLEG): container finished" podID="09c747bc-3b24-4434-ab0a-e69dd6f1ee08" containerID="07e2811d72cde0111ad010fed95c629b84edf64705bb3767387cb3c87a26a27d" exitCode=0
Sep 17 22:07:09.218475 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:09.218160    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hmg4f" event={"ID":"09c747bc-3b24-4434-ab0a-e69dd6f1ee08","Type":"ContainerDied","Data":"07e2811d72cde0111ad010fed95c629b84edf64705bb3767387cb3c87a26a27d"}
Sep 17 22:07:09.220350 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:09.220332    2555 generic.go:356] "Generic (PLEG): container finished" podID="efaa0f2e-64de-4ecb-8bc3-c057102180c2" containerID="7f2ccce6f3259420d05f733145dfe83022b342d9c4782526b05e6780b7325ef2" exitCode=0
Sep 17 22:07:09.220451 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:09.220354    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5spng" event={"ID":"efaa0f2e-64de-4ecb-8bc3-c057102180c2","Type":"ContainerDied","Data":"7f2ccce6f3259420d05f733145dfe83022b342d9c4782526b05e6780b7325ef2"}
Sep 17 22:07:09.359839 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:09.359813    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ztvtj"
Sep 17 22:07:09.387683 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:09.387621    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crdw6x"
Sep 17 22:07:09.466403 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:09.466374    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94271884-d40d-4f07-8680-b56912e0badd-util\") pod \"94271884-d40d-4f07-8680-b56912e0badd\" (UID: \"94271884-d40d-4f07-8680-b56912e0badd\") "
Sep 17 22:07:09.466533 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:09.466428    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ac5ed46-ef53-4750-b80c-de7d7209bfc4-bundle\") pod \"8ac5ed46-ef53-4750-b80c-de7d7209bfc4\" (UID: \"8ac5ed46-ef53-4750-b80c-de7d7209bfc4\") "
Sep 17 22:07:09.466533 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:09.466480    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94271884-d40d-4f07-8680-b56912e0badd-bundle\") pod \"94271884-d40d-4f07-8680-b56912e0badd\" (UID: \"94271884-d40d-4f07-8680-b56912e0badd\") "
Sep 17 22:07:09.466533 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:09.466502    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2mz4\" (UniqueName: \"kubernetes.io/projected/8ac5ed46-ef53-4750-b80c-de7d7209bfc4-kube-api-access-x2mz4\") pod \"8ac5ed46-ef53-4750-b80c-de7d7209bfc4\" (UID: \"8ac5ed46-ef53-4750-b80c-de7d7209bfc4\") "
Sep 17 22:07:09.466684 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:09.466557    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbwz8\" (UniqueName: \"kubernetes.io/projected/94271884-d40d-4f07-8680-b56912e0badd-kube-api-access-lbwz8\") pod \"94271884-d40d-4f07-8680-b56912e0badd\" (UID: \"94271884-d40d-4f07-8680-b56912e0badd\") "
Sep 17 22:07:09.466684 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:09.466607    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ac5ed46-ef53-4750-b80c-de7d7209bfc4-util\") pod \"8ac5ed46-ef53-4750-b80c-de7d7209bfc4\" (UID: \"8ac5ed46-ef53-4750-b80c-de7d7209bfc4\") "
Sep 17 22:07:09.467066 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:09.467041    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94271884-d40d-4f07-8680-b56912e0badd-bundle" (OuterVolumeSpecName: "bundle") pod "94271884-d40d-4f07-8680-b56912e0badd" (UID: "94271884-d40d-4f07-8680-b56912e0badd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:07:09.467203 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:09.467062    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ac5ed46-ef53-4750-b80c-de7d7209bfc4-bundle" (OuterVolumeSpecName: "bundle") pod "8ac5ed46-ef53-4750-b80c-de7d7209bfc4" (UID: "8ac5ed46-ef53-4750-b80c-de7d7209bfc4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:07:09.468711 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:09.468683    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ac5ed46-ef53-4750-b80c-de7d7209bfc4-kube-api-access-x2mz4" (OuterVolumeSpecName: "kube-api-access-x2mz4") pod "8ac5ed46-ef53-4750-b80c-de7d7209bfc4" (UID: "8ac5ed46-ef53-4750-b80c-de7d7209bfc4"). InnerVolumeSpecName "kube-api-access-x2mz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:07:09.468829 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:09.468802    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94271884-d40d-4f07-8680-b56912e0badd-kube-api-access-lbwz8" (OuterVolumeSpecName: "kube-api-access-lbwz8") pod "94271884-d40d-4f07-8680-b56912e0badd" (UID: "94271884-d40d-4f07-8680-b56912e0badd"). InnerVolumeSpecName "kube-api-access-lbwz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:07:09.475223 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:09.475199    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94271884-d40d-4f07-8680-b56912e0badd-util" (OuterVolumeSpecName: "util") pod "94271884-d40d-4f07-8680-b56912e0badd" (UID: "94271884-d40d-4f07-8680-b56912e0badd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:07:09.475709 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:09.475687    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ac5ed46-ef53-4750-b80c-de7d7209bfc4-util" (OuterVolumeSpecName: "util") pod "8ac5ed46-ef53-4750-b80c-de7d7209bfc4" (UID: "8ac5ed46-ef53-4750-b80c-de7d7209bfc4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:07:09.567552 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:09.567523    2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ac5ed46-ef53-4750-b80c-de7d7209bfc4-bundle\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:07:09.567552 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:09.567549    2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94271884-d40d-4f07-8680-b56912e0badd-bundle\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:07:09.567718 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:09.567559    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2mz4\" (UniqueName: \"kubernetes.io/projected/8ac5ed46-ef53-4750-b80c-de7d7209bfc4-kube-api-access-x2mz4\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:07:09.567718 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:09.567569    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbwz8\" (UniqueName: \"kubernetes.io/projected/94271884-d40d-4f07-8680-b56912e0badd-kube-api-access-lbwz8\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:07:09.567718 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:09.567577    2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ac5ed46-ef53-4750-b80c-de7d7209bfc4-util\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:07:09.567718 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:09.567585    2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94271884-d40d-4f07-8680-b56912e0badd-util\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:07:10.227259 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:10.227218    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ztvtj"
Sep 17 22:07:10.227259 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:10.227235    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ztvtj" event={"ID":"94271884-d40d-4f07-8680-b56912e0badd","Type":"ContainerDied","Data":"4903b26082065db45ba96f54ead4ccb1f209055fa1737499e2fc47bef2b4543e"}
Sep 17 22:07:10.227751 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:10.227285    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4903b26082065db45ba96f54ead4ccb1f209055fa1737499e2fc47bef2b4543e"
Sep 17 22:07:10.229042 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:10.229022    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crdw6x"
Sep 17 22:07:10.229156 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:10.229048    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crdw6x" event={"ID":"8ac5ed46-ef53-4750-b80c-de7d7209bfc4","Type":"ContainerDied","Data":"9c98e99e4ff3bf6b6fec7bbe65e74db36b0cc92089f22e8e261b09a65e8ce340"}
Sep 17 22:07:10.229156 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:10.229088    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c98e99e4ff3bf6b6fec7bbe65e74db36b0cc92089f22e8e261b09a65e8ce340"
Sep 17 22:07:10.365401 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:10.365379    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hmg4f"
Sep 17 22:07:10.395481 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:10.395459    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5spng"
Sep 17 22:07:10.475489 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:10.475466    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/efaa0f2e-64de-4ecb-8bc3-c057102180c2-bundle\") pod \"efaa0f2e-64de-4ecb-8bc3-c057102180c2\" (UID: \"efaa0f2e-64de-4ecb-8bc3-c057102180c2\") "
Sep 17 22:07:10.475637 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:10.475522    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/efaa0f2e-64de-4ecb-8bc3-c057102180c2-util\") pod \"efaa0f2e-64de-4ecb-8bc3-c057102180c2\" (UID: \"efaa0f2e-64de-4ecb-8bc3-c057102180c2\") "
Sep 17 22:07:10.475637 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:10.475571    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09c747bc-3b24-4434-ab0a-e69dd6f1ee08-util\") pod \"09c747bc-3b24-4434-ab0a-e69dd6f1ee08\" (UID: \"09c747bc-3b24-4434-ab0a-e69dd6f1ee08\") "
Sep 17 22:07:10.475637 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:10.475597    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jldwr\" (UniqueName: \"kubernetes.io/projected/09c747bc-3b24-4434-ab0a-e69dd6f1ee08-kube-api-access-jldwr\") pod \"09c747bc-3b24-4434-ab0a-e69dd6f1ee08\" (UID: \"09c747bc-3b24-4434-ab0a-e69dd6f1ee08\") "
Sep 17 22:07:10.475637 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:10.475622    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7mzf\" (UniqueName: \"kubernetes.io/projected/efaa0f2e-64de-4ecb-8bc3-c057102180c2-kube-api-access-w7mzf\") pod \"efaa0f2e-64de-4ecb-8bc3-c057102180c2\" (UID: \"efaa0f2e-64de-4ecb-8bc3-c057102180c2\") "
Sep 17 22:07:10.475862 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:10.475642    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09c747bc-3b24-4434-ab0a-e69dd6f1ee08-bundle\") pod \"09c747bc-3b24-4434-ab0a-e69dd6f1ee08\" (UID: \"09c747bc-3b24-4434-ab0a-e69dd6f1ee08\") "
Sep 17 22:07:10.476326 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:10.476258    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efaa0f2e-64de-4ecb-8bc3-c057102180c2-bundle" (OuterVolumeSpecName: "bundle") pod "efaa0f2e-64de-4ecb-8bc3-c057102180c2" (UID: "efaa0f2e-64de-4ecb-8bc3-c057102180c2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:07:10.476326 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:10.476256    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c747bc-3b24-4434-ab0a-e69dd6f1ee08-bundle" (OuterVolumeSpecName: "bundle") pod "09c747bc-3b24-4434-ab0a-e69dd6f1ee08" (UID: "09c747bc-3b24-4434-ab0a-e69dd6f1ee08"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:07:10.477826 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:10.477777    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efaa0f2e-64de-4ecb-8bc3-c057102180c2-kube-api-access-w7mzf" (OuterVolumeSpecName: "kube-api-access-w7mzf") pod "efaa0f2e-64de-4ecb-8bc3-c057102180c2" (UID: "efaa0f2e-64de-4ecb-8bc3-c057102180c2"). InnerVolumeSpecName "kube-api-access-w7mzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:07:10.478092 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:10.478073    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09c747bc-3b24-4434-ab0a-e69dd6f1ee08-kube-api-access-jldwr" (OuterVolumeSpecName: "kube-api-access-jldwr") pod "09c747bc-3b24-4434-ab0a-e69dd6f1ee08" (UID: "09c747bc-3b24-4434-ab0a-e69dd6f1ee08"). InnerVolumeSpecName "kube-api-access-jldwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:07:10.481396 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:10.481362    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c747bc-3b24-4434-ab0a-e69dd6f1ee08-util" (OuterVolumeSpecName: "util") pod "09c747bc-3b24-4434-ab0a-e69dd6f1ee08" (UID: "09c747bc-3b24-4434-ab0a-e69dd6f1ee08"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:07:10.482369 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:10.482350    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efaa0f2e-64de-4ecb-8bc3-c057102180c2-util" (OuterVolumeSpecName: "util") pod "efaa0f2e-64de-4ecb-8bc3-c057102180c2" (UID: "efaa0f2e-64de-4ecb-8bc3-c057102180c2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:07:10.576380 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:10.576356    2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09c747bc-3b24-4434-ab0a-e69dd6f1ee08-util\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:07:10.576380 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:10.576376    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jldwr\" (UniqueName: \"kubernetes.io/projected/09c747bc-3b24-4434-ab0a-e69dd6f1ee08-kube-api-access-jldwr\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:07:10.576515 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:10.576386    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7mzf\" (UniqueName: \"kubernetes.io/projected/efaa0f2e-64de-4ecb-8bc3-c057102180c2-kube-api-access-w7mzf\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:07:10.576515 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:10.576396    2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09c747bc-3b24-4434-ab0a-e69dd6f1ee08-bundle\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:07:10.576515 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:10.576404    2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/efaa0f2e-64de-4ecb-8bc3-c057102180c2-bundle\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:07:10.576515 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:10.576411    2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/efaa0f2e-64de-4ecb-8bc3-c057102180c2-util\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:07:11.235478 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:11.235449    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5spng"
Sep 17 22:07:11.235887 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:11.235448    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5spng" event={"ID":"efaa0f2e-64de-4ecb-8bc3-c057102180c2","Type":"ContainerDied","Data":"ac4e461ce5008021491076716ccc47027834b833b3d5175082a5a209ed6f8868"}
Sep 17 22:07:11.235887 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:11.235578    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac4e461ce5008021491076716ccc47027834b833b3d5175082a5a209ed6f8868"
Sep 17 22:07:11.237600 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:11.237575    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hmg4f" event={"ID":"09c747bc-3b24-4434-ab0a-e69dd6f1ee08","Type":"ContainerDied","Data":"e424ffd842cfe22c020daa479d2ff325031e4e15b4df8a20cdeb8b138f79c8c0"}
Sep 17 22:07:11.237719 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:11.237682    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e424ffd842cfe22c020daa479d2ff325031e4e15b4df8a20cdeb8b138f79c8c0"
Sep 17 22:07:11.237719 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:11.237623    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hmg4f"
Sep 17 22:07:14.841654 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:14.841621    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:15.628411 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:15.628382    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:16.436901 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:16.436870    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:19.221413 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.221377    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-88zj2"]
Sep 17 22:07:19.221799 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.221707    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94271884-d40d-4f07-8680-b56912e0badd" containerName="extract"
Sep 17 22:07:19.221799 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.221717    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="94271884-d40d-4f07-8680-b56912e0badd" containerName="extract"
Sep 17 22:07:19.221799 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.221726    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94271884-d40d-4f07-8680-b56912e0badd" containerName="pull"
Sep 17 22:07:19.221799 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.221732    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="94271884-d40d-4f07-8680-b56912e0badd" containerName="pull"
Sep 17 22:07:19.221799 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.221741    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09c747bc-3b24-4434-ab0a-e69dd6f1ee08" containerName="extract"
Sep 17 22:07:19.221799 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.221746    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="09c747bc-3b24-4434-ab0a-e69dd6f1ee08" containerName="extract"
Sep 17 22:07:19.221799 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.221755    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ac5ed46-ef53-4750-b80c-de7d7209bfc4" containerName="pull"
Sep 17 22:07:19.221799 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.221760    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ac5ed46-ef53-4750-b80c-de7d7209bfc4" containerName="pull"
Sep 17 22:07:19.221799 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.221771    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efaa0f2e-64de-4ecb-8bc3-c057102180c2" containerName="pull"
Sep 17 22:07:19.221799 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.221776    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="efaa0f2e-64de-4ecb-8bc3-c057102180c2" containerName="pull"
Sep 17 22:07:19.221799 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.221782    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09c747bc-3b24-4434-ab0a-e69dd6f1ee08" containerName="pull"
Sep 17 22:07:19.221799 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.221787    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="09c747bc-3b24-4434-ab0a-e69dd6f1ee08" containerName="pull"
Sep 17 22:07:19.221799 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.221792    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ac5ed46-ef53-4750-b80c-de7d7209bfc4" containerName="extract"
Sep 17 22:07:19.221799 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.221797    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ac5ed46-ef53-4750-b80c-de7d7209bfc4" containerName="extract"
Sep 17 22:07:19.221799 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.221804    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ac5ed46-ef53-4750-b80c-de7d7209bfc4" containerName="util"
Sep 17 22:07:19.221799 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.221809    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ac5ed46-ef53-4750-b80c-de7d7209bfc4" containerName="util"
Sep 17 22:07:19.222304 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.221817    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efaa0f2e-64de-4ecb-8bc3-c057102180c2" containerName="extract"
Sep 17 22:07:19.222304 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.221822    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="efaa0f2e-64de-4ecb-8bc3-c057102180c2" containerName="extract"
Sep 17 22:07:19.222304 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.221829    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efaa0f2e-64de-4ecb-8bc3-c057102180c2" containerName="util"
Sep 17 22:07:19.222304 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.221834    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="efaa0f2e-64de-4ecb-8bc3-c057102180c2" containerName="util"
Sep 17 22:07:19.222304 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.221843    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09c747bc-3b24-4434-ab0a-e69dd6f1ee08" containerName="util"
Sep 17 22:07:19.222304 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.221848    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="09c747bc-3b24-4434-ab0a-e69dd6f1ee08" containerName="util"
Sep 17 22:07:19.222304 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.221854    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94271884-d40d-4f07-8680-b56912e0badd" containerName="util"
Sep 17 22:07:19.222304 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.221858    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="94271884-d40d-4f07-8680-b56912e0badd" containerName="util"
Sep 17 22:07:19.222304 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.221912    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="efaa0f2e-64de-4ecb-8bc3-c057102180c2" containerName="extract"
Sep 17 22:07:19.222304 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.221919    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ac5ed46-ef53-4750-b80c-de7d7209bfc4" containerName="extract"
Sep 17 22:07:19.222304 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.221925    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="94271884-d40d-4f07-8680-b56912e0badd" containerName="extract"
Sep 17 22:07:19.222304 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.221930    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="09c747bc-3b24-4434-ab0a-e69dd6f1ee08" containerName="extract"
Sep 17 22:07:19.226016 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.226002    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-88zj2"
Sep 17 22:07:19.229549 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.229525    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 22:07:19.229671 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.229573    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-tq8pj\""
Sep 17 22:07:19.230169 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.230152    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 22:07:19.247938 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.247914    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-88zj2"]
Sep 17 22:07:19.297420 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.297398    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:19.346527 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.346506    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a22ea5b-8d83-4dee-9a49-9183acd54d19-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-88zj2\" (UID: \"7a22ea5b-8d83-4dee-9a49-9183acd54d19\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-88zj2"
Sep 17 22:07:19.346621 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.346563    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r5w6\" (UniqueName: \"kubernetes.io/projected/7a22ea5b-8d83-4dee-9a49-9183acd54d19-kube-api-access-8r5w6\") pod \"kuadrant-operator-controller-manager-6d4cb79c-88zj2\" (UID: \"7a22ea5b-8d83-4dee-9a49-9183acd54d19\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-88zj2"
Sep 17 22:07:19.397358 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.397329    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:19.447956 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.447932    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r5w6\" (UniqueName: \"kubernetes.io/projected/7a22ea5b-8d83-4dee-9a49-9183acd54d19-kube-api-access-8r5w6\") pod \"kuadrant-operator-controller-manager-6d4cb79c-88zj2\" (UID: \"7a22ea5b-8d83-4dee-9a49-9183acd54d19\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-88zj2"
Sep 17 22:07:19.448040 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.447983    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a22ea5b-8d83-4dee-9a49-9183acd54d19-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-88zj2\" (UID: \"7a22ea5b-8d83-4dee-9a49-9183acd54d19\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-88zj2"
Sep 17 22:07:19.448394 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.448378    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a22ea5b-8d83-4dee-9a49-9183acd54d19-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-88zj2\" (UID: \"7a22ea5b-8d83-4dee-9a49-9183acd54d19\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-88zj2"
Sep 17 22:07:19.455621 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.455594    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r5w6\" (UniqueName: \"kubernetes.io/projected/7a22ea5b-8d83-4dee-9a49-9183acd54d19-kube-api-access-8r5w6\") pod \"kuadrant-operator-controller-manager-6d4cb79c-88zj2\" (UID: \"7a22ea5b-8d83-4dee-9a49-9183acd54d19\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-88zj2"
Sep 17 22:07:19.535733 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.535685    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-88zj2"
Sep 17 22:07:19.700743 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.700715    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:19.802121 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.802042    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:19.824063 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.824032    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-88zj2"]
Sep 17 22:07:19.826331 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:07:19.826306    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a22ea5b_8d83_4dee_9a49_9183acd54d19.slice/crio-05fcfe7ceacc8d6bc27100cf693593c7eb2447edf140eddfd1f1893e617a33f2 WatchSource:0}: Error finding container 05fcfe7ceacc8d6bc27100cf693593c7eb2447edf140eddfd1f1893e617a33f2: Status 404 returned error can't find the container with id 05fcfe7ceacc8d6bc27100cf693593c7eb2447edf140eddfd1f1893e617a33f2
Sep 17 22:07:19.900487 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:19.900459    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:20.283649 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:20.283615    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-88zj2" event={"ID":"7a22ea5b-8d83-4dee-9a49-9183acd54d19","Type":"ContainerStarted","Data":"05fcfe7ceacc8d6bc27100cf693593c7eb2447edf140eddfd1f1893e617a33f2"}
Sep 17 22:07:21.458871 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:21.458839    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:21.849872 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:21.849798    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:22.120605 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:22.120527    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:23.804313 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:23.804284    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-ndwgf"]
Sep 17 22:07:23.807878 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:23.807854    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-ndwgf"
Sep 17 22:07:23.810885 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:23.810844    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-t5nj9\""
Sep 17 22:07:23.825157 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:23.825133    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-ndwgf"]
Sep 17 22:07:23.883975 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:23.883933    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxxs2\" (UniqueName: \"kubernetes.io/projected/20a39873-82d6-4f52-8b6e-a2bbf66ae0e2-kube-api-access-kxxs2\") pod \"authorino-operator-6fc8f864f4-ndwgf\" (UID: \"20a39873-82d6-4f52-8b6e-a2bbf66ae0e2\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-ndwgf"
Sep 17 22:07:23.917059 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:23.917020    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:23.985249 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:23.985217    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxxs2\" (UniqueName: \"kubernetes.io/projected/20a39873-82d6-4f52-8b6e-a2bbf66ae0e2-kube-api-access-kxxs2\") pod \"authorino-operator-6fc8f864f4-ndwgf\" (UID: \"20a39873-82d6-4f52-8b6e-a2bbf66ae0e2\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-ndwgf"
Sep 17 22:07:23.994094 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:23.994067    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxxs2\" (UniqueName: \"kubernetes.io/projected/20a39873-82d6-4f52-8b6e-a2bbf66ae0e2-kube-api-access-kxxs2\") pod \"authorino-operator-6fc8f864f4-ndwgf\" (UID: \"20a39873-82d6-4f52-8b6e-a2bbf66ae0e2\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-ndwgf"
Sep 17 22:07:24.120607 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:24.120520    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-ndwgf"
Sep 17 22:07:24.404110 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:24.404047    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-ndwgf"]
Sep 17 22:07:24.408349 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:07:24.408314    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20a39873_82d6_4f52_8b6e_a2bbf66ae0e2.slice/crio-a6d84d542f7823e67b9550b272f385336c8ba14a4b3ef88caa70f48bf13b1514 WatchSource:0}: Error finding container a6d84d542f7823e67b9550b272f385336c8ba14a4b3ef88caa70f48bf13b1514: Status 404 returned error can't find the container with id a6d84d542f7823e67b9550b272f385336c8ba14a4b3ef88caa70f48bf13b1514
Sep 17 22:07:24.443376 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:24.443341    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:25.127442 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:25.127413    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:25.311538 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:25.311495    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-ndwgf" event={"ID":"20a39873-82d6-4f52-8b6e-a2bbf66ae0e2","Type":"ContainerStarted","Data":"a6d84d542f7823e67b9550b272f385336c8ba14a4b3ef88caa70f48bf13b1514"}
Sep 17 22:07:26.318534 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:26.318452    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-88zj2" event={"ID":"7a22ea5b-8d83-4dee-9a49-9183acd54d19","Type":"ContainerStarted","Data":"9bc999812a309f4b7c2684d0481e58a3cd0d5ce3627937fddcd6ed9d72d16089"}
Sep 17 22:07:26.318896 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:26.318601    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-88zj2"
Sep 17 22:07:26.339029 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:26.338981    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-88zj2" podStartSLOduration=1.118372765 podStartE2EDuration="7.338968518s" totalImagesPullingTime="6.220595753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:07:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:07:26.338643922 +0000 UTC m=+547.784689207" watchObservedRunningTime="2026-09-17 22:07:26.338968518 +0000 UTC m=+547.785013790"
Sep 17 22:07:26.934515 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:26.934480    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:28.390720 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:28.390693    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:28.552987 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:28.552952    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:29.841509 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:29.841477    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:29.901154 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:29.901119    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:30.442542 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:30.442512    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:30.499157 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:30.499129    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:30.851876 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:30.851804    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:30.915848 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:30.915821    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:33.232295 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:33.232262    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:33.284305 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:33.284274    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:37.326655 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:37.326620    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-88zj2"
Sep 17 22:07:37.905337 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:37.905296    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:38.019810 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:38.019781    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:38.384658 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:38.384625    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-ndwgf" event={"ID":"20a39873-82d6-4f52-8b6e-a2bbf66ae0e2","Type":"ContainerStarted","Data":"2639f58e19d7bde02994f3df4cb862f661426d0bd3dccc96cec1de382544e31a"}
Sep 17 22:07:38.385025 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:38.384774    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-ndwgf"
Sep 17 22:07:38.411250 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:38.411163    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-6fc8f864f4-ndwgf" podStartSLOduration=2.313935733 podStartE2EDuration="15.411150013s" totalImagesPullingTime="13.09721428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:07:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:07:38.409846442 +0000 UTC m=+559.855891715" watchObservedRunningTime="2026-09-17 22:07:38.411150013 +0000 UTC m=+559.857195285"
Sep 17 22:07:43.260991 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:43.260953    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:43.375550 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:43.375501    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:46.260542 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:46.260505    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:46.358983 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:46.358952    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:49.393100 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:49.393070    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-ndwgf"
Sep 17 22:07:51.140745 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:51.140707    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:51.240973 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:51.240942    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:51.305153 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:51.305123    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:57.623937 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:57.623905    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:57.723447 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:07:57.723413    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:08:04.166843 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:04.166812    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:08:04.266575 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:04.266518    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:08:07.173318 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:07.173281    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:08:07.274283 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:07.274254    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:08:08.786459 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:08.786426    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:08:08.881149 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:08.881122    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:08:08.944327 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:08.944299    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:08:09.154470 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:09.154440    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t2dgh"]
Sep 17 22:08:09.158093 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:09.158073    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-t2dgh"
Sep 17 22:08:09.160023 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:09.160007    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 17 22:08:09.160133 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:09.160103    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-z2qkm\""
Sep 17 22:08:09.168626 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:09.168601    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t2dgh"]
Sep 17 22:08:09.177107 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:09.177084    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t2dgh"]
Sep 17 22:08:09.250356 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:09.250333    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5463fd22-38ab-45a8-862b-6d143f816cd6-config-file\") pod \"limitador-limitador-84994f7ffc-t2dgh\" (UID: \"5463fd22-38ab-45a8-862b-6d143f816cd6\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-t2dgh"
Sep 17 22:08:09.250459 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:09.250394    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngx9g\" (UniqueName: \"kubernetes.io/projected/5463fd22-38ab-45a8-862b-6d143f816cd6-kube-api-access-ngx9g\") pod \"limitador-limitador-84994f7ffc-t2dgh\" (UID: \"5463fd22-38ab-45a8-862b-6d143f816cd6\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-t2dgh"
Sep 17 22:08:09.351174 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:09.351146    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5463fd22-38ab-45a8-862b-6d143f816cd6-config-file\") pod \"limitador-limitador-84994f7ffc-t2dgh\" (UID: \"5463fd22-38ab-45a8-862b-6d143f816cd6\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-t2dgh"
Sep 17 22:08:09.351316 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:09.351270    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngx9g\" (UniqueName: \"kubernetes.io/projected/5463fd22-38ab-45a8-862b-6d143f816cd6-kube-api-access-ngx9g\") pod \"limitador-limitador-84994f7ffc-t2dgh\" (UID: \"5463fd22-38ab-45a8-862b-6d143f816cd6\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-t2dgh"
Sep 17 22:08:09.351821 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:09.351801    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5463fd22-38ab-45a8-862b-6d143f816cd6-config-file\") pod \"limitador-limitador-84994f7ffc-t2dgh\" (UID: \"5463fd22-38ab-45a8-862b-6d143f816cd6\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-t2dgh"
Sep 17 22:08:09.358917 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:09.358894    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngx9g\" (UniqueName: \"kubernetes.io/projected/5463fd22-38ab-45a8-862b-6d143f816cd6-kube-api-access-ngx9g\") pod \"limitador-limitador-84994f7ffc-t2dgh\" (UID: \"5463fd22-38ab-45a8-862b-6d143f816cd6\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-t2dgh"
Sep 17 22:08:09.470658 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:09.470590    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-t2dgh"
Sep 17 22:08:09.625221 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:09.625174    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:08:09.725824 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:09.725750    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:08:09.742000 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:09.741971    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t2dgh"]
Sep 17 22:08:09.745586 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:08:09.745561    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5463fd22_38ab_45a8_862b_6d143f816cd6.slice/crio-9fbb96caf5925e6b5240ff72a847beb8f3c2a6d8640a61b153f76f001e1cf660 WatchSource:0}: Error finding container 9fbb96caf5925e6b5240ff72a847beb8f3c2a6d8640a61b153f76f001e1cf660: Status 404 returned error can't find the container with id 9fbb96caf5925e6b5240ff72a847beb8f3c2a6d8640a61b153f76f001e1cf660
Sep 17 22:08:10.558243 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:10.558211    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-t2dgh" event={"ID":"5463fd22-38ab-45a8-862b-6d143f816cd6","Type":"ContainerStarted","Data":"9fbb96caf5925e6b5240ff72a847beb8f3c2a6d8640a61b153f76f001e1cf660"}
Sep 17 22:08:12.374805 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:12.374774    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:08:12.477932 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:12.477906    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:08:12.571054 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:12.571028    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-t2dgh" event={"ID":"5463fd22-38ab-45a8-862b-6d143f816cd6","Type":"ContainerStarted","Data":"56d88e43a3863cf722cd1a715c589a365107cfc29e0479d18217b0127fd8f608"}
Sep 17 22:08:12.571233 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:12.571137    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-t2dgh"
Sep 17 22:08:12.587503 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:12.587455    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-t2dgh" podStartSLOduration=1.5480446319999999 podStartE2EDuration="3.587441597s" totalImagesPullingTime="2.039396965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:08:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:08:12.586090109 +0000 UTC m=+594.032135394" watchObservedRunningTime="2026-09-17 22:08:12.587441597 +0000 UTC m=+594.033486868"
Sep 17 22:08:12.773370 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:12.773343    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:08:12.873446 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:12.873409    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:08:13.001535 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:13.001503    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:08:13.111807 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:13.111734    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:08:19.438442 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:19.438411    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:08:19.538334 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:19.538306    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:08:20.600058 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:20.600033    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-7njvj_a324c9ab-5d42-4d31-8ff7-449f619be774/console-operator/1.log"
Sep 17 22:08:20.601495 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:20.601467    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-7njvj_a324c9ab-5d42-4d31-8ff7-449f619be774/console-operator/1.log"
Sep 17 22:08:20.603732 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:20.603714    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s7f5_4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf/ovn-acl-logging/0.log"
Sep 17 22:08:20.605407 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:20.605389    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s7f5_4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf/ovn-acl-logging/0.log"
Sep 17 22:08:23.578031 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:23.578001    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-t2dgh"
Sep 17 22:08:25.987195 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:25.987147    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:08:26.088343 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:26.088313    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:08:28.988578 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:28.988541    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:08:29.087526 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:29.087494    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:08:29.692280 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:29.692251    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:08:30.366059 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:30.366003    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:08:37.681454 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:37.681366    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:08:37.741545 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:37.741515    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:08:42.872545 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:42.872513    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:08:42.973654 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:42.973620    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:08:55.690411 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:55.690377    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kdwl9"]
Sep 17 22:08:55.690826 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:55.690599    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kdwl9" podUID="83ff6a8e-b9d4-42d3-9891-728ac2eed63a" containerName="discovery" containerID="cri-o://5988b1e1fe9bec90a639eec14a6a157f84f2c093e86de21b22d426c1d9de9b4f" gracePeriod=30
Sep 17 22:08:55.943214 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:55.943141    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kdwl9"
Sep 17 22:08:56.034972 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:56.034941    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9sdvp\" (UniqueName: \"kubernetes.io/projected/83ff6a8e-b9d4-42d3-9891-728ac2eed63a-kube-api-access-9sdvp\") pod \"83ff6a8e-b9d4-42d3-9891-728ac2eed63a\" (UID: \"83ff6a8e-b9d4-42d3-9891-728ac2eed63a\") "
Sep 17 22:08:56.035142 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:56.034977    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/83ff6a8e-b9d4-42d3-9891-728ac2eed63a-istio-csr-dns-cert\") pod \"83ff6a8e-b9d4-42d3-9891-728ac2eed63a\" (UID: \"83ff6a8e-b9d4-42d3-9891-728ac2eed63a\") "
Sep 17 22:08:56.035142 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:56.035035    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83ff6a8e-b9d4-42d3-9891-728ac2eed63a-istio-token\") pod \"83ff6a8e-b9d4-42d3-9891-728ac2eed63a\" (UID: \"83ff6a8e-b9d4-42d3-9891-728ac2eed63a\") "
Sep 17 22:08:56.035142 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:56.035082    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/83ff6a8e-b9d4-42d3-9891-728ac2eed63a-istio-kubeconfig\") pod \"83ff6a8e-b9d4-42d3-9891-728ac2eed63a\" (UID: \"83ff6a8e-b9d4-42d3-9891-728ac2eed63a\") "
Sep 17 22:08:56.035142 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:56.035105    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/83ff6a8e-b9d4-42d3-9891-728ac2eed63a-istio-csr-ca-configmap\") pod \"83ff6a8e-b9d4-42d3-9891-728ac2eed63a\" (UID: \"83ff6a8e-b9d4-42d3-9891-728ac2eed63a\") "
Sep 17 22:08:56.035142 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:56.035134    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/83ff6a8e-b9d4-42d3-9891-728ac2eed63a-local-certs\") pod \"83ff6a8e-b9d4-42d3-9891-728ac2eed63a\" (UID: \"83ff6a8e-b9d4-42d3-9891-728ac2eed63a\") "
Sep 17 22:08:56.035457 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:56.035251    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/83ff6a8e-b9d4-42d3-9891-728ac2eed63a-cacerts\") pod \"83ff6a8e-b9d4-42d3-9891-728ac2eed63a\" (UID: \"83ff6a8e-b9d4-42d3-9891-728ac2eed63a\") "
Sep 17 22:08:56.035566 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:56.035532    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83ff6a8e-b9d4-42d3-9891-728ac2eed63a-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "83ff6a8e-b9d4-42d3-9891-728ac2eed63a" (UID: "83ff6a8e-b9d4-42d3-9891-728ac2eed63a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:08:56.037542 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:56.037501    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83ff6a8e-b9d4-42d3-9891-728ac2eed63a-kube-api-access-9sdvp" (OuterVolumeSpecName: "kube-api-access-9sdvp") pod "83ff6a8e-b9d4-42d3-9891-728ac2eed63a" (UID: "83ff6a8e-b9d4-42d3-9891-728ac2eed63a"). InnerVolumeSpecName "kube-api-access-9sdvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:08:56.037659 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:56.037627    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83ff6a8e-b9d4-42d3-9891-728ac2eed63a-cacerts" (OuterVolumeSpecName: "cacerts") pod "83ff6a8e-b9d4-42d3-9891-728ac2eed63a" (UID: "83ff6a8e-b9d4-42d3-9891-728ac2eed63a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:08:56.037719 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:56.037686    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83ff6a8e-b9d4-42d3-9891-728ac2eed63a-istio-token" (OuterVolumeSpecName: "istio-token") pod "83ff6a8e-b9d4-42d3-9891-728ac2eed63a" (UID: "83ff6a8e-b9d4-42d3-9891-728ac2eed63a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:08:56.037759 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:56.037738    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83ff6a8e-b9d4-42d3-9891-728ac2eed63a-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "83ff6a8e-b9d4-42d3-9891-728ac2eed63a" (UID: "83ff6a8e-b9d4-42d3-9891-728ac2eed63a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:08:56.037836 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:56.037758    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83ff6a8e-b9d4-42d3-9891-728ac2eed63a-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "83ff6a8e-b9d4-42d3-9891-728ac2eed63a" (UID: "83ff6a8e-b9d4-42d3-9891-728ac2eed63a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:08:56.037937 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:56.037916    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83ff6a8e-b9d4-42d3-9891-728ac2eed63a-local-certs" (OuterVolumeSpecName: "local-certs") pod "83ff6a8e-b9d4-42d3-9891-728ac2eed63a" (UID: "83ff6a8e-b9d4-42d3-9891-728ac2eed63a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:08:56.136295 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:56.136255    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sdvp\" (UniqueName: \"kubernetes.io/projected/83ff6a8e-b9d4-42d3-9891-728ac2eed63a-kube-api-access-9sdvp\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:08:56.136295 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:56.136293    2555 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/83ff6a8e-b9d4-42d3-9891-728ac2eed63a-istio-csr-dns-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:08:56.136467 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:56.136304    2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83ff6a8e-b9d4-42d3-9891-728ac2eed63a-istio-token\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:08:56.136467 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:56.136312    2555 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/83ff6a8e-b9d4-42d3-9891-728ac2eed63a-istio-kubeconfig\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:08:56.136467 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:56.136321    2555 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/83ff6a8e-b9d4-42d3-9891-728ac2eed63a-istio-csr-ca-configmap\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:08:56.136467 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:56.136330    2555 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/83ff6a8e-b9d4-42d3-9891-728ac2eed63a-local-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:08:56.136467 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:56.136338    2555 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/83ff6a8e-b9d4-42d3-9891-728ac2eed63a-cacerts\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:08:56.801106 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:56.801073    2555 generic.go:356] "Generic (PLEG): container finished" podID="83ff6a8e-b9d4-42d3-9891-728ac2eed63a" containerID="5988b1e1fe9bec90a639eec14a6a157f84f2c093e86de21b22d426c1d9de9b4f" exitCode=0
Sep 17 22:08:56.801571 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:56.801122    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kdwl9"
Sep 17 22:08:56.801571 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:56.801165    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kdwl9" event={"ID":"83ff6a8e-b9d4-42d3-9891-728ac2eed63a","Type":"ContainerDied","Data":"5988b1e1fe9bec90a639eec14a6a157f84f2c093e86de21b22d426c1d9de9b4f"}
Sep 17 22:08:56.801571 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:56.801248    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kdwl9" event={"ID":"83ff6a8e-b9d4-42d3-9891-728ac2eed63a","Type":"ContainerDied","Data":"e16b520ee03cca21fe2e47e8ea1bfad079b6840eaa068df9130dc7c3392f5ef4"}
Sep 17 22:08:56.801571 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:56.801280    2555 scope.go:117] "RemoveContainer" containerID="5988b1e1fe9bec90a639eec14a6a157f84f2c093e86de21b22d426c1d9de9b4f"
Sep 17 22:08:56.810105 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:56.810086    2555 scope.go:117] "RemoveContainer" containerID="5988b1e1fe9bec90a639eec14a6a157f84f2c093e86de21b22d426c1d9de9b4f"
Sep 17 22:08:56.810436 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:08:56.810415    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5988b1e1fe9bec90a639eec14a6a157f84f2c093e86de21b22d426c1d9de9b4f\": container with ID starting with 5988b1e1fe9bec90a639eec14a6a157f84f2c093e86de21b22d426c1d9de9b4f not found: ID does not exist" containerID="5988b1e1fe9bec90a639eec14a6a157f84f2c093e86de21b22d426c1d9de9b4f"
Sep 17 22:08:56.810520 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:56.810443    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5988b1e1fe9bec90a639eec14a6a157f84f2c093e86de21b22d426c1d9de9b4f"} err="failed to get container status \"5988b1e1fe9bec90a639eec14a6a157f84f2c093e86de21b22d426c1d9de9b4f\": rpc error: code = NotFound desc = could not find container \"5988b1e1fe9bec90a639eec14a6a157f84f2c093e86de21b22d426c1d9de9b4f\": container with ID starting with 5988b1e1fe9bec90a639eec14a6a157f84f2c093e86de21b22d426c1d9de9b4f not found: ID does not exist"
Sep 17 22:08:56.820731 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:56.820707    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kdwl9"]
Sep 17 22:08:56.826366 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:56.826340    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kdwl9"]
Sep 17 22:08:58.679785 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:08:58.679751    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83ff6a8e-b9d4-42d3-9891-728ac2eed63a" path="/var/lib/kubelet/pods/83ff6a8e-b9d4-42d3-9891-728ac2eed63a/volumes"
Sep 17 22:09:02.915941 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:02.915906    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:09:02.986619 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:02.986585    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-79b679679b-cjswc"]
Sep 17 22:09:02.986957 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:02.986943    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83ff6a8e-b9d4-42d3-9891-728ac2eed63a" containerName="discovery"
Sep 17 22:09:02.986957 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:02.986957    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="83ff6a8e-b9d4-42d3-9891-728ac2eed63a" containerName="discovery"
Sep 17 22:09:02.987056 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:02.987016    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="83ff6a8e-b9d4-42d3-9891-728ac2eed63a" containerName="discovery"
Sep 17 22:09:02.989960 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:02.989942    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-79b679679b-cjswc"
Sep 17 22:09:02.992514 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:02.992492    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 17 22:09:02.992627 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:02.992548    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 22:09:02.992697 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:02.992626    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 22:09:02.993219 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:02.993205    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-hn89m\""
Sep 17 22:09:02.998975 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:02.998956    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:09:03.003621 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:03.003600    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-79b679679b-cjswc"]
Sep 17 22:09:03.008172 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:03.008147    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-7c5846b585-w4kcq"]
Sep 17 22:09:03.011510 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:03.011491    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-7c5846b585-w4kcq"
Sep 17 22:09:03.013979 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:03.013957    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 17 22:09:03.014285 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:03.014266    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-rsfp5\""
Sep 17 22:09:03.023033 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:03.023010    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-7c5846b585-w4kcq"]
Sep 17 22:09:03.028741 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:03.028721    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-4hswx"]
Sep 17 22:09:03.039530 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:03.039507    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-4hswx"
Sep 17 22:09:03.042082 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:03.042058    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 22:09:03.042368 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:03.042350    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-zn22q\""
Sep 17 22:09:03.043635 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:03.043613    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-4hswx"]
Sep 17 22:09:03.091704 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:03.091670    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbqgk\" (UniqueName: \"kubernetes.io/projected/1f4e978e-9098-4418-9e88-4134dd961423-kube-api-access-gbqgk\") pod \"kserve-controller-manager-79b679679b-cjswc\" (UID: \"1f4e978e-9098-4418-9e88-4134dd961423\") " pod="kserve/kserve-controller-manager-79b679679b-cjswc"
Sep 17 22:09:03.091873 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:03.091736    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f4e978e-9098-4418-9e88-4134dd961423-cert\") pod \"kserve-controller-manager-79b679679b-cjswc\" (UID: \"1f4e978e-9098-4418-9e88-4134dd961423\") " pod="kserve/kserve-controller-manager-79b679679b-cjswc"
Sep 17 22:09:03.097466 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:03.097445    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:09:03.192662 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:03.192581    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f4e978e-9098-4418-9e88-4134dd961423-cert\") pod \"kserve-controller-manager-79b679679b-cjswc\" (UID: \"1f4e978e-9098-4418-9e88-4134dd961423\") " pod="kserve/kserve-controller-manager-79b679679b-cjswc"
Sep 17 22:09:03.192819 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:03.192689    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ae29a8f-5039-461a-846e-6215646c155b-cert\") pod \"llmisvc-controller-manager-7c5846b585-w4kcq\" (UID: \"3ae29a8f-5039-461a-846e-6215646c155b\") " pod="kserve/llmisvc-controller-manager-7c5846b585-w4kcq"
Sep 17 22:09:03.192819 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:03.192728    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbs4w\" (UniqueName: \"kubernetes.io/projected/4c524947-6176-4f5b-88bc-2d971a15e0c1-kube-api-access-kbs4w\") pod \"seaweedfs-64568bcd49-4hswx\" (UID: \"4c524947-6176-4f5b-88bc-2d971a15e0c1\") " pod="kserve/seaweedfs-64568bcd49-4hswx"
Sep 17 22:09:03.192819 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:09:03.192733    2555 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 17 22:09:03.192819 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:03.192800    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbqgk\" (UniqueName: \"kubernetes.io/projected/1f4e978e-9098-4418-9e88-4134dd961423-kube-api-access-gbqgk\") pod \"kserve-controller-manager-79b679679b-cjswc\" (UID: \"1f4e978e-9098-4418-9e88-4134dd961423\") " pod="kserve/kserve-controller-manager-79b679679b-cjswc"
Sep 17 22:09:03.192977 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:09:03.192827    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f4e978e-9098-4418-9e88-4134dd961423-cert podName:1f4e978e-9098-4418-9e88-4134dd961423 nodeName:}" failed. No retries permitted until 2026-09-17 22:09:03.692804987 +0000 UTC m=+645.138850239 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1f4e978e-9098-4418-9e88-4134dd961423-cert") pod "kserve-controller-manager-79b679679b-cjswc" (UID: "1f4e978e-9098-4418-9e88-4134dd961423") : secret "kserve-webhook-server-cert" not found
Sep 17 22:09:03.192977 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:03.192857    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4c524947-6176-4f5b-88bc-2d971a15e0c1-data\") pod \"seaweedfs-64568bcd49-4hswx\" (UID: \"4c524947-6176-4f5b-88bc-2d971a15e0c1\") " pod="kserve/seaweedfs-64568bcd49-4hswx"
Sep 17 22:09:03.192977 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:03.192896    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqt5v\" (UniqueName: \"kubernetes.io/projected/3ae29a8f-5039-461a-846e-6215646c155b-kube-api-access-zqt5v\") pod \"llmisvc-controller-manager-7c5846b585-w4kcq\" (UID: \"3ae29a8f-5039-461a-846e-6215646c155b\") " pod="kserve/llmisvc-controller-manager-7c5846b585-w4kcq"
Sep 17 22:09:03.201160 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:03.201135    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbqgk\" (UniqueName: \"kubernetes.io/projected/1f4e978e-9098-4418-9e88-4134dd961423-kube-api-access-gbqgk\") pod \"kserve-controller-manager-79b679679b-cjswc\" (UID: \"1f4e978e-9098-4418-9e88-4134dd961423\") " pod="kserve/kserve-controller-manager-79b679679b-cjswc"
Sep 17 22:09:03.293325 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:03.293289    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ae29a8f-5039-461a-846e-6215646c155b-cert\") pod \"llmisvc-controller-manager-7c5846b585-w4kcq\" (UID: \"3ae29a8f-5039-461a-846e-6215646c155b\") " pod="kserve/llmisvc-controller-manager-7c5846b585-w4kcq"
Sep 17 22:09:03.293325 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:03.293336    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbs4w\" (UniqueName: \"kubernetes.io/projected/4c524947-6176-4f5b-88bc-2d971a15e0c1-kube-api-access-kbs4w\") pod \"seaweedfs-64568bcd49-4hswx\" (UID: \"4c524947-6176-4f5b-88bc-2d971a15e0c1\") " pod="kserve/seaweedfs-64568bcd49-4hswx"
Sep 17 22:09:03.293548 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:03.293391    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4c524947-6176-4f5b-88bc-2d971a15e0c1-data\") pod \"seaweedfs-64568bcd49-4hswx\" (UID: \"4c524947-6176-4f5b-88bc-2d971a15e0c1\") " pod="kserve/seaweedfs-64568bcd49-4hswx"
Sep 17 22:09:03.293548 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:03.293443    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqt5v\" (UniqueName: \"kubernetes.io/projected/3ae29a8f-5039-461a-846e-6215646c155b-kube-api-access-zqt5v\") pod \"llmisvc-controller-manager-7c5846b585-w4kcq\" (UID: \"3ae29a8f-5039-461a-846e-6215646c155b\") " pod="kserve/llmisvc-controller-manager-7c5846b585-w4kcq"
Sep 17 22:09:03.293670 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:09:03.293648    2555 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 17 22:09:03.293718 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:09:03.293708    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ae29a8f-5039-461a-846e-6215646c155b-cert podName:3ae29a8f-5039-461a-846e-6215646c155b nodeName:}" failed. No retries permitted until 2026-09-17 22:09:03.793693211 +0000 UTC m=+645.239738461 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3ae29a8f-5039-461a-846e-6215646c155b-cert") pod "llmisvc-controller-manager-7c5846b585-w4kcq" (UID: "3ae29a8f-5039-461a-846e-6215646c155b") : secret "llmisvc-webhook-server-cert" not found
Sep 17 22:09:03.293818 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:03.293802    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4c524947-6176-4f5b-88bc-2d971a15e0c1-data\") pod \"seaweedfs-64568bcd49-4hswx\" (UID: \"4c524947-6176-4f5b-88bc-2d971a15e0c1\") " pod="kserve/seaweedfs-64568bcd49-4hswx"
Sep 17 22:09:03.302035 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:03.302003    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqt5v\" (UniqueName: \"kubernetes.io/projected/3ae29a8f-5039-461a-846e-6215646c155b-kube-api-access-zqt5v\") pod \"llmisvc-controller-manager-7c5846b585-w4kcq\" (UID: \"3ae29a8f-5039-461a-846e-6215646c155b\") " pod="kserve/llmisvc-controller-manager-7c5846b585-w4kcq"
Sep 17 22:09:03.302161 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:03.302003    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbs4w\" (UniqueName: \"kubernetes.io/projected/4c524947-6176-4f5b-88bc-2d971a15e0c1-kube-api-access-kbs4w\") pod \"seaweedfs-64568bcd49-4hswx\" (UID: \"4c524947-6176-4f5b-88bc-2d971a15e0c1\") " pod="kserve/seaweedfs-64568bcd49-4hswx"
Sep 17 22:09:03.353775 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:03.353746    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-4hswx"
Sep 17 22:09:03.359407 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:03.359379    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:09:03.460722 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:03.460637    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:09:03.629446 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:03.629423    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-4hswx"]
Sep 17 22:09:03.630668 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:09:03.630640    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c524947_6176_4f5b_88bc_2d971a15e0c1.slice/crio-a54fe98683683cef04ad5aa8c56e0fc4b1bc9e570afbf569d41d56eac98dc188 WatchSource:0}: Error finding container a54fe98683683cef04ad5aa8c56e0fc4b1bc9e570afbf569d41d56eac98dc188: Status 404 returned error can't find the container with id a54fe98683683cef04ad5aa8c56e0fc4b1bc9e570afbf569d41d56eac98dc188
Sep 17 22:09:03.697345 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:03.697306    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f4e978e-9098-4418-9e88-4134dd961423-cert\") pod \"kserve-controller-manager-79b679679b-cjswc\" (UID: \"1f4e978e-9098-4418-9e88-4134dd961423\") " pod="kserve/kserve-controller-manager-79b679679b-cjswc"
Sep 17 22:09:03.699815 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:03.699788    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f4e978e-9098-4418-9e88-4134dd961423-cert\") pod \"kserve-controller-manager-79b679679b-cjswc\" (UID: \"1f4e978e-9098-4418-9e88-4134dd961423\") " pod="kserve/kserve-controller-manager-79b679679b-cjswc"
Sep 17 22:09:03.798638 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:03.798545    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ae29a8f-5039-461a-846e-6215646c155b-cert\") pod \"llmisvc-controller-manager-7c5846b585-w4kcq\" (UID: \"3ae29a8f-5039-461a-846e-6215646c155b\") " pod="kserve/llmisvc-controller-manager-7c5846b585-w4kcq"
Sep 17 22:09:03.800969 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:03.800948    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ae29a8f-5039-461a-846e-6215646c155b-cert\") pod \"llmisvc-controller-manager-7c5846b585-w4kcq\" (UID: \"3ae29a8f-5039-461a-846e-6215646c155b\") " pod="kserve/llmisvc-controller-manager-7c5846b585-w4kcq"
Sep 17 22:09:03.821237 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:03.821205    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:09:03.838924 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:03.838887    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-4hswx" event={"ID":"4c524947-6176-4f5b-88bc-2d971a15e0c1","Type":"ContainerStarted","Data":"a54fe98683683cef04ad5aa8c56e0fc4b1bc9e570afbf569d41d56eac98dc188"}
Sep 17 22:09:03.900249 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:03.900212    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-79b679679b-cjswc"
Sep 17 22:09:03.922001 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:03.921957    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-7c5846b585-w4kcq"
Sep 17 22:09:03.922443 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:03.922234    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:09:04.021522 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:04.021494    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:09:04.175285 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:04.175258    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-79b679679b-cjswc"]
Sep 17 22:09:04.176916 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:09:04.176891    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f4e978e_9098_4418_9e88_4134dd961423.slice/crio-b98df75bb6759ee74257c6d399a2b96dbdaee2fff9f217b0f331d31143a004d5 WatchSource:0}: Error finding container b98df75bb6759ee74257c6d399a2b96dbdaee2fff9f217b0f331d31143a004d5: Status 404 returned error can't find the container with id b98df75bb6759ee74257c6d399a2b96dbdaee2fff9f217b0f331d31143a004d5
Sep 17 22:09:04.206554 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:04.206435    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-7c5846b585-w4kcq"]
Sep 17 22:09:04.209208 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:09:04.209162    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3ae29a8f_5039_461a_846e_6215646c155b.slice/crio-2e122ea8d780d39a07a0672f0178cbe1b9c02b9a14bb37decdfbf452988caaf6 WatchSource:0}: Error finding container 2e122ea8d780d39a07a0672f0178cbe1b9c02b9a14bb37decdfbf452988caaf6: Status 404 returned error can't find the container with id 2e122ea8d780d39a07a0672f0178cbe1b9c02b9a14bb37decdfbf452988caaf6
Sep 17 22:09:04.852545 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:04.852506    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-7c5846b585-w4kcq" event={"ID":"3ae29a8f-5039-461a-846e-6215646c155b","Type":"ContainerStarted","Data":"2e122ea8d780d39a07a0672f0178cbe1b9c02b9a14bb37decdfbf452988caaf6"}
Sep 17 22:09:04.854652 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:04.854618    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-79b679679b-cjswc" event={"ID":"1f4e978e-9098-4418-9e88-4134dd961423","Type":"ContainerStarted","Data":"b98df75bb6759ee74257c6d399a2b96dbdaee2fff9f217b0f331d31143a004d5"}
Sep 17 22:09:08.888901 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:08.888874    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-79b679679b-cjswc" event={"ID":"1f4e978e-9098-4418-9e88-4134dd961423","Type":"ContainerStarted","Data":"d332b20bfaed92bf35ed1ae07213ca4fc1d4e2d4691cebacbfe17d39513d066a"}
Sep 17 22:09:08.889315 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:08.889101    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-79b679679b-cjswc"
Sep 17 22:09:08.890945 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:08.890914    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-4hswx" event={"ID":"4c524947-6176-4f5b-88bc-2d971a15e0c1","Type":"ContainerStarted","Data":"8bb844f4442c65824dc621f3401cd3b89cc05d9edc0516428d8723f472b22919"}
Sep 17 22:09:08.891046 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:08.891026    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-4hswx"
Sep 17 22:09:08.893398 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:08.893348    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-7c5846b585-w4kcq"
Sep 17 22:09:08.904696 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:08.904642    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-79b679679b-cjswc" podStartSLOduration=2.37678103 podStartE2EDuration="6.90462765s" totalImagesPullingTime="4.52784662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:09:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:09:08.902557695 +0000 UTC m=+650.348602967" watchObservedRunningTime="2026-09-17 22:09:08.90462765 +0000 UTC m=+650.350672921"
Sep 17 22:09:08.917704 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:08.917659    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-7c5846b585-w4kcq" podStartSLOduration=2.467272444 podStartE2EDuration="6.917646063s" totalImagesPullingTime="4.450373619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:09:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:09:08.916203926 +0000 UTC m=+650.362249196" watchObservedRunningTime="2026-09-17 22:09:08.917646063 +0000 UTC m=+650.363691348"
Sep 17 22:09:08.930640 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:08.930599    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-4hswx" podStartSLOduration=0.766411924 podStartE2EDuration="5.93058454s" totalImagesPullingTime="5.164172616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:09:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:09:08.929582399 +0000 UTC m=+650.375627671" watchObservedRunningTime="2026-09-17 22:09:08.93058454 +0000 UTC m=+650.376629812"
Sep 17 22:09:09.900246 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:09.900208    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-7c5846b585-w4kcq" event={"ID":"3ae29a8f-5039-461a-846e-6215646c155b","Type":"ContainerStarted","Data":"7a18cea63bbb70068ae5bcf4fc88cafe0e042a924ddc11941fff76088d38d9e5"}
Sep 17 22:09:14.903066 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:14.903036    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-4hswx"
Sep 17 22:09:39.905638 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:39.905607    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-79b679679b-cjswc"
Sep 17 22:09:40.907996 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:40.907967    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-7c5846b585-w4kcq"
Sep 17 22:09:47.958263 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:47.958226    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-79b679679b-cjswc"]
Sep 17 22:09:47.958670 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:47.958435    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-79b679679b-cjswc" podUID="1f4e978e-9098-4418-9e88-4134dd961423" containerName="manager" containerID="cri-o://d332b20bfaed92bf35ed1ae07213ca4fc1d4e2d4691cebacbfe17d39513d066a" gracePeriod=10
Sep 17 22:09:47.985057 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:47.985030    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-79b679679b-67wr2"]
Sep 17 22:09:47.987684 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:47.987669    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-79b679679b-67wr2"
Sep 17 22:09:47.996390 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:47.996367    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-79b679679b-67wr2"]
Sep 17 22:09:48.083842 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:48.083816    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c8e4a5c2-827d-4191-9e34-8ce6b6ffda58-cert\") pod \"kserve-controller-manager-79b679679b-67wr2\" (UID: \"c8e4a5c2-827d-4191-9e34-8ce6b6ffda58\") " pod="kserve/kserve-controller-manager-79b679679b-67wr2"
Sep 17 22:09:48.083947 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:48.083851    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjszj\" (UniqueName: \"kubernetes.io/projected/c8e4a5c2-827d-4191-9e34-8ce6b6ffda58-kube-api-access-wjszj\") pod \"kserve-controller-manager-79b679679b-67wr2\" (UID: \"c8e4a5c2-827d-4191-9e34-8ce6b6ffda58\") " pod="kserve/kserve-controller-manager-79b679679b-67wr2"
Sep 17 22:09:48.100325 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:48.100296    2555 generic.go:356] "Generic (PLEG): container finished" podID="1f4e978e-9098-4418-9e88-4134dd961423" containerID="d332b20bfaed92bf35ed1ae07213ca4fc1d4e2d4691cebacbfe17d39513d066a" exitCode=0
Sep 17 22:09:48.100444 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:48.100364    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-79b679679b-cjswc" event={"ID":"1f4e978e-9098-4418-9e88-4134dd961423","Type":"ContainerDied","Data":"d332b20bfaed92bf35ed1ae07213ca4fc1d4e2d4691cebacbfe17d39513d066a"}
Sep 17 22:09:48.185149 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:48.185109    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c8e4a5c2-827d-4191-9e34-8ce6b6ffda58-cert\") pod \"kserve-controller-manager-79b679679b-67wr2\" (UID: \"c8e4a5c2-827d-4191-9e34-8ce6b6ffda58\") " pod="kserve/kserve-controller-manager-79b679679b-67wr2"
Sep 17 22:09:48.185325 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:48.185232    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjszj\" (UniqueName: \"kubernetes.io/projected/c8e4a5c2-827d-4191-9e34-8ce6b6ffda58-kube-api-access-wjszj\") pod \"kserve-controller-manager-79b679679b-67wr2\" (UID: \"c8e4a5c2-827d-4191-9e34-8ce6b6ffda58\") " pod="kserve/kserve-controller-manager-79b679679b-67wr2"
Sep 17 22:09:48.187664 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:48.187641    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c8e4a5c2-827d-4191-9e34-8ce6b6ffda58-cert\") pod \"kserve-controller-manager-79b679679b-67wr2\" (UID: \"c8e4a5c2-827d-4191-9e34-8ce6b6ffda58\") " pod="kserve/kserve-controller-manager-79b679679b-67wr2"
Sep 17 22:09:48.192437 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:48.192419    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjszj\" (UniqueName: \"kubernetes.io/projected/c8e4a5c2-827d-4191-9e34-8ce6b6ffda58-kube-api-access-wjszj\") pod \"kserve-controller-manager-79b679679b-67wr2\" (UID: \"c8e4a5c2-827d-4191-9e34-8ce6b6ffda58\") " pod="kserve/kserve-controller-manager-79b679679b-67wr2"
Sep 17 22:09:48.208006 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:48.207985    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-79b679679b-cjswc"
Sep 17 22:09:48.327968 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:48.327872    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-79b679679b-67wr2"
Sep 17 22:09:48.387266 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:48.386909    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f4e978e-9098-4418-9e88-4134dd961423-cert\") pod \"1f4e978e-9098-4418-9e88-4134dd961423\" (UID: \"1f4e978e-9098-4418-9e88-4134dd961423\") "
Sep 17 22:09:48.387266 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:48.386955    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbqgk\" (UniqueName: \"kubernetes.io/projected/1f4e978e-9098-4418-9e88-4134dd961423-kube-api-access-gbqgk\") pod \"1f4e978e-9098-4418-9e88-4134dd961423\" (UID: \"1f4e978e-9098-4418-9e88-4134dd961423\") "
Sep 17 22:09:48.389926 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:48.389895    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f4e978e-9098-4418-9e88-4134dd961423-kube-api-access-gbqgk" (OuterVolumeSpecName: "kube-api-access-gbqgk") pod "1f4e978e-9098-4418-9e88-4134dd961423" (UID: "1f4e978e-9098-4418-9e88-4134dd961423"). InnerVolumeSpecName "kube-api-access-gbqgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:09:48.390027 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:48.389920    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f4e978e-9098-4418-9e88-4134dd961423-cert" (OuterVolumeSpecName: "cert") pod "1f4e978e-9098-4418-9e88-4134dd961423" (UID: "1f4e978e-9098-4418-9e88-4134dd961423"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:09:48.488513 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:48.488485    2555 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f4e978e-9098-4418-9e88-4134dd961423-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:09:48.488513 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:48.488508    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbqgk\" (UniqueName: \"kubernetes.io/projected/1f4e978e-9098-4418-9e88-4134dd961423-kube-api-access-gbqgk\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:09:48.599154 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:48.599129    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-79b679679b-67wr2"]
Sep 17 22:09:48.600986 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:09:48.600960    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e4a5c2_827d_4191_9e34_8ce6b6ffda58.slice/crio-0c73d48fe270063e71a93b0945a1963fc0bf0462d53677e0ada98edb7742ca79 WatchSource:0}: Error finding container 0c73d48fe270063e71a93b0945a1963fc0bf0462d53677e0ada98edb7742ca79: Status 404 returned error can't find the container with id 0c73d48fe270063e71a93b0945a1963fc0bf0462d53677e0ada98edb7742ca79
Sep 17 22:09:48.602397 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:48.602380    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 22:09:49.107818 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:49.107783    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-79b679679b-67wr2" event={"ID":"c8e4a5c2-827d-4191-9e34-8ce6b6ffda58","Type":"ContainerStarted","Data":"bf0794f8fe8d29734ad57f18a5bb4efbc6b40b13f429462903974cd73b478aa2"}
Sep 17 22:09:49.108307 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:49.107856    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-79b679679b-67wr2"
Sep 17 22:09:49.108307 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:49.107885    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-79b679679b-67wr2" event={"ID":"c8e4a5c2-827d-4191-9e34-8ce6b6ffda58","Type":"ContainerStarted","Data":"0c73d48fe270063e71a93b0945a1963fc0bf0462d53677e0ada98edb7742ca79"}
Sep 17 22:09:49.109338 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:49.109317    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-79b679679b-cjswc"
Sep 17 22:09:49.109471 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:49.109334    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-79b679679b-cjswc" event={"ID":"1f4e978e-9098-4418-9e88-4134dd961423","Type":"ContainerDied","Data":"b98df75bb6759ee74257c6d399a2b96dbdaee2fff9f217b0f331d31143a004d5"}
Sep 17 22:09:49.109471 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:49.109384    2555 scope.go:117] "RemoveContainer" containerID="d332b20bfaed92bf35ed1ae07213ca4fc1d4e2d4691cebacbfe17d39513d066a"
Sep 17 22:09:49.127654 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:49.127611    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-79b679679b-67wr2" podStartSLOduration=1.793523059 podStartE2EDuration="2.127596785s" totalImagesPullingTime="334.073726ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:09:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:09:49.12428028 +0000 UTC m=+690.570325553" watchObservedRunningTime="2026-09-17 22:09:49.127596785 +0000 UTC m=+690.573642056"
Sep 17 22:09:49.140018 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:49.139991    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-79b679679b-cjswc"]
Sep 17 22:09:49.144555 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:49.144534    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-79b679679b-cjswc"]
Sep 17 22:09:50.679364 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:09:50.679333    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f4e978e-9098-4418-9e88-4134dd961423" path="/var/lib/kubelet/pods/1f4e978e-9098-4418-9e88-4134dd961423/volumes"
Sep 17 22:10:20.120533 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:20.120442    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-79b679679b-67wr2"
Sep 17 22:10:21.385804 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:21.385762    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:10:21.431645 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:21.431616    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-cwvkh"]
Sep 17 22:10:21.431984 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:21.431972    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f4e978e-9098-4418-9e88-4134dd961423" containerName="manager"
Sep 17 22:10:21.432030 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:21.431986    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f4e978e-9098-4418-9e88-4134dd961423" containerName="manager"
Sep 17 22:10:21.432065 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:21.432039    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f4e978e-9098-4418-9e88-4134dd961423" containerName="manager"
Sep 17 22:10:21.436391 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:21.436371    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-cwvkh"
Sep 17 22:10:21.438548 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:21.438528    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 17 22:10:21.438652 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:21.438530    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-kz8gv\""
Sep 17 22:10:21.444391 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:21.444368    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-vsj9h"]
Sep 17 22:10:21.447665 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:21.447650    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-vsj9h"
Sep 17 22:10:21.448630 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:21.448609    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-cwvkh"]
Sep 17 22:10:21.449740 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:21.449720    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 17 22:10:21.449849 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:21.449753    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-jzcvd\""
Sep 17 22:10:21.449945 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:21.449922    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 17 22:10:21.464145 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:21.464123    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-vsj9h"]
Sep 17 22:10:21.564262 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:21.564233    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpqsr\" (UniqueName: \"kubernetes.io/projected/eb0ca825-2d2f-4bd7-a6eb-34c2d8b09f84-kube-api-access-vpqsr\") pod \"model-serving-api-7b954ffc95-cwvkh\" (UID: \"eb0ca825-2d2f-4bd7-a6eb-34c2d8b09f84\") " pod="kserve/model-serving-api-7b954ffc95-cwvkh"
Sep 17 22:10:21.564512 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:21.564276    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdfe5af3-14c1-48b0-957c-b8f88779fc27-metrics-certs\") pod \"odh-model-controller-7b966d7984-vsj9h\" (UID: \"bdfe5af3-14c1-48b0-957c-b8f88779fc27\") " pod="kserve/odh-model-controller-7b966d7984-vsj9h"
Sep 17 22:10:21.564512 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:21.564301    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbjvf\" (UniqueName: \"kubernetes.io/projected/bdfe5af3-14c1-48b0-957c-b8f88779fc27-kube-api-access-qbjvf\") pod \"odh-model-controller-7b966d7984-vsj9h\" (UID: \"bdfe5af3-14c1-48b0-957c-b8f88779fc27\") " pod="kserve/odh-model-controller-7b966d7984-vsj9h"
Sep 17 22:10:21.564512 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:21.564344    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bdfe5af3-14c1-48b0-957c-b8f88779fc27-cert\") pod \"odh-model-controller-7b966d7984-vsj9h\" (UID: \"bdfe5af3-14c1-48b0-957c-b8f88779fc27\") " pod="kserve/odh-model-controller-7b966d7984-vsj9h"
Sep 17 22:10:21.564512 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:21.564369    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb0ca825-2d2f-4bd7-a6eb-34c2d8b09f84-tls-certs\") pod \"model-serving-api-7b954ffc95-cwvkh\" (UID: \"eb0ca825-2d2f-4bd7-a6eb-34c2d8b09f84\") " pod="kserve/model-serving-api-7b954ffc95-cwvkh"
Sep 17 22:10:21.665713 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:21.665628    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpqsr\" (UniqueName: \"kubernetes.io/projected/eb0ca825-2d2f-4bd7-a6eb-34c2d8b09f84-kube-api-access-vpqsr\") pod \"model-serving-api-7b954ffc95-cwvkh\" (UID: \"eb0ca825-2d2f-4bd7-a6eb-34c2d8b09f84\") " pod="kserve/model-serving-api-7b954ffc95-cwvkh"
Sep 17 22:10:21.665713 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:21.665677    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdfe5af3-14c1-48b0-957c-b8f88779fc27-metrics-certs\") pod \"odh-model-controller-7b966d7984-vsj9h\" (UID: \"bdfe5af3-14c1-48b0-957c-b8f88779fc27\") " pod="kserve/odh-model-controller-7b966d7984-vsj9h"
Sep 17 22:10:21.665713 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:21.665701    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbjvf\" (UniqueName: \"kubernetes.io/projected/bdfe5af3-14c1-48b0-957c-b8f88779fc27-kube-api-access-qbjvf\") pod \"odh-model-controller-7b966d7984-vsj9h\" (UID: \"bdfe5af3-14c1-48b0-957c-b8f88779fc27\") " pod="kserve/odh-model-controller-7b966d7984-vsj9h"
Sep 17 22:10:21.665982 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:21.665720    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bdfe5af3-14c1-48b0-957c-b8f88779fc27-cert\") pod \"odh-model-controller-7b966d7984-vsj9h\" (UID: \"bdfe5af3-14c1-48b0-957c-b8f88779fc27\") " pod="kserve/odh-model-controller-7b966d7984-vsj9h"
Sep 17 22:10:21.665982 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:21.665749    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb0ca825-2d2f-4bd7-a6eb-34c2d8b09f84-tls-certs\") pod \"model-serving-api-7b954ffc95-cwvkh\" (UID: \"eb0ca825-2d2f-4bd7-a6eb-34c2d8b09f84\") " pod="kserve/model-serving-api-7b954ffc95-cwvkh"
Sep 17 22:10:21.665982 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:10:21.665861    2555 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 17 22:10:21.665982 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:10:21.665920    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdfe5af3-14c1-48b0-957c-b8f88779fc27-cert podName:bdfe5af3-14c1-48b0-957c-b8f88779fc27 nodeName:}" failed. No retries permitted until 2026-09-17 22:10:22.165904162 +0000 UTC m=+723.611949416 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/bdfe5af3-14c1-48b0-957c-b8f88779fc27-cert") pod "odh-model-controller-7b966d7984-vsj9h" (UID: "bdfe5af3-14c1-48b0-957c-b8f88779fc27") : secret "odh-model-controller-webhook-cert" not found
Sep 17 22:10:21.668175 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:21.668147    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdfe5af3-14c1-48b0-957c-b8f88779fc27-metrics-certs\") pod \"odh-model-controller-7b966d7984-vsj9h\" (UID: \"bdfe5af3-14c1-48b0-957c-b8f88779fc27\") " pod="kserve/odh-model-controller-7b966d7984-vsj9h"
Sep 17 22:10:21.668367 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:21.668347    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb0ca825-2d2f-4bd7-a6eb-34c2d8b09f84-tls-certs\") pod \"model-serving-api-7b954ffc95-cwvkh\" (UID: \"eb0ca825-2d2f-4bd7-a6eb-34c2d8b09f84\") " pod="kserve/model-serving-api-7b954ffc95-cwvkh"
Sep 17 22:10:21.674793 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:21.674767    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpqsr\" (UniqueName: \"kubernetes.io/projected/eb0ca825-2d2f-4bd7-a6eb-34c2d8b09f84-kube-api-access-vpqsr\") pod \"model-serving-api-7b954ffc95-cwvkh\" (UID: \"eb0ca825-2d2f-4bd7-a6eb-34c2d8b09f84\") " pod="kserve/model-serving-api-7b954ffc95-cwvkh"
Sep 17 22:10:21.674900 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:21.674890    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbjvf\" (UniqueName: \"kubernetes.io/projected/bdfe5af3-14c1-48b0-957c-b8f88779fc27-kube-api-access-qbjvf\") pod \"odh-model-controller-7b966d7984-vsj9h\" (UID: \"bdfe5af3-14c1-48b0-957c-b8f88779fc27\") " pod="kserve/odh-model-controller-7b966d7984-vsj9h"
Sep 17 22:10:21.747131 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:21.747098    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-cwvkh"
Sep 17 22:10:22.022418 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:22.022391    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-cwvkh"]
Sep 17 22:10:22.024132 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:10:22.024108    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb0ca825_2d2f_4bd7_a6eb_34c2d8b09f84.slice/crio-a82c6c56a25a17d15cb76db7f3fdc20c035d54a5726e6dd2580e0e666252b11a WatchSource:0}: Error finding container a82c6c56a25a17d15cb76db7f3fdc20c035d54a5726e6dd2580e0e666252b11a: Status 404 returned error can't find the container with id a82c6c56a25a17d15cb76db7f3fdc20c035d54a5726e6dd2580e0e666252b11a
Sep 17 22:10:22.169904 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:22.169875    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bdfe5af3-14c1-48b0-957c-b8f88779fc27-cert\") pod \"odh-model-controller-7b966d7984-vsj9h\" (UID: \"bdfe5af3-14c1-48b0-957c-b8f88779fc27\") " pod="kserve/odh-model-controller-7b966d7984-vsj9h"
Sep 17 22:10:22.172266 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:22.172244    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bdfe5af3-14c1-48b0-957c-b8f88779fc27-cert\") pod \"odh-model-controller-7b966d7984-vsj9h\" (UID: \"bdfe5af3-14c1-48b0-957c-b8f88779fc27\") " pod="kserve/odh-model-controller-7b966d7984-vsj9h"
Sep 17 22:10:22.281577 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:22.281510    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-cwvkh" event={"ID":"eb0ca825-2d2f-4bd7-a6eb-34c2d8b09f84","Type":"ContainerStarted","Data":"a82c6c56a25a17d15cb76db7f3fdc20c035d54a5726e6dd2580e0e666252b11a"}
Sep 17 22:10:22.364870 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:22.364844    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-vsj9h"
Sep 17 22:10:22.639176 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:22.639148    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-vsj9h"]
Sep 17 22:10:22.641169 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:10:22.641137    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdfe5af3_14c1_48b0_957c_b8f88779fc27.slice/crio-f5fc480b053642d8ba01fe451d0387af6cf26491b68177adbaf4b1887e2e8d2a WatchSource:0}: Error finding container f5fc480b053642d8ba01fe451d0387af6cf26491b68177adbaf4b1887e2e8d2a: Status 404 returned error can't find the container with id f5fc480b053642d8ba01fe451d0387af6cf26491b68177adbaf4b1887e2e8d2a
Sep 17 22:10:23.064924 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:23.064771    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:10:23.292159 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:23.292119    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-vsj9h" event={"ID":"bdfe5af3-14c1-48b0-957c-b8f88779fc27","Type":"ContainerStarted","Data":"f5fc480b053642d8ba01fe451d0387af6cf26491b68177adbaf4b1887e2e8d2a"}
Sep 17 22:10:25.306854 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:25.306816    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-cwvkh" event={"ID":"eb0ca825-2d2f-4bd7-a6eb-34c2d8b09f84","Type":"ContainerStarted","Data":"73fa8fef3ea759300fa34fcc097be143b3edadb6b12a9ad72193bd692f36ed3e"}
Sep 17 22:10:25.307354 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:25.306941    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-cwvkh"
Sep 17 22:10:25.322605 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:25.322349    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-cwvkh" podStartSLOduration=1.934445008 podStartE2EDuration="4.322333464s" totalImagesPullingTime="2.387888456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:10:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:10:25.321851804 +0000 UTC m=+726.767897078" watchObservedRunningTime="2026-09-17 22:10:25.322333464 +0000 UTC m=+726.768378730"
Sep 17 22:10:26.314314 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:26.314278    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-vsj9h" event={"ID":"bdfe5af3-14c1-48b0-957c-b8f88779fc27","Type":"ContainerStarted","Data":"992c6531cb107b5f4403c54c51b71a42ff641d3fa4ef873a4eaacacd1e1e3148"}
Sep 17 22:10:26.332791 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:26.332738    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-vsj9h" podStartSLOduration=2.16795186 podStartE2EDuration="5.332722671s" totalImagesPullingTime="3.164770811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:10:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:10:26.330089584 +0000 UTC m=+727.776134870" watchObservedRunningTime="2026-09-17 22:10:26.332722671 +0000 UTC m=+727.778767962"
Sep 17 22:10:27.319916 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:27.319885    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-vsj9h"
Sep 17 22:10:29.944601 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:29.944571    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:10:36.318910 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:36.318877    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-cwvkh"
Sep 17 22:10:38.327303 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:38.327274    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-vsj9h"
Sep 17 22:10:39.135074 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:39.135041    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-2q2fq"]
Sep 17 22:10:39.138737 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:39.138721    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-2q2fq"
Sep 17 22:10:39.140904 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:39.140887    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 22:10:39.145273 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:39.145244    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-2q2fq"]
Sep 17 22:10:39.320627 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:39.320602    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvbk7\" (UniqueName: \"kubernetes.io/projected/a3815a4c-d508-4bda-bb25-9ed3166b9332-kube-api-access-qvbk7\") pod \"s3-init-2q2fq\" (UID: \"a3815a4c-d508-4bda-bb25-9ed3166b9332\") " pod="kserve/s3-init-2q2fq"
Sep 17 22:10:39.320776 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:39.320658    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3815a4c-d508-4bda-bb25-9ed3166b9332-model-cache\") pod \"s3-init-2q2fq\" (UID: \"a3815a4c-d508-4bda-bb25-9ed3166b9332\") " pod="kserve/s3-init-2q2fq"
Sep 17 22:10:39.421673 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:39.421593    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3815a4c-d508-4bda-bb25-9ed3166b9332-model-cache\") pod \"s3-init-2q2fq\" (UID: \"a3815a4c-d508-4bda-bb25-9ed3166b9332\") " pod="kserve/s3-init-2q2fq"
Sep 17 22:10:39.421673 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:39.421662    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvbk7\" (UniqueName: \"kubernetes.io/projected/a3815a4c-d508-4bda-bb25-9ed3166b9332-kube-api-access-qvbk7\") pod \"s3-init-2q2fq\" (UID: \"a3815a4c-d508-4bda-bb25-9ed3166b9332\") " pod="kserve/s3-init-2q2fq"
Sep 17 22:10:39.422100 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:39.422013    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3815a4c-d508-4bda-bb25-9ed3166b9332-model-cache\") pod \"s3-init-2q2fq\" (UID: \"a3815a4c-d508-4bda-bb25-9ed3166b9332\") " pod="kserve/s3-init-2q2fq"
Sep 17 22:10:39.429721 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:39.429695    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvbk7\" (UniqueName: \"kubernetes.io/projected/a3815a4c-d508-4bda-bb25-9ed3166b9332-kube-api-access-qvbk7\") pod \"s3-init-2q2fq\" (UID: \"a3815a4c-d508-4bda-bb25-9ed3166b9332\") " pod="kserve/s3-init-2q2fq"
Sep 17 22:10:39.448602 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:39.448580    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-2q2fq"
Sep 17 22:10:39.719771 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:39.719743    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-2q2fq"]
Sep 17 22:10:39.721139 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:10:39.721112    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3815a4c_d508_4bda_bb25_9ed3166b9332.slice/crio-581b6f9de587726a289023020ac512c89985776e843d3b96a28b5aab705e0235 WatchSource:0}: Error finding container 581b6f9de587726a289023020ac512c89985776e843d3b96a28b5aab705e0235: Status 404 returned error can't find the container with id 581b6f9de587726a289023020ac512c89985776e843d3b96a28b5aab705e0235
Sep 17 22:10:40.392118 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:40.392065    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2q2fq" event={"ID":"a3815a4c-d508-4bda-bb25-9ed3166b9332","Type":"ContainerStarted","Data":"581b6f9de587726a289023020ac512c89985776e843d3b96a28b5aab705e0235"}
Sep 17 22:10:43.414051 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:43.414015    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2q2fq" event={"ID":"a3815a4c-d508-4bda-bb25-9ed3166b9332","Type":"ContainerStarted","Data":"98828ad88a7fa2d18475d0d05a390d725f590d966fb8386720219ad6de240dfd"}
Sep 17 22:10:45.426652 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:45.426565    2555 generic.go:356] "Generic (PLEG): container finished" podID="a3815a4c-d508-4bda-bb25-9ed3166b9332" containerID="98828ad88a7fa2d18475d0d05a390d725f590d966fb8386720219ad6de240dfd" exitCode=0
Sep 17 22:10:45.426652 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:45.426635    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2q2fq" event={"ID":"a3815a4c-d508-4bda-bb25-9ed3166b9332","Type":"ContainerDied","Data":"98828ad88a7fa2d18475d0d05a390d725f590d966fb8386720219ad6de240dfd"}
Sep 17 22:10:50.462350 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:50.462318    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2q2fq" event={"ID":"a3815a4c-d508-4bda-bb25-9ed3166b9332","Type":"ContainerStarted","Data":"2977343d113214de4cefa0df25177a56651d313c1fe696d3b988f73d08df4026"}
Sep 17 22:10:50.481891 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:10:50.481846    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-2q2fq" podStartSLOduration=1.7436588899999999 podStartE2EDuration="11.481831573s" totalImagesPullingTime="7.844702977s" totalInitContainerRuntime="1.893469706s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:10:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:10:50.480992971 +0000 UTC m=+751.927038245" watchObservedRunningTime="2026-09-17 22:10:50.481831573 +0000 UTC m=+751.927876845"
Sep 17 22:11:04.485496 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:11:04.485455    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:11:41.013011 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:11:41.012929    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:11:55.825001 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:11:55.824962    2555 generic.go:356] "Generic (PLEG): container finished" podID="a3815a4c-d508-4bda-bb25-9ed3166b9332" containerID="2977343d113214de4cefa0df25177a56651d313c1fe696d3b988f73d08df4026" exitCode=0
Sep 17 22:11:55.825427 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:11:55.825029    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2q2fq" event={"ID":"a3815a4c-d508-4bda-bb25-9ed3166b9332","Type":"ContainerDied","Data":"2977343d113214de4cefa0df25177a56651d313c1fe696d3b988f73d08df4026"}
Sep 17 22:11:56.959594 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:11:56.959568    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-2q2fq"
Sep 17 22:11:56.979309 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:11:56.979277    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvbk7\" (UniqueName: \"kubernetes.io/projected/a3815a4c-d508-4bda-bb25-9ed3166b9332-kube-api-access-qvbk7\") pod \"a3815a4c-d508-4bda-bb25-9ed3166b9332\" (UID: \"a3815a4c-d508-4bda-bb25-9ed3166b9332\") "
Sep 17 22:11:56.979475 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:11:56.979350    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3815a4c-d508-4bda-bb25-9ed3166b9332-model-cache\") pod \"a3815a4c-d508-4bda-bb25-9ed3166b9332\" (UID: \"a3815a4c-d508-4bda-bb25-9ed3166b9332\") "
Sep 17 22:11:56.981468 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:11:56.981438    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3815a4c-d508-4bda-bb25-9ed3166b9332-kube-api-access-qvbk7" (OuterVolumeSpecName: "kube-api-access-qvbk7") pod "a3815a4c-d508-4bda-bb25-9ed3166b9332" (UID: "a3815a4c-d508-4bda-bb25-9ed3166b9332"). InnerVolumeSpecName "kube-api-access-qvbk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:11:56.999375 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:11:56.999344    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3815a4c-d508-4bda-bb25-9ed3166b9332-model-cache" (OuterVolumeSpecName: "model-cache") pod "a3815a4c-d508-4bda-bb25-9ed3166b9332" (UID: "a3815a4c-d508-4bda-bb25-9ed3166b9332"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:11:57.080042 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:11:57.080004    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3815a4c-d508-4bda-bb25-9ed3166b9332-model-cache\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:11:57.080042 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:11:57.080043    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvbk7\" (UniqueName: \"kubernetes.io/projected/a3815a4c-d508-4bda-bb25-9ed3166b9332-kube-api-access-qvbk7\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:11:57.838286 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:11:57.838257    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-2q2fq"
Sep 17 22:11:57.838464 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:11:57.838289    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2q2fq" event={"ID":"a3815a4c-d508-4bda-bb25-9ed3166b9332","Type":"ContainerDied","Data":"581b6f9de587726a289023020ac512c89985776e843d3b96a28b5aab705e0235"}
Sep 17 22:11:57.838464 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:11:57.838322    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="581b6f9de587726a289023020ac512c89985776e843d3b96a28b5aab705e0235"
Sep 17 22:11:58.200632 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:11:58.200600    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:11:58.300828 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:11:58.300784    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:12:11.625480 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:11.625440    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:12:11.718899 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:11.718861    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:12:29.440853 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:29.440816    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-cwtd8"]
Sep 17 22:12:29.441375 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:29.441354    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3815a4c-d508-4bda-bb25-9ed3166b9332" containerName="download-hf-model"
Sep 17 22:12:29.441450 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:29.441379    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3815a4c-d508-4bda-bb25-9ed3166b9332" containerName="download-hf-model"
Sep 17 22:12:29.441450 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:29.441408    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3815a4c-d508-4bda-bb25-9ed3166b9332" containerName="s3-init"
Sep 17 22:12:29.441450 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:29.441417    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3815a4c-d508-4bda-bb25-9ed3166b9332" containerName="s3-init"
Sep 17 22:12:29.441600 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:29.441518    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3815a4c-d508-4bda-bb25-9ed3166b9332" containerName="s3-init"
Sep 17 22:12:29.444523 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:29.444503    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-cwtd8"
Sep 17 22:12:29.446974 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:29.446949    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 17 22:12:29.447092 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:29.446987    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-w4qqf\""
Sep 17 22:12:29.447092 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:29.447057    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 17 22:12:29.447092 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:29.447059    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 17 22:12:29.453283 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:29.452817    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-cwtd8"]
Sep 17 22:12:29.475154 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:29.475074    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:12:29.543987 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:29.543956    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa4070e1-9ebc-457e-8eb3-504cf9271dcc-home\") pod \"fc-smoke-test-kserve-7f4687897-cwtd8\" (UID: \"aa4070e1-9ebc-457e-8eb3-504cf9271dcc\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-cwtd8"
Sep 17 22:12:29.543987 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:29.543987    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa4070e1-9ebc-457e-8eb3-504cf9271dcc-dshm\") pod \"fc-smoke-test-kserve-7f4687897-cwtd8\" (UID: \"aa4070e1-9ebc-457e-8eb3-504cf9271dcc\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-cwtd8"
Sep 17 22:12:29.544212 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:29.544010    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa4070e1-9ebc-457e-8eb3-504cf9271dcc-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-cwtd8\" (UID: \"aa4070e1-9ebc-457e-8eb3-504cf9271dcc\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-cwtd8"
Sep 17 22:12:29.544212 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:29.544124    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa4070e1-9ebc-457e-8eb3-504cf9271dcc-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-cwtd8\" (UID: \"aa4070e1-9ebc-457e-8eb3-504cf9271dcc\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-cwtd8"
Sep 17 22:12:29.544212 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:29.544152    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa4070e1-9ebc-457e-8eb3-504cf9271dcc-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-cwtd8\" (UID: \"aa4070e1-9ebc-457e-8eb3-504cf9271dcc\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-cwtd8"
Sep 17 22:12:29.544313 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:29.544253    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsrvl\" (UniqueName: \"kubernetes.io/projected/aa4070e1-9ebc-457e-8eb3-504cf9271dcc-kube-api-access-dsrvl\") pod \"fc-smoke-test-kserve-7f4687897-cwtd8\" (UID: \"aa4070e1-9ebc-457e-8eb3-504cf9271dcc\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-cwtd8"
Sep 17 22:12:29.644817 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:29.644779    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa4070e1-9ebc-457e-8eb3-504cf9271dcc-home\") pod \"fc-smoke-test-kserve-7f4687897-cwtd8\" (UID: \"aa4070e1-9ebc-457e-8eb3-504cf9271dcc\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-cwtd8"
Sep 17 22:12:29.644817 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:29.644819    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa4070e1-9ebc-457e-8eb3-504cf9271dcc-dshm\") pod \"fc-smoke-test-kserve-7f4687897-cwtd8\" (UID: \"aa4070e1-9ebc-457e-8eb3-504cf9271dcc\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-cwtd8"
Sep 17 22:12:29.645073 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:29.644850    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa4070e1-9ebc-457e-8eb3-504cf9271dcc-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-cwtd8\" (UID: \"aa4070e1-9ebc-457e-8eb3-504cf9271dcc\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-cwtd8"
Sep 17 22:12:29.645073 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:29.644905    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa4070e1-9ebc-457e-8eb3-504cf9271dcc-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-cwtd8\" (UID: \"aa4070e1-9ebc-457e-8eb3-504cf9271dcc\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-cwtd8"
Sep 17 22:12:29.645073 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:29.644928    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa4070e1-9ebc-457e-8eb3-504cf9271dcc-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-cwtd8\" (UID: \"aa4070e1-9ebc-457e-8eb3-504cf9271dcc\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-cwtd8"
Sep 17 22:12:29.645073 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:29.644971    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsrvl\" (UniqueName: \"kubernetes.io/projected/aa4070e1-9ebc-457e-8eb3-504cf9271dcc-kube-api-access-dsrvl\") pod \"fc-smoke-test-kserve-7f4687897-cwtd8\" (UID: \"aa4070e1-9ebc-457e-8eb3-504cf9271dcc\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-cwtd8"
Sep 17 22:12:29.645315 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:29.645216    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa4070e1-9ebc-457e-8eb3-504cf9271dcc-home\") pod \"fc-smoke-test-kserve-7f4687897-cwtd8\" (UID: \"aa4070e1-9ebc-457e-8eb3-504cf9271dcc\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-cwtd8"
Sep 17 22:12:29.645315 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:29.645271    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa4070e1-9ebc-457e-8eb3-504cf9271dcc-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-cwtd8\" (UID: \"aa4070e1-9ebc-457e-8eb3-504cf9271dcc\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-cwtd8"
Sep 17 22:12:29.645405 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:29.645332    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa4070e1-9ebc-457e-8eb3-504cf9271dcc-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-cwtd8\" (UID: \"aa4070e1-9ebc-457e-8eb3-504cf9271dcc\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-cwtd8"
Sep 17 22:12:29.647136 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:29.647110    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa4070e1-9ebc-457e-8eb3-504cf9271dcc-dshm\") pod \"fc-smoke-test-kserve-7f4687897-cwtd8\" (UID: \"aa4070e1-9ebc-457e-8eb3-504cf9271dcc\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-cwtd8"
Sep 17 22:12:29.647530 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:29.647509    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa4070e1-9ebc-457e-8eb3-504cf9271dcc-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-cwtd8\" (UID: \"aa4070e1-9ebc-457e-8eb3-504cf9271dcc\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-cwtd8"
Sep 17 22:12:29.652581 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:29.652555    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsrvl\" (UniqueName: \"kubernetes.io/projected/aa4070e1-9ebc-457e-8eb3-504cf9271dcc-kube-api-access-dsrvl\") pod \"fc-smoke-test-kserve-7f4687897-cwtd8\" (UID: \"aa4070e1-9ebc-457e-8eb3-504cf9271dcc\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-cwtd8"
Sep 17 22:12:29.756729 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:29.756655    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-cwtd8"
Sep 17 22:12:29.864102 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:29.864068    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:12:30.033661 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:30.033630    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-cwtd8"]
Sep 17 22:12:30.035232 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:12:30.035204    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa4070e1_9ebc_457e_8eb3_504cf9271dcc.slice/crio-8bc301c0ea8bd52f36f8240cc491b831904347c90501c0b1ce8e557b4e00160a WatchSource:0}: Error finding container 8bc301c0ea8bd52f36f8240cc491b831904347c90501c0b1ce8e557b4e00160a: Status 404 returned error can't find the container with id 8bc301c0ea8bd52f36f8240cc491b831904347c90501c0b1ce8e557b4e00160a
Sep 17 22:12:31.034462 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:31.034417    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-cwtd8" event={"ID":"aa4070e1-9ebc-457e-8eb3-504cf9271dcc","Type":"ContainerStarted","Data":"8bc301c0ea8bd52f36f8240cc491b831904347c90501c0b1ce8e557b4e00160a"}
Sep 17 22:12:32.041538 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:32.041501    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-cwtd8" event={"ID":"aa4070e1-9ebc-457e-8eb3-504cf9271dcc","Type":"ContainerStarted","Data":"addb18a224d51bd965bb353c9dc5013c054864b3358577d34bd95ac24a361574"}
Sep 17 22:12:32.061830 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:32.061778    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-cwtd8" podStartSLOduration=1.197414394 podStartE2EDuration="3.061763714s" totalImagesPullingTime="1.86434932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:12:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:12:32.05924061 +0000 UTC m=+853.505285882" watchObservedRunningTime="2026-09-17 22:12:32.061763714 +0000 UTC m=+853.507808986"
Sep 17 22:12:32.757258 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:32.757226    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-cwtd8"
Sep 17 22:12:33.554447 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:33.554406    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:12:39.756911 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:39.756882    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-cwtd8"
Sep 17 22:12:39.769929 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:39.769904    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-cwtd8"
Sep 17 22:12:40.088262 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:40.088144    2555 generic.go:356] "Generic (PLEG): container finished" podID="296255f9-9ebf-49bb-8fcb-be7c8b392d12" containerID="a58e13e1433d9e6650dfdc7b28263fd7431fa01b2f65b7c57273df6c6e607102" exitCode=0
Sep 17 22:12:40.088262 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:40.088222    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8" event={"ID":"296255f9-9ebf-49bb-8fcb-be7c8b392d12","Type":"ContainerDied","Data":"a58e13e1433d9e6650dfdc7b28263fd7431fa01b2f65b7c57273df6c6e607102"}
Sep 17 22:12:40.088786 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:40.088749    2555 scope.go:117] "RemoveContainer" containerID="a58e13e1433d9e6650dfdc7b28263fd7431fa01b2f65b7c57273df6c6e607102"
Sep 17 22:12:40.089756 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:40.089721    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 22:12:40.089872 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:40.089806    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 22:12:40.089872 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:40.089852    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 22:12:40.100348 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:40.100329    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-cwtd8"
Sep 17 22:12:41.101700 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:41.101658    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8" event={"ID":"296255f9-9ebf-49bb-8fcb-be7c8b392d12","Type":"ContainerStarted","Data":"0183f355264d3a747081bf5526e0fcfa755d5940005950a05aa130bcbd8311b4"}
Sep 17 22:12:41.220001 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:41.219959    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8"
Sep 17 22:12:41.220748 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:41.220727    2555 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Sep 17 22:12:41.220823 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:41.220785    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8" podUID="296255f9-9ebf-49bb-8fcb-be7c8b392d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Sep 17 22:12:42.220288 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:42.220247    2555 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Sep 17 22:12:42.220701 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:42.220326    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8" podUID="296255f9-9ebf-49bb-8fcb-be7c8b392d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Sep 17 22:12:43.219888 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:43.219851    2555 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Sep 17 22:12:43.220153 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:43.219910    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8" podUID="296255f9-9ebf-49bb-8fcb-be7c8b392d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Sep 17 22:12:44.219840 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:44.219798    2555 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Sep 17 22:12:44.220258 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:44.219860    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8" podUID="296255f9-9ebf-49bb-8fcb-be7c8b392d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Sep 17 22:12:45.220470 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:45.220428    2555 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Sep 17 22:12:45.220866 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:45.220490    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8" podUID="296255f9-9ebf-49bb-8fcb-be7c8b392d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Sep 17 22:12:46.157821 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:46.157790    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8"
Sep 17 22:12:46.219390 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:46.219349    2555 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Sep 17 22:12:46.219561 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:46.219411    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8" podUID="296255f9-9ebf-49bb-8fcb-be7c8b392d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Sep 17 22:12:47.222906 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:47.222875    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8"
Sep 17 22:12:47.223929 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:47.223913    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-2q5j8"
Sep 17 22:12:54.556288 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:54.556254    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:12:54.778098 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:54.778063    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dl8lk"]
Sep 17 22:12:54.786711 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:54.786683    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dl8lk"
Sep 17 22:12:54.790305 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:54.790017    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dl8lk"]
Sep 17 22:12:54.790305 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:54.790067    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 17 22:12:54.790305 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:54.790141    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 17 22:12:54.790305 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:54.790159    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 17 22:12:54.790629 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:54.790420    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-bs5v2\""
Sep 17 22:12:54.790687 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:54.790635    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-tj9m4\""
Sep 17 22:12:54.806660 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:54.806587    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:12:54.877993 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:54.877955    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27bffec2-8f4e-4bf8-9231-4728b93f4f8e-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dl8lk\" (UID: \"27bffec2-8f4e-4bf8-9231-4728b93f4f8e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dl8lk"
Sep 17 22:12:54.878238 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:54.878119    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qgcb\" (UniqueName: \"kubernetes.io/projected/27bffec2-8f4e-4bf8-9231-4728b93f4f8e-kube-api-access-5qgcb\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dl8lk\" (UID: \"27bffec2-8f4e-4bf8-9231-4728b93f4f8e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dl8lk"
Sep 17 22:12:54.978651 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:54.978619    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27bffec2-8f4e-4bf8-9231-4728b93f4f8e-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dl8lk\" (UID: \"27bffec2-8f4e-4bf8-9231-4728b93f4f8e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dl8lk"
Sep 17 22:12:54.978790 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:54.978712    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qgcb\" (UniqueName: \"kubernetes.io/projected/27bffec2-8f4e-4bf8-9231-4728b93f4f8e-kube-api-access-5qgcb\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dl8lk\" (UID: \"27bffec2-8f4e-4bf8-9231-4728b93f4f8e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dl8lk"
Sep 17 22:12:54.981406 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:54.981353    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27bffec2-8f4e-4bf8-9231-4728b93f4f8e-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dl8lk\" (UID: \"27bffec2-8f4e-4bf8-9231-4728b93f4f8e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dl8lk"
Sep 17 22:12:54.989853 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:54.989830    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qgcb\" (UniqueName: \"kubernetes.io/projected/27bffec2-8f4e-4bf8-9231-4728b93f4f8e-kube-api-access-5qgcb\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dl8lk\" (UID: \"27bffec2-8f4e-4bf8-9231-4728b93f4f8e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dl8lk"
Sep 17 22:12:55.099561 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:55.099476    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dl8lk"
Sep 17 22:12:55.380858 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:55.380823    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dl8lk"]
Sep 17 22:12:56.190092 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:56.189975    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dl8lk" event={"ID":"27bffec2-8f4e-4bf8-9231-4728b93f4f8e","Type":"ContainerStarted","Data":"2a77c69bb5650dc4757908078dc582a592a4dcbfc1cffefb82c16a36e4b461fd"}
Sep 17 22:12:58.204702 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:58.204664    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dl8lk" event={"ID":"27bffec2-8f4e-4bf8-9231-4728b93f4f8e","Type":"ContainerStarted","Data":"c44a706159b70c012aaca7a71fc9f51ba8185cf2ed30992d846fbae9c44903dc"}
Sep 17 22:12:58.205105 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:58.204728    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dl8lk"
Sep 17 22:12:58.222511 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:12:58.222458    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dl8lk" podStartSLOduration=1.786237128 podStartE2EDuration="4.222442809s" totalImagesPullingTime="2.436205681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:12:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:12:58.219804975 +0000 UTC m=+879.665850249" watchObservedRunningTime="2026-09-17 22:12:58.222442809 +0000 UTC m=+879.668488080"
Sep 17 22:13:15.837159 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:15.837075    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:13:20.640197 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:20.640147    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-7njvj_a324c9ab-5d42-4d31-8ff7-449f619be774/console-operator/1.log"
Sep 17 22:13:20.640703 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:20.640415    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-7njvj_a324c9ab-5d42-4d31-8ff7-449f619be774/console-operator/1.log"
Sep 17 22:13:20.644243 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:20.644222    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s7f5_4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf/ovn-acl-logging/0.log"
Sep 17 22:13:20.644454 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:20.644442    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s7f5_4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf/ovn-acl-logging/0.log"
Sep 17 22:13:21.906592 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:21.906556    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:13:28.383152 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:28.383123    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dl8lk"]
Sep 17 22:13:28.385402 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:28.385365    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dl8lk" podUID="27bffec2-8f4e-4bf8-9231-4728b93f4f8e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:13:28.556442 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:28.556409    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:13:28.656961 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:28.656873    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:13:30.457778 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:30.457742    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:13:30.464016 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:30.463987    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-cwtd8"]
Sep 17 22:13:30.475892 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:13:30.475829    2555 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 22:13:30.475892 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:13:30.475892 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:13:30.475892 ip-10-0-140-157 kubenswrapper[2555]:  > containerID="addb18a224d51bd965bb353c9dc5013c054864b3358577d34bd95ac24a361574" cmd=["/bin/sleep","15"]
Sep 17 22:13:30.476480 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:30.476457    2555 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-cwtd8" message="" err=<
Sep 17 22:13:30.476480 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:13:30.476480 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:13:30.476480 ip-10-0-140-157 kubenswrapper[2555]:  >
Sep 17 22:13:30.476827 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:13:30.476806    2555 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 22:13:30.476827 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:13:30.476827 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:13:30.476827 ip-10-0-140-157 kubenswrapper[2555]:  > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-cwtd8" podUID="aa4070e1-9ebc-457e-8eb3-504cf9271dcc" containerName="main" containerID="cri-o://addb18a224d51bd965bb353c9dc5013c054864b3358577d34bd95ac24a361574"
Sep 17 22:13:30.477166 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:30.477141    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-cwtd8" podUID="aa4070e1-9ebc-457e-8eb3-504cf9271dcc" containerName="main" containerID="cri-o://addb18a224d51bd965bb353c9dc5013c054864b3358577d34bd95ac24a361574" gracePeriod=60
Sep 17 22:13:30.555028 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:30.554999    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:13:30.838051 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:30.838029    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-cwtd8"
Sep 17 22:13:30.892127 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:30.892099    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa4070e1-9ebc-457e-8eb3-504cf9271dcc-tls-certs\") pod \"aa4070e1-9ebc-457e-8eb3-504cf9271dcc\" (UID: \"aa4070e1-9ebc-457e-8eb3-504cf9271dcc\") "
Sep 17 22:13:30.892326 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:30.892143    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dsrvl\" (UniqueName: \"kubernetes.io/projected/aa4070e1-9ebc-457e-8eb3-504cf9271dcc-kube-api-access-dsrvl\") pod \"aa4070e1-9ebc-457e-8eb3-504cf9271dcc\" (UID: \"aa4070e1-9ebc-457e-8eb3-504cf9271dcc\") "
Sep 17 22:13:30.892326 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:30.892209    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa4070e1-9ebc-457e-8eb3-504cf9271dcc-tmp-dir\") pod \"aa4070e1-9ebc-457e-8eb3-504cf9271dcc\" (UID: \"aa4070e1-9ebc-457e-8eb3-504cf9271dcc\") "
Sep 17 22:13:30.892326 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:30.892271    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa4070e1-9ebc-457e-8eb3-504cf9271dcc-model-cache\") pod \"aa4070e1-9ebc-457e-8eb3-504cf9271dcc\" (UID: \"aa4070e1-9ebc-457e-8eb3-504cf9271dcc\") "
Sep 17 22:13:30.892326 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:30.892301    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa4070e1-9ebc-457e-8eb3-504cf9271dcc-home\") pod \"aa4070e1-9ebc-457e-8eb3-504cf9271dcc\" (UID: \"aa4070e1-9ebc-457e-8eb3-504cf9271dcc\") "
Sep 17 22:13:30.892560 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:30.892379    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa4070e1-9ebc-457e-8eb3-504cf9271dcc-dshm\") pod \"aa4070e1-9ebc-457e-8eb3-504cf9271dcc\" (UID: \"aa4070e1-9ebc-457e-8eb3-504cf9271dcc\") "
Sep 17 22:13:30.892614 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:30.892562    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4070e1-9ebc-457e-8eb3-504cf9271dcc-model-cache" (OuterVolumeSpecName: "model-cache") pod "aa4070e1-9ebc-457e-8eb3-504cf9271dcc" (UID: "aa4070e1-9ebc-457e-8eb3-504cf9271dcc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:13:30.892798 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:30.892766    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4070e1-9ebc-457e-8eb3-504cf9271dcc-home" (OuterVolumeSpecName: "home") pod "aa4070e1-9ebc-457e-8eb3-504cf9271dcc" (UID: "aa4070e1-9ebc-457e-8eb3-504cf9271dcc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:13:30.892894 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:30.892768    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa4070e1-9ebc-457e-8eb3-504cf9271dcc-model-cache\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:13:30.893015 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:30.892988    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4070e1-9ebc-457e-8eb3-504cf9271dcc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aa4070e1-9ebc-457e-8eb3-504cf9271dcc" (UID: "aa4070e1-9ebc-457e-8eb3-504cf9271dcc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:13:30.894934 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:30.894904    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa4070e1-9ebc-457e-8eb3-504cf9271dcc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aa4070e1-9ebc-457e-8eb3-504cf9271dcc" (UID: "aa4070e1-9ebc-457e-8eb3-504cf9271dcc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:13:30.895033 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:30.894951    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa4070e1-9ebc-457e-8eb3-504cf9271dcc-kube-api-access-dsrvl" (OuterVolumeSpecName: "kube-api-access-dsrvl") pod "aa4070e1-9ebc-457e-8eb3-504cf9271dcc" (UID: "aa4070e1-9ebc-457e-8eb3-504cf9271dcc"). InnerVolumeSpecName "kube-api-access-dsrvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:13:30.895033 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:30.894955    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4070e1-9ebc-457e-8eb3-504cf9271dcc-dshm" (OuterVolumeSpecName: "dshm") pod "aa4070e1-9ebc-457e-8eb3-504cf9271dcc" (UID: "aa4070e1-9ebc-457e-8eb3-504cf9271dcc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:13:30.994115 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:30.994079    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa4070e1-9ebc-457e-8eb3-504cf9271dcc-tls-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:13:30.994115 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:30.994108    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsrvl\" (UniqueName: \"kubernetes.io/projected/aa4070e1-9ebc-457e-8eb3-504cf9271dcc-kube-api-access-dsrvl\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:13:30.994115 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:30.994120    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa4070e1-9ebc-457e-8eb3-504cf9271dcc-tmp-dir\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:13:30.994430 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:30.994130    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa4070e1-9ebc-457e-8eb3-504cf9271dcc-home\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:13:30.994430 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:30.994138    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa4070e1-9ebc-457e-8eb3-504cf9271dcc-dshm\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:13:31.401935 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:31.401903    2555 generic.go:356] "Generic (PLEG): container finished" podID="aa4070e1-9ebc-457e-8eb3-504cf9271dcc" containerID="addb18a224d51bd965bb353c9dc5013c054864b3358577d34bd95ac24a361574" exitCode=0
Sep 17 22:13:31.402113 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:31.401977    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-cwtd8" event={"ID":"aa4070e1-9ebc-457e-8eb3-504cf9271dcc","Type":"ContainerDied","Data":"addb18a224d51bd965bb353c9dc5013c054864b3358577d34bd95ac24a361574"}
Sep 17 22:13:31.402113 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:31.402019    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-cwtd8"
Sep 17 22:13:31.402113 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:31.402031    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-cwtd8" event={"ID":"aa4070e1-9ebc-457e-8eb3-504cf9271dcc","Type":"ContainerDied","Data":"8bc301c0ea8bd52f36f8240cc491b831904347c90501c0b1ce8e557b4e00160a"}
Sep 17 22:13:31.402113 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:31.402063    2555 scope.go:117] "RemoveContainer" containerID="addb18a224d51bd965bb353c9dc5013c054864b3358577d34bd95ac24a361574"
Sep 17 22:13:31.412119 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:31.412094    2555 scope.go:117] "RemoveContainer" containerID="addb18a224d51bd965bb353c9dc5013c054864b3358577d34bd95ac24a361574"
Sep 17 22:13:31.412415 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:13:31.412399    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"addb18a224d51bd965bb353c9dc5013c054864b3358577d34bd95ac24a361574\": container with ID starting with addb18a224d51bd965bb353c9dc5013c054864b3358577d34bd95ac24a361574 not found: ID does not exist" containerID="addb18a224d51bd965bb353c9dc5013c054864b3358577d34bd95ac24a361574"
Sep 17 22:13:31.412476 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:31.412423    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"addb18a224d51bd965bb353c9dc5013c054864b3358577d34bd95ac24a361574"} err="failed to get container status \"addb18a224d51bd965bb353c9dc5013c054864b3358577d34bd95ac24a361574\": rpc error: code = NotFound desc = could not find container \"addb18a224d51bd965bb353c9dc5013c054864b3358577d34bd95ac24a361574\": container with ID starting with addb18a224d51bd965bb353c9dc5013c054864b3358577d34bd95ac24a361574 not found: ID does not exist"
Sep 17 22:13:31.423833 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:31.423809    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-cwtd8"]
Sep 17 22:13:31.429784 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:31.429763    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-cwtd8"]
Sep 17 22:13:32.679127 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:32.679094    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa4070e1-9ebc-457e-8eb3-504cf9271dcc" path="/var/lib/kubelet/pods/aa4070e1-9ebc-457e-8eb3-504cf9271dcc/volumes"
Sep 17 22:13:38.385003 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:38.384955    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dl8lk" podUID="27bffec2-8f4e-4bf8-9231-4728b93f4f8e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:13:43.404398 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:43.404362    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dl8lk" podUID="27bffec2-8f4e-4bf8-9231-4728b93f4f8e" containerName="main" containerID="cri-o://c44a706159b70c012aaca7a71fc9f51ba8185cf2ed30992d846fbae9c44903dc" gracePeriod=45
Sep 17 22:13:48.385205 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:48.385151    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dl8lk" podUID="27bffec2-8f4e-4bf8-9231-4728b93f4f8e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:13:58.385846 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:13:58.385806    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dl8lk" podUID="27bffec2-8f4e-4bf8-9231-4728b93f4f8e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:14:08.385499 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:08.385455    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dl8lk" podUID="27bffec2-8f4e-4bf8-9231-4728b93f4f8e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:14:12.513861 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:12.513823    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:14:13.633596 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:13.633564    2555 generic.go:356] "Generic (PLEG): container finished" podID="27bffec2-8f4e-4bf8-9231-4728b93f4f8e" containerID="c44a706159b70c012aaca7a71fc9f51ba8185cf2ed30992d846fbae9c44903dc" exitCode=0
Sep 17 22:14:13.633962 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:13.633634    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dl8lk" event={"ID":"27bffec2-8f4e-4bf8-9231-4728b93f4f8e","Type":"ContainerDied","Data":"c44a706159b70c012aaca7a71fc9f51ba8185cf2ed30992d846fbae9c44903dc"}
Sep 17 22:14:13.658330 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:13.658306    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dl8lk"
Sep 17 22:14:13.757111 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:13.757024    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27bffec2-8f4e-4bf8-9231-4728b93f4f8e-tls-certs\") pod \"27bffec2-8f4e-4bf8-9231-4728b93f4f8e\" (UID: \"27bffec2-8f4e-4bf8-9231-4728b93f4f8e\") "
Sep 17 22:14:13.757111 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:13.757068    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qgcb\" (UniqueName: \"kubernetes.io/projected/27bffec2-8f4e-4bf8-9231-4728b93f4f8e-kube-api-access-5qgcb\") pod \"27bffec2-8f4e-4bf8-9231-4728b93f4f8e\" (UID: \"27bffec2-8f4e-4bf8-9231-4728b93f4f8e\") "
Sep 17 22:14:13.759244 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:13.759214    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27bffec2-8f4e-4bf8-9231-4728b93f4f8e-kube-api-access-5qgcb" (OuterVolumeSpecName: "kube-api-access-5qgcb") pod "27bffec2-8f4e-4bf8-9231-4728b93f4f8e" (UID: "27bffec2-8f4e-4bf8-9231-4728b93f4f8e"). InnerVolumeSpecName "kube-api-access-5qgcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:14:13.759360 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:13.759264    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27bffec2-8f4e-4bf8-9231-4728b93f4f8e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "27bffec2-8f4e-4bf8-9231-4728b93f4f8e" (UID: "27bffec2-8f4e-4bf8-9231-4728b93f4f8e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:14:13.858349 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:13.858320    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27bffec2-8f4e-4bf8-9231-4728b93f4f8e-tls-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:14:13.858349 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:13.858348    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qgcb\" (UniqueName: \"kubernetes.io/projected/27bffec2-8f4e-4bf8-9231-4728b93f4f8e-kube-api-access-5qgcb\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:14:14.641359 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:14.641331    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dl8lk"
Sep 17 22:14:14.641763 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:14.641384    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dl8lk" event={"ID":"27bffec2-8f4e-4bf8-9231-4728b93f4f8e","Type":"ContainerDied","Data":"2a77c69bb5650dc4757908078dc582a592a4dcbfc1cffefb82c16a36e4b461fd"}
Sep 17 22:14:14.641763 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:14.641449    2555 scope.go:117] "RemoveContainer" containerID="c44a706159b70c012aaca7a71fc9f51ba8185cf2ed30992d846fbae9c44903dc"
Sep 17 22:14:14.662448 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:14.662425    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dl8lk"]
Sep 17 22:14:14.665425 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:14.665401    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dl8lk"]
Sep 17 22:14:14.679057 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:14.679028    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27bffec2-8f4e-4bf8-9231-4728b93f4f8e" path="/var/lib/kubelet/pods/27bffec2-8f4e-4bf8-9231-4728b93f4f8e/volumes"
Sep 17 22:14:16.671759 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:16.671728    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:14:18.378229 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:18.378168    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:14:18.578313 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:18.578280    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:14:20.154765 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:20.154731    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:14:20.250027 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:20.249995    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:14:24.282140 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:24.282099    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:14:26.037555 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:26.037515    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:14:28.938237 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:28.938205    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-zr699"]
Sep 17 22:14:28.938816 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:28.938765    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27bffec2-8f4e-4bf8-9231-4728b93f4f8e" containerName="main"
Sep 17 22:14:28.938816 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:28.938785    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="27bffec2-8f4e-4bf8-9231-4728b93f4f8e" containerName="main"
Sep 17 22:14:28.938816 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:28.938810    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa4070e1-9ebc-457e-8eb3-504cf9271dcc" containerName="main"
Sep 17 22:14:28.938816 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:28.938815    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa4070e1-9ebc-457e-8eb3-504cf9271dcc" containerName="main"
Sep 17 22:14:28.938965 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:28.938880    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="27bffec2-8f4e-4bf8-9231-4728b93f4f8e" containerName="main"
Sep 17 22:14:28.938965 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:28.938894    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa4070e1-9ebc-457e-8eb3-504cf9271dcc" containerName="main"
Sep 17 22:14:28.943374 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:28.943358    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-zr699"
Sep 17 22:14:28.945772 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:28.945751    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 17 22:14:28.945879 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:28.945794    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 17 22:14:28.946704 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:28.946682    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 17 22:14:28.946785 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:28.946771    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-qkgs4\""
Sep 17 22:14:28.953871 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:28.953847    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-zr699"]
Sep 17 22:14:28.962664 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:28.962640    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:14:29.085664 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:29.085629    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7378e659-14ef-4a43-b38b-4925a388e20c-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-zr699\" (UID: \"7378e659-14ef-4a43-b38b-4925a388e20c\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-zr699"
Sep 17 22:14:29.085859 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:29.085673    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7378e659-14ef-4a43-b38b-4925a388e20c-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-zr699\" (UID: \"7378e659-14ef-4a43-b38b-4925a388e20c\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-zr699"
Sep 17 22:14:29.085859 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:29.085703    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7378e659-14ef-4a43-b38b-4925a388e20c-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-zr699\" (UID: \"7378e659-14ef-4a43-b38b-4925a388e20c\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-zr699"
Sep 17 22:14:29.085859 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:29.085724    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rrcn\" (UniqueName: \"kubernetes.io/projected/7378e659-14ef-4a43-b38b-4925a388e20c-kube-api-access-7rrcn\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-zr699\" (UID: \"7378e659-14ef-4a43-b38b-4925a388e20c\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-zr699"
Sep 17 22:14:29.085859 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:29.085748    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7378e659-14ef-4a43-b38b-4925a388e20c-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-zr699\" (UID: \"7378e659-14ef-4a43-b38b-4925a388e20c\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-zr699"
Sep 17 22:14:29.085859 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:29.085763    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7378e659-14ef-4a43-b38b-4925a388e20c-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-zr699\" (UID: \"7378e659-14ef-4a43-b38b-4925a388e20c\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-zr699"
Sep 17 22:14:29.186978 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:29.186946    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7378e659-14ef-4a43-b38b-4925a388e20c-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-zr699\" (UID: \"7378e659-14ef-4a43-b38b-4925a388e20c\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-zr699"
Sep 17 22:14:29.187211 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:29.186986    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7378e659-14ef-4a43-b38b-4925a388e20c-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-zr699\" (UID: \"7378e659-14ef-4a43-b38b-4925a388e20c\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-zr699"
Sep 17 22:14:29.187211 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:29.187005    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rrcn\" (UniqueName: \"kubernetes.io/projected/7378e659-14ef-4a43-b38b-4925a388e20c-kube-api-access-7rrcn\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-zr699\" (UID: \"7378e659-14ef-4a43-b38b-4925a388e20c\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-zr699"
Sep 17 22:14:29.187211 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:29.187024    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7378e659-14ef-4a43-b38b-4925a388e20c-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-zr699\" (UID: \"7378e659-14ef-4a43-b38b-4925a388e20c\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-zr699"
Sep 17 22:14:29.187211 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:29.187038    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7378e659-14ef-4a43-b38b-4925a388e20c-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-zr699\" (UID: \"7378e659-14ef-4a43-b38b-4925a388e20c\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-zr699"
Sep 17 22:14:29.187211 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:29.187115    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7378e659-14ef-4a43-b38b-4925a388e20c-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-zr699\" (UID: \"7378e659-14ef-4a43-b38b-4925a388e20c\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-zr699"
Sep 17 22:14:29.187524 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:29.187502    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7378e659-14ef-4a43-b38b-4925a388e20c-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-zr699\" (UID: \"7378e659-14ef-4a43-b38b-4925a388e20c\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-zr699"
Sep 17 22:14:29.187572 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:29.187522    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7378e659-14ef-4a43-b38b-4925a388e20c-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-zr699\" (UID: \"7378e659-14ef-4a43-b38b-4925a388e20c\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-zr699"
Sep 17 22:14:29.187612 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:29.187595    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7378e659-14ef-4a43-b38b-4925a388e20c-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-zr699\" (UID: \"7378e659-14ef-4a43-b38b-4925a388e20c\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-zr699"
Sep 17 22:14:29.189402 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:29.189341    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7378e659-14ef-4a43-b38b-4925a388e20c-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-zr699\" (UID: \"7378e659-14ef-4a43-b38b-4925a388e20c\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-zr699"
Sep 17 22:14:29.189534 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:29.189516    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7378e659-14ef-4a43-b38b-4925a388e20c-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-zr699\" (UID: \"7378e659-14ef-4a43-b38b-4925a388e20c\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-zr699"
Sep 17 22:14:29.194648 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:29.194621    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rrcn\" (UniqueName: \"kubernetes.io/projected/7378e659-14ef-4a43-b38b-4925a388e20c-kube-api-access-7rrcn\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-zr699\" (UID: \"7378e659-14ef-4a43-b38b-4925a388e20c\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-zr699"
Sep 17 22:14:29.229577 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:29.229543    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:14:29.255743 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:29.255718    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-zr699"
Sep 17 22:14:29.533616 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:29.533583    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-zr699"]
Sep 17 22:14:29.535292 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:14:29.535248    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7378e659_14ef_4a43_b38b_4925a388e20c.slice/crio-a67e3f4241202bb349457abc223c7cb9984c796397dbdb17baf52cc8725b86a2 WatchSource:0}: Error finding container a67e3f4241202bb349457abc223c7cb9984c796397dbdb17baf52cc8725b86a2: Status 404 returned error can't find the container with id a67e3f4241202bb349457abc223c7cb9984c796397dbdb17baf52cc8725b86a2
Sep 17 22:14:29.722135 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:29.722097    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-zr699" event={"ID":"7378e659-14ef-4a43-b38b-4925a388e20c","Type":"ContainerStarted","Data":"0990a2d3475c85fcb8b3bc23178b5d0fdf13f01494b81d9fdff9fe17187d0e9d"}
Sep 17 22:14:29.722321 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:29.722156    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-zr699" event={"ID":"7378e659-14ef-4a43-b38b-4925a388e20c","Type":"ContainerStarted","Data":"a67e3f4241202bb349457abc223c7cb9984c796397dbdb17baf52cc8725b86a2"}
Sep 17 22:14:29.738350 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:29.738300    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-zr699" podStartSLOduration=1.738286497 podStartE2EDuration="1.738286497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:14:29.736218444 +0000 UTC m=+971.182263718" watchObservedRunningTime="2026-09-17 22:14:29.738286497 +0000 UTC m=+971.184331768"
Sep 17 22:14:30.256407 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:30.256372    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-zr699"
Sep 17 22:14:30.487971 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:30.487936    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:14:30.588859 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:30.588761    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:14:31.883240 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:31.883145    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:14:31.983532 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:31.983449    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:14:39.256261 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:39.256162    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-zr699"
Sep 17 22:14:39.268587 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:39.268565    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-zr699"
Sep 17 22:14:39.788912 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:39.788873    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-zr699"
Sep 17 22:14:44.904114 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:44.904085    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:14:45.004004 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:45.003949    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:14:50.429592 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:50.429553    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-4lf7w"]
Sep 17 22:14:50.433482 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:50.433460    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-4lf7w"
Sep 17 22:14:50.435938 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:50.435915    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 17 22:14:50.436808 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:50.436778    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-fd9d5\""
Sep 17 22:14:50.436931 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:50.436841    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 17 22:14:50.436931 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:50.436845    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 17 22:14:50.444630 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:50.443415    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-4lf7w"]
Sep 17 22:14:50.463680 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:50.463655    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:14:50.579289 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:50.579256    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2e5ef51-c042-464d-87ac-7a0508142d79-home\") pod \"fc-concurrency-test-kserve-b5d44467c-4lf7w\" (UID: \"d2e5ef51-c042-464d-87ac-7a0508142d79\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-4lf7w"
Sep 17 22:14:50.579473 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:50.579319    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2e5ef51-c042-464d-87ac-7a0508142d79-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-4lf7w\" (UID: \"d2e5ef51-c042-464d-87ac-7a0508142d79\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-4lf7w"
Sep 17 22:14:50.579473 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:50.579404    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2e5ef51-c042-464d-87ac-7a0508142d79-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-4lf7w\" (UID: \"d2e5ef51-c042-464d-87ac-7a0508142d79\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-4lf7w"
Sep 17 22:14:50.579473 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:50.579437    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2e5ef51-c042-464d-87ac-7a0508142d79-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-4lf7w\" (UID: \"d2e5ef51-c042-464d-87ac-7a0508142d79\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-4lf7w"
Sep 17 22:14:50.579579 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:50.579495    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn2l5\" (UniqueName: \"kubernetes.io/projected/d2e5ef51-c042-464d-87ac-7a0508142d79-kube-api-access-pn2l5\") pod \"fc-concurrency-test-kserve-b5d44467c-4lf7w\" (UID: \"d2e5ef51-c042-464d-87ac-7a0508142d79\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-4lf7w"
Sep 17 22:14:50.579579 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:50.579515    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2e5ef51-c042-464d-87ac-7a0508142d79-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-4lf7w\" (UID: \"d2e5ef51-c042-464d-87ac-7a0508142d79\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-4lf7w"
Sep 17 22:14:50.666785 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:50.666742    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:14:50.680680 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:50.680600    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2e5ef51-c042-464d-87ac-7a0508142d79-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-4lf7w\" (UID: \"d2e5ef51-c042-464d-87ac-7a0508142d79\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-4lf7w"
Sep 17 22:14:50.680680 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:50.680635    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2e5ef51-c042-464d-87ac-7a0508142d79-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-4lf7w\" (UID: \"d2e5ef51-c042-464d-87ac-7a0508142d79\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-4lf7w"
Sep 17 22:14:50.680894 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:50.680822    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn2l5\" (UniqueName: \"kubernetes.io/projected/d2e5ef51-c042-464d-87ac-7a0508142d79-kube-api-access-pn2l5\") pod \"fc-concurrency-test-kserve-b5d44467c-4lf7w\" (UID: \"d2e5ef51-c042-464d-87ac-7a0508142d79\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-4lf7w"
Sep 17 22:14:50.680894 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:50.680847    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2e5ef51-c042-464d-87ac-7a0508142d79-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-4lf7w\" (UID: \"d2e5ef51-c042-464d-87ac-7a0508142d79\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-4lf7w"
Sep 17 22:14:50.680894 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:50.680878    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2e5ef51-c042-464d-87ac-7a0508142d79-home\") pod \"fc-concurrency-test-kserve-b5d44467c-4lf7w\" (UID: \"d2e5ef51-c042-464d-87ac-7a0508142d79\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-4lf7w"
Sep 17 22:14:50.681016 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:50.680925    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2e5ef51-c042-464d-87ac-7a0508142d79-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-4lf7w\" (UID: \"d2e5ef51-c042-464d-87ac-7a0508142d79\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-4lf7w"
Sep 17 22:14:50.681060 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:50.681013    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2e5ef51-c042-464d-87ac-7a0508142d79-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-4lf7w\" (UID: \"d2e5ef51-c042-464d-87ac-7a0508142d79\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-4lf7w"
Sep 17 22:14:50.681306 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:50.681283    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2e5ef51-c042-464d-87ac-7a0508142d79-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-4lf7w\" (UID: \"d2e5ef51-c042-464d-87ac-7a0508142d79\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-4lf7w"
Sep 17 22:14:50.681407 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:50.681325    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2e5ef51-c042-464d-87ac-7a0508142d79-home\") pod \"fc-concurrency-test-kserve-b5d44467c-4lf7w\" (UID: \"d2e5ef51-c042-464d-87ac-7a0508142d79\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-4lf7w"
Sep 17 22:14:50.683098 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:50.683077    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2e5ef51-c042-464d-87ac-7a0508142d79-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-4lf7w\" (UID: \"d2e5ef51-c042-464d-87ac-7a0508142d79\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-4lf7w"
Sep 17 22:14:50.683325 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:50.683306    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2e5ef51-c042-464d-87ac-7a0508142d79-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-4lf7w\" (UID: \"d2e5ef51-c042-464d-87ac-7a0508142d79\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-4lf7w"
Sep 17 22:14:50.689105 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:50.689079    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn2l5\" (UniqueName: \"kubernetes.io/projected/d2e5ef51-c042-464d-87ac-7a0508142d79-kube-api-access-pn2l5\") pod \"fc-concurrency-test-kserve-b5d44467c-4lf7w\" (UID: \"d2e5ef51-c042-464d-87ac-7a0508142d79\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-4lf7w"
Sep 17 22:14:50.750561 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:50.750524    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-4lf7w"
Sep 17 22:14:51.026885 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:51.026859    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-4lf7w"]
Sep 17 22:14:51.027724 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:14:51.027702    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2e5ef51_c042_464d_87ac_7a0508142d79.slice/crio-94af464679e91f44b37b6b4ea713497e7d37ba46ae49bf9d6ab42f5046e7a984 WatchSource:0}: Error finding container 94af464679e91f44b37b6b4ea713497e7d37ba46ae49bf9d6ab42f5046e7a984: Status 404 returned error can't find the container with id 94af464679e91f44b37b6b4ea713497e7d37ba46ae49bf9d6ab42f5046e7a984
Sep 17 22:14:51.845855 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:51.845816    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-4lf7w" event={"ID":"d2e5ef51-c042-464d-87ac-7a0508142d79","Type":"ContainerStarted","Data":"74f7aa549b5e3fc886202fe470e17c6e8ca16eb8364028c8327a82a3991a7d17"}
Sep 17 22:14:51.846359 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:51.845872    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-4lf7w" event={"ID":"d2e5ef51-c042-464d-87ac-7a0508142d79","Type":"ContainerStarted","Data":"94af464679e91f44b37b6b4ea713497e7d37ba46ae49bf9d6ab42f5046e7a984"}
Sep 17 22:14:51.864372 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:51.864325    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-4lf7w" podStartSLOduration=1.864309574 podStartE2EDuration="1.864309574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:14:51.862051239 +0000 UTC m=+993.308096539" watchObservedRunningTime="2026-09-17 22:14:51.864309574 +0000 UTC m=+993.310354846"
Sep 17 22:14:52.751343 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:14:52.751311    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-4lf7w"
Sep 17 22:15:00.751357 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:00.751321    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-4lf7w"
Sep 17 22:15:00.763792 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:00.763768    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-4lf7w"
Sep 17 22:15:00.907658 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:00.907626    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-4lf7w"
Sep 17 22:15:39.440035 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:39.439989    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:15:42.471037 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:42.471000    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:15:42.487011 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:42.486977    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-zr699"]
Sep 17 22:15:42.502445 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:15:42.502404    2555 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 22:15:42.502445 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:15:42.502445 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:15:42.502445 ip-10-0-140-157 kubenswrapper[2555]:  > containerID="0990a2d3475c85fcb8b3bc23178b5d0fdf13f01494b81d9fdff9fe17187d0e9d" cmd=["/bin/sleep","15"]
Sep 17 22:15:42.502719 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:42.502469    2555 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-zr699" message="" err=<
Sep 17 22:15:42.502719 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:15:42.502719 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:15:42.502719 ip-10-0-140-157 kubenswrapper[2555]:  >
Sep 17 22:15:42.502719 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:15:42.502486    2555 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 22:15:42.502719 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:15:42.502719 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:15:42.502719 ip-10-0-140-157 kubenswrapper[2555]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-zr699" podUID="7378e659-14ef-4a43-b38b-4925a388e20c" containerName="main" containerID="cri-o://0990a2d3475c85fcb8b3bc23178b5d0fdf13f01494b81d9fdff9fe17187d0e9d"
Sep 17 22:15:42.502719 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:42.502532    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-zr699" podUID="7378e659-14ef-4a43-b38b-4925a388e20c" containerName="main" containerID="cri-o://0990a2d3475c85fcb8b3bc23178b5d0fdf13f01494b81d9fdff9fe17187d0e9d" gracePeriod=60
Sep 17 22:15:42.565491 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:42.565455    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:15:42.845425 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:42.845394    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-zr699"
Sep 17 22:15:42.933365 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:42.933339    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7378e659-14ef-4a43-b38b-4925a388e20c-dshm\") pod \"7378e659-14ef-4a43-b38b-4925a388e20c\" (UID: \"7378e659-14ef-4a43-b38b-4925a388e20c\") "
Sep 17 22:15:42.933365 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:42.933372    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rrcn\" (UniqueName: \"kubernetes.io/projected/7378e659-14ef-4a43-b38b-4925a388e20c-kube-api-access-7rrcn\") pod \"7378e659-14ef-4a43-b38b-4925a388e20c\" (UID: \"7378e659-14ef-4a43-b38b-4925a388e20c\") "
Sep 17 22:15:42.933592 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:42.933403    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7378e659-14ef-4a43-b38b-4925a388e20c-tmp-dir\") pod \"7378e659-14ef-4a43-b38b-4925a388e20c\" (UID: \"7378e659-14ef-4a43-b38b-4925a388e20c\") "
Sep 17 22:15:42.933649 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:42.933614    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7378e659-14ef-4a43-b38b-4925a388e20c-tls-certs\") pod \"7378e659-14ef-4a43-b38b-4925a388e20c\" (UID: \"7378e659-14ef-4a43-b38b-4925a388e20c\") "
Sep 17 22:15:42.933704 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:42.933673    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7378e659-14ef-4a43-b38b-4925a388e20c-home\") pod \"7378e659-14ef-4a43-b38b-4925a388e20c\" (UID: \"7378e659-14ef-4a43-b38b-4925a388e20c\") "
Sep 17 22:15:42.933704 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:42.933666    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7378e659-14ef-4a43-b38b-4925a388e20c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7378e659-14ef-4a43-b38b-4925a388e20c" (UID: "7378e659-14ef-4a43-b38b-4925a388e20c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:15:42.933805 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:42.933716    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7378e659-14ef-4a43-b38b-4925a388e20c-model-cache\") pod \"7378e659-14ef-4a43-b38b-4925a388e20c\" (UID: \"7378e659-14ef-4a43-b38b-4925a388e20c\") "
Sep 17 22:15:42.933880 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:42.933860    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7378e659-14ef-4a43-b38b-4925a388e20c-home" (OuterVolumeSpecName: "home") pod "7378e659-14ef-4a43-b38b-4925a388e20c" (UID: "7378e659-14ef-4a43-b38b-4925a388e20c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:15:42.934003 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:42.933962    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7378e659-14ef-4a43-b38b-4925a388e20c-model-cache" (OuterVolumeSpecName: "model-cache") pod "7378e659-14ef-4a43-b38b-4925a388e20c" (UID: "7378e659-14ef-4a43-b38b-4925a388e20c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:15:42.934056 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:42.933991    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7378e659-14ef-4a43-b38b-4925a388e20c-tmp-dir\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:15:42.934056 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:42.934025    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7378e659-14ef-4a43-b38b-4925a388e20c-home\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:15:42.936291 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:42.936260    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7378e659-14ef-4a43-b38b-4925a388e20c-dshm" (OuterVolumeSpecName: "dshm") pod "7378e659-14ef-4a43-b38b-4925a388e20c" (UID: "7378e659-14ef-4a43-b38b-4925a388e20c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:15:42.936291 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:42.936272    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7378e659-14ef-4a43-b38b-4925a388e20c-kube-api-access-7rrcn" (OuterVolumeSpecName: "kube-api-access-7rrcn") pod "7378e659-14ef-4a43-b38b-4925a388e20c" (UID: "7378e659-14ef-4a43-b38b-4925a388e20c"). InnerVolumeSpecName "kube-api-access-7rrcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:15:42.936467 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:42.936270    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7378e659-14ef-4a43-b38b-4925a388e20c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7378e659-14ef-4a43-b38b-4925a388e20c" (UID: "7378e659-14ef-4a43-b38b-4925a388e20c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:15:43.035409 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:43.035324    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7378e659-14ef-4a43-b38b-4925a388e20c-tls-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:15:43.035409 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:43.035355    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7378e659-14ef-4a43-b38b-4925a388e20c-model-cache\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:15:43.035409 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:43.035364    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7378e659-14ef-4a43-b38b-4925a388e20c-dshm\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:15:43.035409 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:43.035372    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rrcn\" (UniqueName: \"kubernetes.io/projected/7378e659-14ef-4a43-b38b-4925a388e20c-kube-api-access-7rrcn\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:15:43.128098 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:43.128062    2555 generic.go:356] "Generic (PLEG): container finished" podID="7378e659-14ef-4a43-b38b-4925a388e20c" containerID="0990a2d3475c85fcb8b3bc23178b5d0fdf13f01494b81d9fdff9fe17187d0e9d" exitCode=0
Sep 17 22:15:43.128293 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:43.128112    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-zr699"
Sep 17 22:15:43.128293 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:43.128117    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-zr699" event={"ID":"7378e659-14ef-4a43-b38b-4925a388e20c","Type":"ContainerDied","Data":"0990a2d3475c85fcb8b3bc23178b5d0fdf13f01494b81d9fdff9fe17187d0e9d"}
Sep 17 22:15:43.128293 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:43.128173    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-zr699" event={"ID":"7378e659-14ef-4a43-b38b-4925a388e20c","Type":"ContainerDied","Data":"a67e3f4241202bb349457abc223c7cb9984c796397dbdb17baf52cc8725b86a2"}
Sep 17 22:15:43.128293 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:43.128233    2555 scope.go:117] "RemoveContainer" containerID="0990a2d3475c85fcb8b3bc23178b5d0fdf13f01494b81d9fdff9fe17187d0e9d"
Sep 17 22:15:43.137772 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:43.137754    2555 scope.go:117] "RemoveContainer" containerID="0990a2d3475c85fcb8b3bc23178b5d0fdf13f01494b81d9fdff9fe17187d0e9d"
Sep 17 22:15:43.138051 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:15:43.138033    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0990a2d3475c85fcb8b3bc23178b5d0fdf13f01494b81d9fdff9fe17187d0e9d\": container with ID starting with 0990a2d3475c85fcb8b3bc23178b5d0fdf13f01494b81d9fdff9fe17187d0e9d not found: ID does not exist" containerID="0990a2d3475c85fcb8b3bc23178b5d0fdf13f01494b81d9fdff9fe17187d0e9d"
Sep 17 22:15:43.138099 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:43.138061    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0990a2d3475c85fcb8b3bc23178b5d0fdf13f01494b81d9fdff9fe17187d0e9d"} err="failed to get container status \"0990a2d3475c85fcb8b3bc23178b5d0fdf13f01494b81d9fdff9fe17187d0e9d\": rpc error: code = NotFound desc = could not find container \"0990a2d3475c85fcb8b3bc23178b5d0fdf13f01494b81d9fdff9fe17187d0e9d\": container with ID starting with 0990a2d3475c85fcb8b3bc23178b5d0fdf13f01494b81d9fdff9fe17187d0e9d not found: ID does not exist"
Sep 17 22:15:43.150200 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:43.150163    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-zr699"]
Sep 17 22:15:43.154200 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:43.154161    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-zr699"]
Sep 17 22:15:43.778530 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:43.778489    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-zr699" podUID="7378e659-14ef-4a43-b38b-4925a388e20c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 17 22:15:44.679293 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:44.679258    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7378e659-14ef-4a43-b38b-4925a388e20c" path="/var/lib/kubelet/pods/7378e659-14ef-4a43-b38b-4925a388e20c/volumes"
Sep 17 22:15:48.365262 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:48.365229    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:15:48.370854 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:48.370828    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-4lf7w"]
Sep 17 22:15:48.380730 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:15:48.380690    2555 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 22:15:48.380730 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:15:48.380730 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:15:48.380730 ip-10-0-140-157 kubenswrapper[2555]:  > containerID="74f7aa549b5e3fc886202fe470e17c6e8ca16eb8364028c8327a82a3991a7d17" cmd=["/bin/sleep","15"]
Sep 17 22:15:48.380971 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:48.380753    2555 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-4lf7w" message="" err=<
Sep 17 22:15:48.380971 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:15:48.380971 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:15:48.380971 ip-10-0-140-157 kubenswrapper[2555]:  >
Sep 17 22:15:48.380971 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:15:48.380768    2555 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 22:15:48.380971 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:15:48.380971 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:15:48.380971 ip-10-0-140-157 kubenswrapper[2555]:  > pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-4lf7w" podUID="d2e5ef51-c042-464d-87ac-7a0508142d79" containerName="main" containerID="cri-o://74f7aa549b5e3fc886202fe470e17c6e8ca16eb8364028c8327a82a3991a7d17"
Sep 17 22:15:48.380971 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:48.380846    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-4lf7w" podUID="d2e5ef51-c042-464d-87ac-7a0508142d79" containerName="main" containerID="cri-o://74f7aa549b5e3fc886202fe470e17c6e8ca16eb8364028c8327a82a3991a7d17" gracePeriod=60
Sep 17 22:15:48.464120 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:48.464082    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:15:48.897594 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:48.897523    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-4lf7w" podUID="d2e5ef51-c042-464d-87ac-7a0508142d79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 17 22:15:49.130638 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:49.130615    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-4lf7w"
Sep 17 22:15:49.167129 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:49.167043    2555 generic.go:356] "Generic (PLEG): container finished" podID="d2e5ef51-c042-464d-87ac-7a0508142d79" containerID="74f7aa549b5e3fc886202fe470e17c6e8ca16eb8364028c8327a82a3991a7d17" exitCode=0
Sep 17 22:15:49.167129 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:49.167085    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-4lf7w"
Sep 17 22:15:49.167129 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:49.167105    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-4lf7w" event={"ID":"d2e5ef51-c042-464d-87ac-7a0508142d79","Type":"ContainerDied","Data":"74f7aa549b5e3fc886202fe470e17c6e8ca16eb8364028c8327a82a3991a7d17"}
Sep 17 22:15:49.167431 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:49.167144    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-4lf7w" event={"ID":"d2e5ef51-c042-464d-87ac-7a0508142d79","Type":"ContainerDied","Data":"94af464679e91f44b37b6b4ea713497e7d37ba46ae49bf9d6ab42f5046e7a984"}
Sep 17 22:15:49.167431 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:49.167167    2555 scope.go:117] "RemoveContainer" containerID="74f7aa549b5e3fc886202fe470e17c6e8ca16eb8364028c8327a82a3991a7d17"
Sep 17 22:15:49.178032 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:49.178003    2555 scope.go:117] "RemoveContainer" containerID="74f7aa549b5e3fc886202fe470e17c6e8ca16eb8364028c8327a82a3991a7d17"
Sep 17 22:15:49.178321 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:15:49.178302    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74f7aa549b5e3fc886202fe470e17c6e8ca16eb8364028c8327a82a3991a7d17\": container with ID starting with 74f7aa549b5e3fc886202fe470e17c6e8ca16eb8364028c8327a82a3991a7d17 not found: ID does not exist" containerID="74f7aa549b5e3fc886202fe470e17c6e8ca16eb8364028c8327a82a3991a7d17"
Sep 17 22:15:49.178403 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:49.178329    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74f7aa549b5e3fc886202fe470e17c6e8ca16eb8364028c8327a82a3991a7d17"} err="failed to get container status \"74f7aa549b5e3fc886202fe470e17c6e8ca16eb8364028c8327a82a3991a7d17\": rpc error: code = NotFound desc = could not find container \"74f7aa549b5e3fc886202fe470e17c6e8ca16eb8364028c8327a82a3991a7d17\": container with ID starting with 74f7aa549b5e3fc886202fe470e17c6e8ca16eb8364028c8327a82a3991a7d17 not found: ID does not exist"
Sep 17 22:15:49.185593 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:49.185577    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2e5ef51-c042-464d-87ac-7a0508142d79-tmp-dir\") pod \"d2e5ef51-c042-464d-87ac-7a0508142d79\" (UID: \"d2e5ef51-c042-464d-87ac-7a0508142d79\") "
Sep 17 22:15:49.185656 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:49.185606    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2e5ef51-c042-464d-87ac-7a0508142d79-model-cache\") pod \"d2e5ef51-c042-464d-87ac-7a0508142d79\" (UID: \"d2e5ef51-c042-464d-87ac-7a0508142d79\") "
Sep 17 22:15:49.185656 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:49.185647    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2e5ef51-c042-464d-87ac-7a0508142d79-tls-certs\") pod \"d2e5ef51-c042-464d-87ac-7a0508142d79\" (UID: \"d2e5ef51-c042-464d-87ac-7a0508142d79\") "
Sep 17 22:15:49.185775 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:49.185676    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pn2l5\" (UniqueName: \"kubernetes.io/projected/d2e5ef51-c042-464d-87ac-7a0508142d79-kube-api-access-pn2l5\") pod \"d2e5ef51-c042-464d-87ac-7a0508142d79\" (UID: \"d2e5ef51-c042-464d-87ac-7a0508142d79\") "
Sep 17 22:15:49.185775 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:49.185731    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2e5ef51-c042-464d-87ac-7a0508142d79-home\") pod \"d2e5ef51-c042-464d-87ac-7a0508142d79\" (UID: \"d2e5ef51-c042-464d-87ac-7a0508142d79\") "
Sep 17 22:15:49.185775 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:49.185744    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2e5ef51-c042-464d-87ac-7a0508142d79-dshm\") pod \"d2e5ef51-c042-464d-87ac-7a0508142d79\" (UID: \"d2e5ef51-c042-464d-87ac-7a0508142d79\") "
Sep 17 22:15:49.185938 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:49.185853    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2e5ef51-c042-464d-87ac-7a0508142d79-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d2e5ef51-c042-464d-87ac-7a0508142d79" (UID: "d2e5ef51-c042-464d-87ac-7a0508142d79"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:15:49.185938 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:49.185912    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2e5ef51-c042-464d-87ac-7a0508142d79-model-cache" (OuterVolumeSpecName: "model-cache") pod "d2e5ef51-c042-464d-87ac-7a0508142d79" (UID: "d2e5ef51-c042-464d-87ac-7a0508142d79"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:15:49.186031 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:49.185959    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2e5ef51-c042-464d-87ac-7a0508142d79-tmp-dir\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:15:49.186139 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:49.186117    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2e5ef51-c042-464d-87ac-7a0508142d79-home" (OuterVolumeSpecName: "home") pod "d2e5ef51-c042-464d-87ac-7a0508142d79" (UID: "d2e5ef51-c042-464d-87ac-7a0508142d79"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:15:49.187949 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:49.187922    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2e5ef51-c042-464d-87ac-7a0508142d79-dshm" (OuterVolumeSpecName: "dshm") pod "d2e5ef51-c042-464d-87ac-7a0508142d79" (UID: "d2e5ef51-c042-464d-87ac-7a0508142d79"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:15:49.188046 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:49.188026    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2e5ef51-c042-464d-87ac-7a0508142d79-kube-api-access-pn2l5" (OuterVolumeSpecName: "kube-api-access-pn2l5") pod "d2e5ef51-c042-464d-87ac-7a0508142d79" (UID: "d2e5ef51-c042-464d-87ac-7a0508142d79"). InnerVolumeSpecName "kube-api-access-pn2l5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:15:49.188046 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:49.188036    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2e5ef51-c042-464d-87ac-7a0508142d79-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d2e5ef51-c042-464d-87ac-7a0508142d79" (UID: "d2e5ef51-c042-464d-87ac-7a0508142d79"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:15:49.286578 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:49.286543    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2e5ef51-c042-464d-87ac-7a0508142d79-model-cache\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:15:49.286578 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:49.286574    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2e5ef51-c042-464d-87ac-7a0508142d79-tls-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:15:49.286578 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:49.286584    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn2l5\" (UniqueName: \"kubernetes.io/projected/d2e5ef51-c042-464d-87ac-7a0508142d79-kube-api-access-pn2l5\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:15:49.286816 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:49.286594    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2e5ef51-c042-464d-87ac-7a0508142d79-home\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:15:49.286816 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:49.286602    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2e5ef51-c042-464d-87ac-7a0508142d79-dshm\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:15:49.488812 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:49.488772    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-4lf7w"]
Sep 17 22:15:49.491774 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:49.491749    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-4lf7w"]
Sep 17 22:15:50.679867 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:15:50.679835    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2e5ef51-c042-464d-87ac-7a0508142d79" path="/var/lib/kubelet/pods/d2e5ef51-c042-464d-87ac-7a0508142d79/volumes"
Sep 17 22:16:32.343096 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:32.343051    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:16:32.444979 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:32.444946    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:16:38.165673 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:38.165635    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:16:38.430601 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:38.430488    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:16:38.526689 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:38.526654    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:16:38.735976 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:38.735944    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x"]
Sep 17 22:16:38.736354 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:38.736340    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2e5ef51-c042-464d-87ac-7a0508142d79" containerName="main"
Sep 17 22:16:38.736403 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:38.736356    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2e5ef51-c042-464d-87ac-7a0508142d79" containerName="main"
Sep 17 22:16:38.736403 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:38.736376    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7378e659-14ef-4a43-b38b-4925a388e20c" containerName="main"
Sep 17 22:16:38.736403 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:38.736381    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7378e659-14ef-4a43-b38b-4925a388e20c" containerName="main"
Sep 17 22:16:38.736499 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:38.736440    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7378e659-14ef-4a43-b38b-4925a388e20c" containerName="main"
Sep 17 22:16:38.736499 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:38.736450    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2e5ef51-c042-464d-87ac-7a0508142d79" containerName="main"
Sep 17 22:16:38.739432 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:38.739417    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x"
Sep 17 22:16:38.742220 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:38.742196    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Sep 17 22:16:38.742602 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:38.742578    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-2xjv7\""
Sep 17 22:16:38.743090 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:38.743069    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Sep 17 22:16:38.743215 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:38.743072    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Sep 17 22:16:38.767932 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:38.767910    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x"]
Sep 17 22:16:38.801353 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:38.801330    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:16:38.899351 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:38.899316    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:16:38.911192 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:38.911150    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24e93f8-fd0c-4ded-86dc-027e50f1a286-workload-certs\") pod \"router-gateway-1-openshift-default-6b46477c5b-g5q8x\" (UID: \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x"
Sep 17 22:16:38.911319 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:38.911205    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24e93f8-fd0c-4ded-86dc-027e50f1a286-istio-podinfo\") pod \"router-gateway-1-openshift-default-6b46477c5b-g5q8x\" (UID: \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x"
Sep 17 22:16:38.911319 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:38.911275    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b24e93f8-fd0c-4ded-86dc-027e50f1a286-istio-token\") pod \"router-gateway-1-openshift-default-6b46477c5b-g5q8x\" (UID: \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x"
Sep 17 22:16:38.911394 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:38.911317    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24e93f8-fd0c-4ded-86dc-027e50f1a286-istio-data\") pod \"router-gateway-1-openshift-default-6b46477c5b-g5q8x\" (UID: \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x"
Sep 17 22:16:38.911394 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:38.911344    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24e93f8-fd0c-4ded-86dc-027e50f1a286-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6b46477c5b-g5q8x\" (UID: \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x"
Sep 17 22:16:38.911479 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:38.911397    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24e93f8-fd0c-4ded-86dc-027e50f1a286-istio-envoy\") pod \"router-gateway-1-openshift-default-6b46477c5b-g5q8x\" (UID: \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x"
Sep 17 22:16:38.911479 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:38.911420    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24e93f8-fd0c-4ded-86dc-027e50f1a286-credential-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-g5q8x\" (UID: \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x"
Sep 17 22:16:38.911479 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:38.911448    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24e93f8-fd0c-4ded-86dc-027e50f1a286-workload-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-g5q8x\" (UID: \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x"
Sep 17 22:16:38.911479 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:38.911469    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2tph\" (UniqueName: \"kubernetes.io/projected/b24e93f8-fd0c-4ded-86dc-027e50f1a286-kube-api-access-r2tph\") pod \"router-gateway-1-openshift-default-6b46477c5b-g5q8x\" (UID: \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x"
Sep 17 22:16:39.013663 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:39.013562    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b24e93f8-fd0c-4ded-86dc-027e50f1a286-istio-token\") pod \"router-gateway-1-openshift-default-6b46477c5b-g5q8x\" (UID: \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x"
Sep 17 22:16:39.014037 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:39.014017    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24e93f8-fd0c-4ded-86dc-027e50f1a286-istio-data\") pod \"router-gateway-1-openshift-default-6b46477c5b-g5q8x\" (UID: \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x"
Sep 17 22:16:39.014324 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:39.014160    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24e93f8-fd0c-4ded-86dc-027e50f1a286-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6b46477c5b-g5q8x\" (UID: \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x"
Sep 17 22:16:39.014505 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:39.014490    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24e93f8-fd0c-4ded-86dc-027e50f1a286-istio-envoy\") pod \"router-gateway-1-openshift-default-6b46477c5b-g5q8x\" (UID: \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x"
Sep 17 22:16:39.014775 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:39.014756    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24e93f8-fd0c-4ded-86dc-027e50f1a286-credential-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-g5q8x\" (UID: \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x"
Sep 17 22:16:39.014900 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:39.014881    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24e93f8-fd0c-4ded-86dc-027e50f1a286-workload-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-g5q8x\" (UID: \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x"
Sep 17 22:16:39.015013 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:39.015000    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2tph\" (UniqueName: \"kubernetes.io/projected/b24e93f8-fd0c-4ded-86dc-027e50f1a286-kube-api-access-r2tph\") pod \"router-gateway-1-openshift-default-6b46477c5b-g5q8x\" (UID: \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x"
Sep 17 22:16:39.015296 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:39.015272    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24e93f8-fd0c-4ded-86dc-027e50f1a286-workload-certs\") pod \"router-gateway-1-openshift-default-6b46477c5b-g5q8x\" (UID: \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x"
Sep 17 22:16:39.015296 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:39.015288    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24e93f8-fd0c-4ded-86dc-027e50f1a286-credential-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-g5q8x\" (UID: \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x"
Sep 17 22:16:39.015506 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:39.015326    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24e93f8-fd0c-4ded-86dc-027e50f1a286-istio-podinfo\") pod \"router-gateway-1-openshift-default-6b46477c5b-g5q8x\" (UID: \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x"
Sep 17 22:16:39.015762 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:39.015700    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24e93f8-fd0c-4ded-86dc-027e50f1a286-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6b46477c5b-g5q8x\" (UID: \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x"
Sep 17 22:16:39.015762 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:39.014762    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24e93f8-fd0c-4ded-86dc-027e50f1a286-istio-data\") pod \"router-gateway-1-openshift-default-6b46477c5b-g5q8x\" (UID: \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x"
Sep 17 22:16:39.015954 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:39.015533    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24e93f8-fd0c-4ded-86dc-027e50f1a286-workload-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-g5q8x\" (UID: \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x"
Sep 17 22:16:39.016212 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:39.016167    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24e93f8-fd0c-4ded-86dc-027e50f1a286-workload-certs\") pod \"router-gateway-1-openshift-default-6b46477c5b-g5q8x\" (UID: \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x"
Sep 17 22:16:39.018337 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:39.018311    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24e93f8-fd0c-4ded-86dc-027e50f1a286-istio-envoy\") pod \"router-gateway-1-openshift-default-6b46477c5b-g5q8x\" (UID: \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x"
Sep 17 22:16:39.018765 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:39.018741    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24e93f8-fd0c-4ded-86dc-027e50f1a286-istio-podinfo\") pod \"router-gateway-1-openshift-default-6b46477c5b-g5q8x\" (UID: \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x"
Sep 17 22:16:39.023988 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:39.023964    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b24e93f8-fd0c-4ded-86dc-027e50f1a286-istio-token\") pod \"router-gateway-1-openshift-default-6b46477c5b-g5q8x\" (UID: \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x"
Sep 17 22:16:39.024497 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:39.024480    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2tph\" (UniqueName: \"kubernetes.io/projected/b24e93f8-fd0c-4ded-86dc-027e50f1a286-kube-api-access-r2tph\") pod \"router-gateway-1-openshift-default-6b46477c5b-g5q8x\" (UID: \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x"
Sep 17 22:16:39.050463 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:39.050439    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x"
Sep 17 22:16:39.326791 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:39.326767    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x"]
Sep 17 22:16:39.328523 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:16:39.328493    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb24e93f8_fd0c_4ded_86dc_027e50f1a286.slice/crio-e867563ad486b62b5835a4977c47f76aa815737f5448c5d74d6cfb72636caa71 WatchSource:0}: Error finding container e867563ad486b62b5835a4977c47f76aa815737f5448c5d74d6cfb72636caa71: Status 404 returned error can't find the container with id e867563ad486b62b5835a4977c47f76aa815737f5448c5d74d6cfb72636caa71
Sep 17 22:16:39.330888 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:39.330855    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 22:16:39.330962 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:39.330928    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 22:16:39.330962 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:39.330957    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 22:16:39.451195 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:39.451146    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x" event={"ID":"b24e93f8-fd0c-4ded-86dc-027e50f1a286","Type":"ContainerStarted","Data":"ef5c5ef81e430634d0e78252d978ff7f2c9ce22d335c4ba4e40963b6202d0b53"}
Sep 17 22:16:39.451374 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:39.451246    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x" event={"ID":"b24e93f8-fd0c-4ded-86dc-027e50f1a286","Type":"ContainerStarted","Data":"e867563ad486b62b5835a4977c47f76aa815737f5448c5d74d6cfb72636caa71"}
Sep 17 22:16:39.472751 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:39.472697    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x" podStartSLOduration=1.472680408 podStartE2EDuration="1.472680408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:16:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:16:39.46905512 +0000 UTC m=+1100.915100398" watchObservedRunningTime="2026-09-17 22:16:39.472680408 +0000 UTC m=+1100.918725682"
Sep 17 22:16:40.050802 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:40.050764    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x"
Sep 17 22:16:40.055934 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:40.055913    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x"
Sep 17 22:16:40.456679 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:40.456640    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x"
Sep 17 22:16:40.457567 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:40.457550    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x"
Sep 17 22:16:41.975078 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:41.975041    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:16:44.168072 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:44.168035    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:16:44.268882 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:44.268848    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:16:44.666488 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:44.666458    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:16:48.468710 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:48.468674    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb"]
Sep 17 22:16:48.472429 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:48.472406    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb"
Sep 17 22:16:48.475583 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:48.475558    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 17 22:16:48.475692 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:48.475586    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 17 22:16:48.475692 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:48.475558    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-fkrzb\""
Sep 17 22:16:48.475692 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:48.475558    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 17 22:16:48.485797 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:48.485766    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb"]
Sep 17 22:16:48.498517 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:48.498494    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebb825a1-dc44-4958-99bc-90203461d6fe-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb\" (UID: \"ebb825a1-dc44-4958-99bc-90203461d6fe\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb"
Sep 17 22:16:48.498655 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:48.498523    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebb825a1-dc44-4958-99bc-90203461d6fe-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb\" (UID: \"ebb825a1-dc44-4958-99bc-90203461d6fe\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb"
Sep 17 22:16:48.498655 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:48.498554    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebb825a1-dc44-4958-99bc-90203461d6fe-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb\" (UID: \"ebb825a1-dc44-4958-99bc-90203461d6fe\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb"
Sep 17 22:16:48.498655 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:48.498618    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebb825a1-dc44-4958-99bc-90203461d6fe-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb\" (UID: \"ebb825a1-dc44-4958-99bc-90203461d6fe\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb"
Sep 17 22:16:48.498655 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:48.498634    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spsk6\" (UniqueName: \"kubernetes.io/projected/ebb825a1-dc44-4958-99bc-90203461d6fe-kube-api-access-spsk6\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb\" (UID: \"ebb825a1-dc44-4958-99bc-90203461d6fe\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb"
Sep 17 22:16:48.498655 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:48.498655    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebb825a1-dc44-4958-99bc-90203461d6fe-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb\" (UID: \"ebb825a1-dc44-4958-99bc-90203461d6fe\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb"
Sep 17 22:16:48.535762 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:48.535735    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:16:48.599997 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:48.599972    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebb825a1-dc44-4958-99bc-90203461d6fe-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb\" (UID: \"ebb825a1-dc44-4958-99bc-90203461d6fe\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb"
Sep 17 22:16:48.600142 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:48.600012    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebb825a1-dc44-4958-99bc-90203461d6fe-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb\" (UID: \"ebb825a1-dc44-4958-99bc-90203461d6fe\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb"
Sep 17 22:16:48.600142 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:48.600033    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spsk6\" (UniqueName: \"kubernetes.io/projected/ebb825a1-dc44-4958-99bc-90203461d6fe-kube-api-access-spsk6\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb\" (UID: \"ebb825a1-dc44-4958-99bc-90203461d6fe\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb"
Sep 17 22:16:48.600142 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:48.600063    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebb825a1-dc44-4958-99bc-90203461d6fe-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb\" (UID: \"ebb825a1-dc44-4958-99bc-90203461d6fe\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb"
Sep 17 22:16:48.600382 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:48.600148    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebb825a1-dc44-4958-99bc-90203461d6fe-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb\" (UID: \"ebb825a1-dc44-4958-99bc-90203461d6fe\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb"
Sep 17 22:16:48.600382 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:48.600170    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebb825a1-dc44-4958-99bc-90203461d6fe-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb\" (UID: \"ebb825a1-dc44-4958-99bc-90203461d6fe\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb"
Sep 17 22:16:48.600520 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:48.600495    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebb825a1-dc44-4958-99bc-90203461d6fe-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb\" (UID: \"ebb825a1-dc44-4958-99bc-90203461d6fe\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb"
Sep 17 22:16:48.600583 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:48.600531    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebb825a1-dc44-4958-99bc-90203461d6fe-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb\" (UID: \"ebb825a1-dc44-4958-99bc-90203461d6fe\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb"
Sep 17 22:16:48.600639 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:48.600585    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebb825a1-dc44-4958-99bc-90203461d6fe-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb\" (UID: \"ebb825a1-dc44-4958-99bc-90203461d6fe\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb"
Sep 17 22:16:48.602451 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:48.602432    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebb825a1-dc44-4958-99bc-90203461d6fe-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb\" (UID: \"ebb825a1-dc44-4958-99bc-90203461d6fe\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb"
Sep 17 22:16:48.602711 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:48.602691    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebb825a1-dc44-4958-99bc-90203461d6fe-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb\" (UID: \"ebb825a1-dc44-4958-99bc-90203461d6fe\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb"
Sep 17 22:16:48.609294 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:48.609269    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spsk6\" (UniqueName: \"kubernetes.io/projected/ebb825a1-dc44-4958-99bc-90203461d6fe-kube-api-access-spsk6\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb\" (UID: \"ebb825a1-dc44-4958-99bc-90203461d6fe\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb"
Sep 17 22:16:48.787620 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:48.787522    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb"
Sep 17 22:16:48.826553 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:48.826508    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:16:49.063807 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:49.063784    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb"]
Sep 17 22:16:49.065699 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:16:49.065660    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebb825a1_dc44_4958_99bc_90203461d6fe.slice/crio-e2331f56e51391fdd02d7562a7909c50b756968cdbe1cf4c3f0cc824520cf9b8 WatchSource:0}: Error finding container e2331f56e51391fdd02d7562a7909c50b756968cdbe1cf4c3f0cc824520cf9b8: Status 404 returned error can't find the container with id e2331f56e51391fdd02d7562a7909c50b756968cdbe1cf4c3f0cc824520cf9b8
Sep 17 22:16:49.509403 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:49.509368    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb" event={"ID":"ebb825a1-dc44-4958-99bc-90203461d6fe","Type":"ContainerStarted","Data":"b0c95c530f6e222c065d640236be772896f57ca4f0106e47dda80879a1366a19"}
Sep 17 22:16:49.509800 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:49.509423    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb" event={"ID":"ebb825a1-dc44-4958-99bc-90203461d6fe","Type":"ContainerStarted","Data":"e2331f56e51391fdd02d7562a7909c50b756968cdbe1cf4c3f0cc824520cf9b8"}
Sep 17 22:16:49.528486 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:49.528425    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb" podStartSLOduration=1.528409102 podStartE2EDuration="1.528409102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:16:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:16:49.526222207 +0000 UTC m=+1110.972267476" watchObservedRunningTime="2026-09-17 22:16:49.528409102 +0000 UTC m=+1110.974454374"
Sep 17 22:16:49.787751 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:49.787629    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb"
Sep 17 22:16:50.353617 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:50.353586    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:16:50.419057 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:50.419016    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:16:54.766837 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:54.766795    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:16:58.788281 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:58.788246    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb"
Sep 17 22:16:58.800868 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:58.800844    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb"
Sep 17 22:16:59.575949 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:16:59.575920    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb"
Sep 17 22:17:22.041678 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:22.041640    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl"]
Sep 17 22:17:22.046149 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:22.046128    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl"
Sep 17 22:17:22.049423 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:22.049392    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Sep 17 22:17:22.049423 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:22.049412    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-z6qmx\""
Sep 17 22:17:22.053460 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:22.053433    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl"]
Sep 17 22:17:22.077713 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:22.077692    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:17:22.201265 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:22.201232    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0c9bcf0-90d5-414a-9228-a35ee92619b8-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl\" (UID: \"e0c9bcf0-90d5-414a-9228-a35ee92619b8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl"
Sep 17 22:17:22.201442 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:22.201273    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0c9bcf0-90d5-414a-9228-a35ee92619b8-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl\" (UID: \"e0c9bcf0-90d5-414a-9228-a35ee92619b8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl"
Sep 17 22:17:22.201442 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:22.201301    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0c9bcf0-90d5-414a-9228-a35ee92619b8-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl\" (UID: \"e0c9bcf0-90d5-414a-9228-a35ee92619b8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl"
Sep 17 22:17:22.201442 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:22.201337    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0c9bcf0-90d5-414a-9228-a35ee92619b8-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl\" (UID: \"e0c9bcf0-90d5-414a-9228-a35ee92619b8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl"
Sep 17 22:17:22.201442 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:22.201392    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0c9bcf0-90d5-414a-9228-a35ee92619b8-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl\" (UID: \"e0c9bcf0-90d5-414a-9228-a35ee92619b8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl"
Sep 17 22:17:22.201442 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:22.201430    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfss2\" (UniqueName: \"kubernetes.io/projected/e0c9bcf0-90d5-414a-9228-a35ee92619b8-kube-api-access-wfss2\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl\" (UID: \"e0c9bcf0-90d5-414a-9228-a35ee92619b8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl"
Sep 17 22:17:22.302667 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:22.302578    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0c9bcf0-90d5-414a-9228-a35ee92619b8-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl\" (UID: \"e0c9bcf0-90d5-414a-9228-a35ee92619b8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl"
Sep 17 22:17:22.302667 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:22.302638    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfss2\" (UniqueName: \"kubernetes.io/projected/e0c9bcf0-90d5-414a-9228-a35ee92619b8-kube-api-access-wfss2\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl\" (UID: \"e0c9bcf0-90d5-414a-9228-a35ee92619b8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl"
Sep 17 22:17:22.302908 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:22.302699    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0c9bcf0-90d5-414a-9228-a35ee92619b8-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl\" (UID: \"e0c9bcf0-90d5-414a-9228-a35ee92619b8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl"
Sep 17 22:17:22.302908 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:22.302719    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0c9bcf0-90d5-414a-9228-a35ee92619b8-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl\" (UID: \"e0c9bcf0-90d5-414a-9228-a35ee92619b8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl"
Sep 17 22:17:22.302908 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:22.302741    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0c9bcf0-90d5-414a-9228-a35ee92619b8-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl\" (UID: \"e0c9bcf0-90d5-414a-9228-a35ee92619b8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl"
Sep 17 22:17:22.302908 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:22.302777    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0c9bcf0-90d5-414a-9228-a35ee92619b8-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl\" (UID: \"e0c9bcf0-90d5-414a-9228-a35ee92619b8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl"
Sep 17 22:17:22.303163 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:22.303137    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0c9bcf0-90d5-414a-9228-a35ee92619b8-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl\" (UID: \"e0c9bcf0-90d5-414a-9228-a35ee92619b8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl"
Sep 17 22:17:22.303278 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:22.303252    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0c9bcf0-90d5-414a-9228-a35ee92619b8-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl\" (UID: \"e0c9bcf0-90d5-414a-9228-a35ee92619b8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl"
Sep 17 22:17:22.303329 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:22.303272    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0c9bcf0-90d5-414a-9228-a35ee92619b8-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl\" (UID: \"e0c9bcf0-90d5-414a-9228-a35ee92619b8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl"
Sep 17 22:17:22.305070 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:22.305052    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0c9bcf0-90d5-414a-9228-a35ee92619b8-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl\" (UID: \"e0c9bcf0-90d5-414a-9228-a35ee92619b8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl"
Sep 17 22:17:22.305361 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:22.305345    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0c9bcf0-90d5-414a-9228-a35ee92619b8-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl\" (UID: \"e0c9bcf0-90d5-414a-9228-a35ee92619b8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl"
Sep 17 22:17:22.310533 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:22.310514    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfss2\" (UniqueName: \"kubernetes.io/projected/e0c9bcf0-90d5-414a-9228-a35ee92619b8-kube-api-access-wfss2\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl\" (UID: \"e0c9bcf0-90d5-414a-9228-a35ee92619b8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl"
Sep 17 22:17:22.359087 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:22.359067    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl"
Sep 17 22:17:22.637136 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:22.637105    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl"]
Sep 17 22:17:22.638588 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:17:22.638554    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0c9bcf0_90d5_414a_9228_a35ee92619b8.slice/crio-1e07eaf5c5c694715cd6ccc0445cb0e6329ac7f76b9f24c6bd66d3c83c435a4d WatchSource:0}: Error finding container 1e07eaf5c5c694715cd6ccc0445cb0e6329ac7f76b9f24c6bd66d3c83c435a4d: Status 404 returned error can't find the container with id 1e07eaf5c5c694715cd6ccc0445cb0e6329ac7f76b9f24c6bd66d3c83c435a4d
Sep 17 22:17:22.705352 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:22.705316    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl" event={"ID":"e0c9bcf0-90d5-414a-9228-a35ee92619b8","Type":"ContainerStarted","Data":"d56d762a3aef049390b78bd1927332231c73a80341a078e034ed59bc4bee69d7"}
Sep 17 22:17:22.705481 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:22.705382    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl" event={"ID":"e0c9bcf0-90d5-414a-9228-a35ee92619b8","Type":"ContainerStarted","Data":"1e07eaf5c5c694715cd6ccc0445cb0e6329ac7f76b9f24c6bd66d3c83c435a4d"}
Sep 17 22:17:22.731442 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:22.731375    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl" podStartSLOduration=0.731357965 podStartE2EDuration="731.357965ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:17:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:17:22.730642497 +0000 UTC m=+1144.176687783" watchObservedRunningTime="2026-09-17 22:17:22.731357965 +0000 UTC m=+1144.177403237"
Sep 17 22:17:23.359695 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:23.359662    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl"
Sep 17 22:17:25.061070 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:25.061036    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:17:25.162000 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:25.161965    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:17:25.277195 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:25.277147    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:17:25.430026 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:25.429991    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:17:25.522567 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:25.522535    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:17:28.217454 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:28.217423    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:17:28.829541 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:28.829504    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl"]
Sep 17 22:17:28.840041 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:17:28.839994    2555 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 22:17:28.840041 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:17:28.840041 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:17:28.840041 ip-10-0-140-157 kubenswrapper[2555]:  > containerID="d56d762a3aef049390b78bd1927332231c73a80341a078e034ed59bc4bee69d7" cmd=["/bin/sleep","15"]
Sep 17 22:17:28.840530 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:28.840047    2555 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl" message="" err=<
Sep 17 22:17:28.840530 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:17:28.840530 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:17:28.840530 ip-10-0-140-157 kubenswrapper[2555]:  >
Sep 17 22:17:28.840530 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:17:28.840061    2555 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 22:17:28.840530 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:17:28.840530 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:17:28.840530 ip-10-0-140-157 kubenswrapper[2555]:  > pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl" podUID="e0c9bcf0-90d5-414a-9228-a35ee92619b8" containerName="main" containerID="cri-o://d56d762a3aef049390b78bd1927332231c73a80341a078e034ed59bc4bee69d7"
Sep 17 22:17:28.840530 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:28.840107    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl" podUID="e0c9bcf0-90d5-414a-9228-a35ee92619b8" containerName="main" containerID="cri-o://d56d762a3aef049390b78bd1927332231c73a80341a078e034ed59bc4bee69d7" gracePeriod=60
Sep 17 22:17:28.851421 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:28.851393    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:17:29.100224 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:29.100140    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl"
Sep 17 22:17:29.268992 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:29.268959    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfss2\" (UniqueName: \"kubernetes.io/projected/e0c9bcf0-90d5-414a-9228-a35ee92619b8-kube-api-access-wfss2\") pod \"e0c9bcf0-90d5-414a-9228-a35ee92619b8\" (UID: \"e0c9bcf0-90d5-414a-9228-a35ee92619b8\") "
Sep 17 22:17:29.269451 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:29.269021    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0c9bcf0-90d5-414a-9228-a35ee92619b8-home\") pod \"e0c9bcf0-90d5-414a-9228-a35ee92619b8\" (UID: \"e0c9bcf0-90d5-414a-9228-a35ee92619b8\") "
Sep 17 22:17:29.269451 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:29.269090    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0c9bcf0-90d5-414a-9228-a35ee92619b8-tmp-dir\") pod \"e0c9bcf0-90d5-414a-9228-a35ee92619b8\" (UID: \"e0c9bcf0-90d5-414a-9228-a35ee92619b8\") "
Sep 17 22:17:29.269451 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:29.269128    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0c9bcf0-90d5-414a-9228-a35ee92619b8-dshm\") pod \"e0c9bcf0-90d5-414a-9228-a35ee92619b8\" (UID: \"e0c9bcf0-90d5-414a-9228-a35ee92619b8\") "
Sep 17 22:17:29.269451 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:29.269162    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0c9bcf0-90d5-414a-9228-a35ee92619b8-tls-certs\") pod \"e0c9bcf0-90d5-414a-9228-a35ee92619b8\" (UID: \"e0c9bcf0-90d5-414a-9228-a35ee92619b8\") "
Sep 17 22:17:29.269451 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:29.269204    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0c9bcf0-90d5-414a-9228-a35ee92619b8-model-cache\") pod \"e0c9bcf0-90d5-414a-9228-a35ee92619b8\" (UID: \"e0c9bcf0-90d5-414a-9228-a35ee92619b8\") "
Sep 17 22:17:29.269451 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:29.269360    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0c9bcf0-90d5-414a-9228-a35ee92619b8-home" (OuterVolumeSpecName: "home") pod "e0c9bcf0-90d5-414a-9228-a35ee92619b8" (UID: "e0c9bcf0-90d5-414a-9228-a35ee92619b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:17:29.269451 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:29.269429    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0c9bcf0-90d5-414a-9228-a35ee92619b8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e0c9bcf0-90d5-414a-9228-a35ee92619b8" (UID: "e0c9bcf0-90d5-414a-9228-a35ee92619b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:17:29.269802 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:29.269524    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0c9bcf0-90d5-414a-9228-a35ee92619b8-home\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:17:29.269802 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:29.269551    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0c9bcf0-90d5-414a-9228-a35ee92619b8-tmp-dir\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:17:29.269802 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:29.269607    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0c9bcf0-90d5-414a-9228-a35ee92619b8-model-cache" (OuterVolumeSpecName: "model-cache") pod "e0c9bcf0-90d5-414a-9228-a35ee92619b8" (UID: "e0c9bcf0-90d5-414a-9228-a35ee92619b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:17:29.271342 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:29.271316    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0c9bcf0-90d5-414a-9228-a35ee92619b8-kube-api-access-wfss2" (OuterVolumeSpecName: "kube-api-access-wfss2") pod "e0c9bcf0-90d5-414a-9228-a35ee92619b8" (UID: "e0c9bcf0-90d5-414a-9228-a35ee92619b8"). InnerVolumeSpecName "kube-api-access-wfss2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:17:29.271342 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:29.271329    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0c9bcf0-90d5-414a-9228-a35ee92619b8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e0c9bcf0-90d5-414a-9228-a35ee92619b8" (UID: "e0c9bcf0-90d5-414a-9228-a35ee92619b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:17:29.271579 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:29.271565    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0c9bcf0-90d5-414a-9228-a35ee92619b8-dshm" (OuterVolumeSpecName: "dshm") pod "e0c9bcf0-90d5-414a-9228-a35ee92619b8" (UID: "e0c9bcf0-90d5-414a-9228-a35ee92619b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:17:29.370791 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:29.370706    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0c9bcf0-90d5-414a-9228-a35ee92619b8-dshm\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:17:29.370791 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:29.370735    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0c9bcf0-90d5-414a-9228-a35ee92619b8-tls-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:17:29.370791 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:29.370750    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0c9bcf0-90d5-414a-9228-a35ee92619b8-model-cache\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:17:29.370791 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:29.370762    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfss2\" (UniqueName: \"kubernetes.io/projected/e0c9bcf0-90d5-414a-9228-a35ee92619b8-kube-api-access-wfss2\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:17:29.746657 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:29.746618    2555 generic.go:356] "Generic (PLEG): container finished" podID="e0c9bcf0-90d5-414a-9228-a35ee92619b8" containerID="d56d762a3aef049390b78bd1927332231c73a80341a078e034ed59bc4bee69d7" exitCode=0
Sep 17 22:17:29.746858 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:29.746674    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl"
Sep 17 22:17:29.746858 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:29.746680    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl" event={"ID":"e0c9bcf0-90d5-414a-9228-a35ee92619b8","Type":"ContainerDied","Data":"d56d762a3aef049390b78bd1927332231c73a80341a078e034ed59bc4bee69d7"}
Sep 17 22:17:29.746858 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:29.746802    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl" event={"ID":"e0c9bcf0-90d5-414a-9228-a35ee92619b8","Type":"ContainerDied","Data":"1e07eaf5c5c694715cd6ccc0445cb0e6329ac7f76b9f24c6bd66d3c83c435a4d"}
Sep 17 22:17:29.746858 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:29.746830    2555 scope.go:117] "RemoveContainer" containerID="d56d762a3aef049390b78bd1927332231c73a80341a078e034ed59bc4bee69d7"
Sep 17 22:17:29.760531 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:29.760439    2555 scope.go:117] "RemoveContainer" containerID="d56d762a3aef049390b78bd1927332231c73a80341a078e034ed59bc4bee69d7"
Sep 17 22:17:29.761009 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:17:29.760988    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d56d762a3aef049390b78bd1927332231c73a80341a078e034ed59bc4bee69d7\": container with ID starting with d56d762a3aef049390b78bd1927332231c73a80341a078e034ed59bc4bee69d7 not found: ID does not exist" containerID="d56d762a3aef049390b78bd1927332231c73a80341a078e034ed59bc4bee69d7"
Sep 17 22:17:29.761103 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:29.761016    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d56d762a3aef049390b78bd1927332231c73a80341a078e034ed59bc4bee69d7"} err="failed to get container status \"d56d762a3aef049390b78bd1927332231c73a80341a078e034ed59bc4bee69d7\": rpc error: code = NotFound desc = could not find container \"d56d762a3aef049390b78bd1927332231c73a80341a078e034ed59bc4bee69d7\": container with ID starting with d56d762a3aef049390b78bd1927332231c73a80341a078e034ed59bc4bee69d7 not found: ID does not exist"
Sep 17 22:17:29.770302 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:29.770280    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl"]
Sep 17 22:17:29.778657 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:29.778632    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bm2sl"]
Sep 17 22:17:30.049752 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:30.049663    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x"]
Sep 17 22:17:30.049946 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:30.049921    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x" podUID="b24e93f8-fd0c-4ded-86dc-027e50f1a286" containerName="istio-proxy" containerID="cri-o://ef5c5ef81e430634d0e78252d978ff7f2c9ce22d335c4ba4e40963b6202d0b53" gracePeriod=30
Sep 17 22:17:30.143550 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:30.143518    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:17:30.681672 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:30.681627    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0c9bcf0-90d5-414a-9228-a35ee92619b8" path="/var/lib/kubelet/pods/e0c9bcf0-90d5-414a-9228-a35ee92619b8/volumes"
Sep 17 22:17:32.015651 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:32.015625    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x"
Sep 17 22:17:32.090804 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:32.090775    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24e93f8-fd0c-4ded-86dc-027e50f1a286-workload-certs\") pod \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\" (UID: \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\") "
Sep 17 22:17:32.090804 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:32.090803    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24e93f8-fd0c-4ded-86dc-027e50f1a286-istio-envoy\") pod \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\" (UID: \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\") "
Sep 17 22:17:32.091033 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:32.090824    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24e93f8-fd0c-4ded-86dc-027e50f1a286-credential-socket\") pod \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\" (UID: \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\") "
Sep 17 22:17:32.091033 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:32.090853    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24e93f8-fd0c-4ded-86dc-027e50f1a286-istiod-ca-cert\") pod \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\" (UID: \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\") "
Sep 17 22:17:32.091033 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:32.090883    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2tph\" (UniqueName: \"kubernetes.io/projected/b24e93f8-fd0c-4ded-86dc-027e50f1a286-kube-api-access-r2tph\") pod \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\" (UID: \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\") "
Sep 17 22:17:32.091033 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:32.090904    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b24e93f8-fd0c-4ded-86dc-027e50f1a286-istio-token\") pod \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\" (UID: \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\") "
Sep 17 22:17:32.091033 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:32.090927    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24e93f8-fd0c-4ded-86dc-027e50f1a286-istio-data\") pod \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\" (UID: \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\") "
Sep 17 22:17:32.091033 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:32.090979    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24e93f8-fd0c-4ded-86dc-027e50f1a286-workload-socket\") pod \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\" (UID: \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\") "
Sep 17 22:17:32.091033 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:32.091015    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24e93f8-fd0c-4ded-86dc-027e50f1a286-istio-podinfo\") pod \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\" (UID: \"b24e93f8-fd0c-4ded-86dc-027e50f1a286\") "
Sep 17 22:17:32.091447 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:32.091079    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24e93f8-fd0c-4ded-86dc-027e50f1a286-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b24e93f8-fd0c-4ded-86dc-027e50f1a286" (UID: "b24e93f8-fd0c-4ded-86dc-027e50f1a286"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:17:32.091447 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:32.091165    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24e93f8-fd0c-4ded-86dc-027e50f1a286-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b24e93f8-fd0c-4ded-86dc-027e50f1a286" (UID: "b24e93f8-fd0c-4ded-86dc-027e50f1a286"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:17:32.091447 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:32.091305    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b24e93f8-fd0c-4ded-86dc-027e50f1a286-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b24e93f8-fd0c-4ded-86dc-027e50f1a286" (UID: "b24e93f8-fd0c-4ded-86dc-027e50f1a286"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:17:32.091447 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:32.091349    2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24e93f8-fd0c-4ded-86dc-027e50f1a286-workload-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:17:32.091447 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:32.091369    2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24e93f8-fd0c-4ded-86dc-027e50f1a286-credential-socket\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:17:32.091713 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:32.091589    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24e93f8-fd0c-4ded-86dc-027e50f1a286-istio-data" (OuterVolumeSpecName: "istio-data") pod "b24e93f8-fd0c-4ded-86dc-027e50f1a286" (UID: "b24e93f8-fd0c-4ded-86dc-027e50f1a286"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:17:32.091713 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:32.091613    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24e93f8-fd0c-4ded-86dc-027e50f1a286-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b24e93f8-fd0c-4ded-86dc-027e50f1a286" (UID: "b24e93f8-fd0c-4ded-86dc-027e50f1a286"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:17:32.093326 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:32.093304    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b24e93f8-fd0c-4ded-86dc-027e50f1a286-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b24e93f8-fd0c-4ded-86dc-027e50f1a286" (UID: "b24e93f8-fd0c-4ded-86dc-027e50f1a286"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 22:17:32.093472 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:32.093455    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b24e93f8-fd0c-4ded-86dc-027e50f1a286-istio-token" (OuterVolumeSpecName: "istio-token") pod "b24e93f8-fd0c-4ded-86dc-027e50f1a286" (UID: "b24e93f8-fd0c-4ded-86dc-027e50f1a286"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:17:32.093551 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:32.093525    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24e93f8-fd0c-4ded-86dc-027e50f1a286-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b24e93f8-fd0c-4ded-86dc-027e50f1a286" (UID: "b24e93f8-fd0c-4ded-86dc-027e50f1a286"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:17:32.093616 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:32.093603    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b24e93f8-fd0c-4ded-86dc-027e50f1a286-kube-api-access-r2tph" (OuterVolumeSpecName: "kube-api-access-r2tph") pod "b24e93f8-fd0c-4ded-86dc-027e50f1a286" (UID: "b24e93f8-fd0c-4ded-86dc-027e50f1a286"). InnerVolumeSpecName "kube-api-access-r2tph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:17:32.192537 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:32.192504    2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24e93f8-fd0c-4ded-86dc-027e50f1a286-istio-envoy\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:17:32.192537 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:32.192530    2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24e93f8-fd0c-4ded-86dc-027e50f1a286-istiod-ca-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:17:32.192537 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:32.192539    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2tph\" (UniqueName: \"kubernetes.io/projected/b24e93f8-fd0c-4ded-86dc-027e50f1a286-kube-api-access-r2tph\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:17:32.192794 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:32.192550    2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b24e93f8-fd0c-4ded-86dc-027e50f1a286-istio-token\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:17:32.192794 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:32.192559    2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24e93f8-fd0c-4ded-86dc-027e50f1a286-istio-data\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:17:32.192794 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:32.192566    2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24e93f8-fd0c-4ded-86dc-027e50f1a286-workload-socket\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:17:32.192794 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:32.192576    2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24e93f8-fd0c-4ded-86dc-027e50f1a286-istio-podinfo\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:17:32.766893 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:32.766856    2555 generic.go:356] "Generic (PLEG): container finished" podID="b24e93f8-fd0c-4ded-86dc-027e50f1a286" containerID="ef5c5ef81e430634d0e78252d978ff7f2c9ce22d335c4ba4e40963b6202d0b53" exitCode=0
Sep 17 22:17:32.767061 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:32.766908    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x" event={"ID":"b24e93f8-fd0c-4ded-86dc-027e50f1a286","Type":"ContainerDied","Data":"ef5c5ef81e430634d0e78252d978ff7f2c9ce22d335c4ba4e40963b6202d0b53"}
Sep 17 22:17:32.767061 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:32.766957    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x" event={"ID":"b24e93f8-fd0c-4ded-86dc-027e50f1a286","Type":"ContainerDied","Data":"e867563ad486b62b5835a4977c47f76aa815737f5448c5d74d6cfb72636caa71"}
Sep 17 22:17:32.767061 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:32.766913    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x"
Sep 17 22:17:32.767230 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:32.766984    2555 scope.go:117] "RemoveContainer" containerID="ef5c5ef81e430634d0e78252d978ff7f2c9ce22d335c4ba4e40963b6202d0b53"
Sep 17 22:17:32.777384 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:32.777353    2555 scope.go:117] "RemoveContainer" containerID="ef5c5ef81e430634d0e78252d978ff7f2c9ce22d335c4ba4e40963b6202d0b53"
Sep 17 22:17:32.777650 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:17:32.777630    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef5c5ef81e430634d0e78252d978ff7f2c9ce22d335c4ba4e40963b6202d0b53\": container with ID starting with ef5c5ef81e430634d0e78252d978ff7f2c9ce22d335c4ba4e40963b6202d0b53 not found: ID does not exist" containerID="ef5c5ef81e430634d0e78252d978ff7f2c9ce22d335c4ba4e40963b6202d0b53"
Sep 17 22:17:32.777730 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:32.777657    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef5c5ef81e430634d0e78252d978ff7f2c9ce22d335c4ba4e40963b6202d0b53"} err="failed to get container status \"ef5c5ef81e430634d0e78252d978ff7f2c9ce22d335c4ba4e40963b6202d0b53\": rpc error: code = NotFound desc = could not find container \"ef5c5ef81e430634d0e78252d978ff7f2c9ce22d335c4ba4e40963b6202d0b53\": container with ID starting with ef5c5ef81e430634d0e78252d978ff7f2c9ce22d335c4ba4e40963b6202d0b53 not found: ID does not exist"
Sep 17 22:17:32.786068 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:32.786042    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x"]
Sep 17 22:17:32.790080 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:32.790061    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-g5q8x"]
Sep 17 22:17:34.679502 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:34.679425    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b24e93f8-fd0c-4ded-86dc-027e50f1a286" path="/var/lib/kubelet/pods/b24e93f8-fd0c-4ded-86dc-027e50f1a286/volumes"
Sep 17 22:17:35.464438 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:35.464400    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:17:36.192416 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:36.192383    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:17:39.017811 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:39.017782    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:17:40.937778 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:40.937745    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7"]
Sep 17 22:17:40.938271 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:40.938254    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b24e93f8-fd0c-4ded-86dc-027e50f1a286" containerName="istio-proxy"
Sep 17 22:17:40.938372 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:40.938273    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b24e93f8-fd0c-4ded-86dc-027e50f1a286" containerName="istio-proxy"
Sep 17 22:17:40.938372 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:40.938297    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0c9bcf0-90d5-414a-9228-a35ee92619b8" containerName="main"
Sep 17 22:17:40.938372 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:40.938304    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0c9bcf0-90d5-414a-9228-a35ee92619b8" containerName="main"
Sep 17 22:17:40.938539 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:40.938375    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b24e93f8-fd0c-4ded-86dc-027e50f1a286" containerName="istio-proxy"
Sep 17 22:17:40.938539 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:40.938387    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0c9bcf0-90d5-414a-9228-a35ee92619b8" containerName="main"
Sep 17 22:17:40.941464 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:40.941443    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7"
Sep 17 22:17:40.943761 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:40.943732    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 17 22:17:40.944583 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:40.944561    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 17 22:17:40.944709 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:40.944577    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Sep 17 22:17:40.944709 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:40.944698    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-tj9gv\""
Sep 17 22:17:40.951008 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:40.950985    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7"]
Sep 17 22:17:40.967070 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:40.967043    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51860aef-6a79-4443-ac26-b90d019d7b14-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7\" (UID: \"51860aef-6a79-4443-ac26-b90d019d7b14\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7"
Sep 17 22:17:40.967234 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:40.967100    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51860aef-6a79-4443-ac26-b90d019d7b14-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7\" (UID: \"51860aef-6a79-4443-ac26-b90d019d7b14\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7"
Sep 17 22:17:40.967234 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:40.967126    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:17:40.967234 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:40.967131    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51860aef-6a79-4443-ac26-b90d019d7b14-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7\" (UID: \"51860aef-6a79-4443-ac26-b90d019d7b14\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7"
Sep 17 22:17:40.967427 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:40.967301    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51860aef-6a79-4443-ac26-b90d019d7b14-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7\" (UID: \"51860aef-6a79-4443-ac26-b90d019d7b14\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7"
Sep 17 22:17:40.967427 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:40.967330    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg6d8\" (UniqueName: \"kubernetes.io/projected/51860aef-6a79-4443-ac26-b90d019d7b14-kube-api-access-hg6d8\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7\" (UID: \"51860aef-6a79-4443-ac26-b90d019d7b14\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7"
Sep 17 22:17:40.967427 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:40.967372    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51860aef-6a79-4443-ac26-b90d019d7b14-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7\" (UID: \"51860aef-6a79-4443-ac26-b90d019d7b14\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7"
Sep 17 22:17:41.068305 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:41.068271    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51860aef-6a79-4443-ac26-b90d019d7b14-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7\" (UID: \"51860aef-6a79-4443-ac26-b90d019d7b14\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7"
Sep 17 22:17:41.068511 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:41.068312    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51860aef-6a79-4443-ac26-b90d019d7b14-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7\" (UID: \"51860aef-6a79-4443-ac26-b90d019d7b14\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7"
Sep 17 22:17:41.068511 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:41.068335    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51860aef-6a79-4443-ac26-b90d019d7b14-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7\" (UID: \"51860aef-6a79-4443-ac26-b90d019d7b14\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7"
Sep 17 22:17:41.068511 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:41.068396    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51860aef-6a79-4443-ac26-b90d019d7b14-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7\" (UID: \"51860aef-6a79-4443-ac26-b90d019d7b14\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7"
Sep 17 22:17:41.068511 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:41.068422    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg6d8\" (UniqueName: \"kubernetes.io/projected/51860aef-6a79-4443-ac26-b90d019d7b14-kube-api-access-hg6d8\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7\" (UID: \"51860aef-6a79-4443-ac26-b90d019d7b14\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7"
Sep 17 22:17:41.068724 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:41.068584    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51860aef-6a79-4443-ac26-b90d019d7b14-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7\" (UID: \"51860aef-6a79-4443-ac26-b90d019d7b14\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7"
Sep 17 22:17:41.068796 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:41.068740    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51860aef-6a79-4443-ac26-b90d019d7b14-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7\" (UID: \"51860aef-6a79-4443-ac26-b90d019d7b14\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7"
Sep 17 22:17:41.068796 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:41.068753    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51860aef-6a79-4443-ac26-b90d019d7b14-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7\" (UID: \"51860aef-6a79-4443-ac26-b90d019d7b14\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7"
Sep 17 22:17:41.068908 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:41.068839    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51860aef-6a79-4443-ac26-b90d019d7b14-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7\" (UID: \"51860aef-6a79-4443-ac26-b90d019d7b14\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7"
Sep 17 22:17:41.070746 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:41.070720    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51860aef-6a79-4443-ac26-b90d019d7b14-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7\" (UID: \"51860aef-6a79-4443-ac26-b90d019d7b14\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7"
Sep 17 22:17:41.071076 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:41.071054    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51860aef-6a79-4443-ac26-b90d019d7b14-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7\" (UID: \"51860aef-6a79-4443-ac26-b90d019d7b14\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7"
Sep 17 22:17:41.076351 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:41.076330    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg6d8\" (UniqueName: \"kubernetes.io/projected/51860aef-6a79-4443-ac26-b90d019d7b14-kube-api-access-hg6d8\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7\" (UID: \"51860aef-6a79-4443-ac26-b90d019d7b14\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7"
Sep 17 22:17:41.253811 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:41.253720    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7"
Sep 17 22:17:41.533166 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:41.533139    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7"]
Sep 17 22:17:41.535012 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:17:41.534989    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51860aef_6a79_4443_ac26_b90d019d7b14.slice/crio-c5ecd584bc76e03f51203202a171f836d947bd09e78956750a079840cf7ad63e WatchSource:0}: Error finding container c5ecd584bc76e03f51203202a171f836d947bd09e78956750a079840cf7ad63e: Status 404 returned error can't find the container with id c5ecd584bc76e03f51203202a171f836d947bd09e78956750a079840cf7ad63e
Sep 17 22:17:41.821354 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:41.821262    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7" event={"ID":"51860aef-6a79-4443-ac26-b90d019d7b14","Type":"ContainerStarted","Data":"f8198960942852dfd6a12d637b0bd2bcfff5da3dfc2092ed2600fb57b0416564"}
Sep 17 22:17:41.821354 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:41.821318    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7" event={"ID":"51860aef-6a79-4443-ac26-b90d019d7b14","Type":"ContainerStarted","Data":"c5ecd584bc76e03f51203202a171f836d947bd09e78956750a079840cf7ad63e"}
Sep 17 22:17:41.837892 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:41.837837    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7" podStartSLOduration=1.8378224570000001 podStartE2EDuration="1.837822457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:17:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:17:41.836364861 +0000 UTC m=+1163.282410135" watchObservedRunningTime="2026-09-17 22:17:41.837822457 +0000 UTC m=+1163.283867728"
Sep 17 22:17:42.254174 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:42.254143    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7"
Sep 17 22:17:43.748037 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:43.748004    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:17:43.761233 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:43.761200    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb"]
Sep 17 22:17:43.770854 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:17:43.770815    2555 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 22:17:43.770854 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:17:43.770854 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:17:43.770854 ip-10-0-140-157 kubenswrapper[2555]:  > containerID="b0c95c530f6e222c065d640236be772896f57ca4f0106e47dda80879a1366a19" cmd=["/bin/sleep","15"]
Sep 17 22:17:43.771167 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:43.770879    2555 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb" message="" err=<
Sep 17 22:17:43.771167 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:17:43.771167 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:17:43.771167 ip-10-0-140-157 kubenswrapper[2555]:  >
Sep 17 22:17:43.771167 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:17:43.770896    2555 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 22:17:43.771167 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:17:43.771167 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:17:43.771167 ip-10-0-140-157 kubenswrapper[2555]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb" podUID="ebb825a1-dc44-4958-99bc-90203461d6fe" containerName="main" containerID="cri-o://b0c95c530f6e222c065d640236be772896f57ca4f0106e47dda80879a1366a19"
Sep 17 22:17:43.771167 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:43.770941    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb" podUID="ebb825a1-dc44-4958-99bc-90203461d6fe" containerName="main" containerID="cri-o://b0c95c530f6e222c065d640236be772896f57ca4f0106e47dda80879a1366a19" gracePeriod=60
Sep 17 22:17:43.847711 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:43.847681    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:17:43.945384 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:43.945352    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:17:44.124599 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:44.124579    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb"
Sep 17 22:17:44.199927 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:44.199893    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebb825a1-dc44-4958-99bc-90203461d6fe-model-cache\") pod \"ebb825a1-dc44-4958-99bc-90203461d6fe\" (UID: \"ebb825a1-dc44-4958-99bc-90203461d6fe\") "
Sep 17 22:17:44.200104 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:44.199958    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-spsk6\" (UniqueName: \"kubernetes.io/projected/ebb825a1-dc44-4958-99bc-90203461d6fe-kube-api-access-spsk6\") pod \"ebb825a1-dc44-4958-99bc-90203461d6fe\" (UID: \"ebb825a1-dc44-4958-99bc-90203461d6fe\") "
Sep 17 22:17:44.200104 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:44.200016    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebb825a1-dc44-4958-99bc-90203461d6fe-home\") pod \"ebb825a1-dc44-4958-99bc-90203461d6fe\" (UID: \"ebb825a1-dc44-4958-99bc-90203461d6fe\") "
Sep 17 22:17:44.200104 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:44.200060    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebb825a1-dc44-4958-99bc-90203461d6fe-tmp-dir\") pod \"ebb825a1-dc44-4958-99bc-90203461d6fe\" (UID: \"ebb825a1-dc44-4958-99bc-90203461d6fe\") "
Sep 17 22:17:44.200304 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:44.200108    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebb825a1-dc44-4958-99bc-90203461d6fe-dshm\") pod \"ebb825a1-dc44-4958-99bc-90203461d6fe\" (UID: \"ebb825a1-dc44-4958-99bc-90203461d6fe\") "
Sep 17 22:17:44.200304 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:44.200153    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebb825a1-dc44-4958-99bc-90203461d6fe-tls-certs\") pod \"ebb825a1-dc44-4958-99bc-90203461d6fe\" (UID: \"ebb825a1-dc44-4958-99bc-90203461d6fe\") "
Sep 17 22:17:44.200304 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:44.200153    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebb825a1-dc44-4958-99bc-90203461d6fe-model-cache" (OuterVolumeSpecName: "model-cache") pod "ebb825a1-dc44-4958-99bc-90203461d6fe" (UID: "ebb825a1-dc44-4958-99bc-90203461d6fe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:17:44.200515 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:44.200320    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebb825a1-dc44-4958-99bc-90203461d6fe-home" (OuterVolumeSpecName: "home") pod "ebb825a1-dc44-4958-99bc-90203461d6fe" (UID: "ebb825a1-dc44-4958-99bc-90203461d6fe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:17:44.200515 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:44.200373    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebb825a1-dc44-4958-99bc-90203461d6fe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ebb825a1-dc44-4958-99bc-90203461d6fe" (UID: "ebb825a1-dc44-4958-99bc-90203461d6fe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:17:44.200603 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:44.200548    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebb825a1-dc44-4958-99bc-90203461d6fe-tmp-dir\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:17:44.200603 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:44.200574    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebb825a1-dc44-4958-99bc-90203461d6fe-model-cache\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:17:44.200603 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:44.200591    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebb825a1-dc44-4958-99bc-90203461d6fe-home\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:17:44.202336 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:44.202310    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebb825a1-dc44-4958-99bc-90203461d6fe-dshm" (OuterVolumeSpecName: "dshm") pod "ebb825a1-dc44-4958-99bc-90203461d6fe" (UID: "ebb825a1-dc44-4958-99bc-90203461d6fe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:17:44.202442 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:44.202389    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebb825a1-dc44-4958-99bc-90203461d6fe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ebb825a1-dc44-4958-99bc-90203461d6fe" (UID: "ebb825a1-dc44-4958-99bc-90203461d6fe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:17:44.202502 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:44.202447    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebb825a1-dc44-4958-99bc-90203461d6fe-kube-api-access-spsk6" (OuterVolumeSpecName: "kube-api-access-spsk6") pod "ebb825a1-dc44-4958-99bc-90203461d6fe" (UID: "ebb825a1-dc44-4958-99bc-90203461d6fe"). InnerVolumeSpecName "kube-api-access-spsk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:17:44.301822 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:44.301723    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebb825a1-dc44-4958-99bc-90203461d6fe-dshm\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:17:44.301822 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:44.301758    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebb825a1-dc44-4958-99bc-90203461d6fe-tls-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:17:44.301822 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:44.301786    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spsk6\" (UniqueName: \"kubernetes.io/projected/ebb825a1-dc44-4958-99bc-90203461d6fe-kube-api-access-spsk6\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:17:44.532276 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:44.532241    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:17:44.533940 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:44.533916    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7"]
Sep 17 22:17:44.839550 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:44.839517    2555 generic.go:356] "Generic (PLEG): container finished" podID="ebb825a1-dc44-4958-99bc-90203461d6fe" containerID="b0c95c530f6e222c065d640236be772896f57ca4f0106e47dda80879a1366a19" exitCode=0
Sep 17 22:17:44.840037 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:44.839570    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb"
Sep 17 22:17:44.840037 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:44.839608    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb" event={"ID":"ebb825a1-dc44-4958-99bc-90203461d6fe","Type":"ContainerDied","Data":"b0c95c530f6e222c065d640236be772896f57ca4f0106e47dda80879a1366a19"}
Sep 17 22:17:44.840037 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:44.839671    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb" event={"ID":"ebb825a1-dc44-4958-99bc-90203461d6fe","Type":"ContainerDied","Data":"e2331f56e51391fdd02d7562a7909c50b756968cdbe1cf4c3f0cc824520cf9b8"}
Sep 17 22:17:44.840037 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:44.839707    2555 scope.go:117] "RemoveContainer" containerID="b0c95c530f6e222c065d640236be772896f57ca4f0106e47dda80879a1366a19"
Sep 17 22:17:44.851644 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:17:44.851263    2555 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 22:17:44.851644 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:17:44.851644 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:17:44.851644 ip-10-0-140-157 kubenswrapper[2555]:  > containerID="f8198960942852dfd6a12d637b0bd2bcfff5da3dfc2092ed2600fb57b0416564" cmd=["/bin/sleep","15"]
Sep 17 22:17:44.851644 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:44.851314    2555 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7" message="" err=<
Sep 17 22:17:44.851644 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:17:44.851644 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:17:44.851644 ip-10-0-140-157 kubenswrapper[2555]:  >
Sep 17 22:17:44.851644 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:17:44.851333    2555 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 22:17:44.851644 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:17:44.851644 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:17:44.851644 ip-10-0-140-157 kubenswrapper[2555]:  > pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7" podUID="51860aef-6a79-4443-ac26-b90d019d7b14" containerName="main" containerID="cri-o://f8198960942852dfd6a12d637b0bd2bcfff5da3dfc2092ed2600fb57b0416564"
Sep 17 22:17:44.851644 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:44.851377    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7" podUID="51860aef-6a79-4443-ac26-b90d019d7b14" containerName="main" containerID="cri-o://f8198960942852dfd6a12d637b0bd2bcfff5da3dfc2092ed2600fb57b0416564" gracePeriod=60
Sep 17 22:17:44.853132 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:44.853115    2555 scope.go:117] "RemoveContainer" containerID="b0c95c530f6e222c065d640236be772896f57ca4f0106e47dda80879a1366a19"
Sep 17 22:17:44.853473 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:17:44.853449    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0c95c530f6e222c065d640236be772896f57ca4f0106e47dda80879a1366a19\": container with ID starting with b0c95c530f6e222c065d640236be772896f57ca4f0106e47dda80879a1366a19 not found: ID does not exist" containerID="b0c95c530f6e222c065d640236be772896f57ca4f0106e47dda80879a1366a19"
Sep 17 22:17:44.853559 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:44.853482    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0c95c530f6e222c065d640236be772896f57ca4f0106e47dda80879a1366a19"} err="failed to get container status \"b0c95c530f6e222c065d640236be772896f57ca4f0106e47dda80879a1366a19\": rpc error: code = NotFound desc = could not find container \"b0c95c530f6e222c065d640236be772896f57ca4f0106e47dda80879a1366a19\": container with ID starting with b0c95c530f6e222c065d640236be772896f57ca4f0106e47dda80879a1366a19 not found: ID does not exist"
Sep 17 22:17:44.857899 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:44.857878    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb"]
Sep 17 22:17:44.866925 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:44.862898    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8f9xb"]
Sep 17 22:17:45.115814 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:45.115787    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7"
Sep 17 22:17:45.209750 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:45.209720    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51860aef-6a79-4443-ac26-b90d019d7b14-tmp-dir\") pod \"51860aef-6a79-4443-ac26-b90d019d7b14\" (UID: \"51860aef-6a79-4443-ac26-b90d019d7b14\") "
Sep 17 22:17:45.209939 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:45.209808    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51860aef-6a79-4443-ac26-b90d019d7b14-tls-certs\") pod \"51860aef-6a79-4443-ac26-b90d019d7b14\" (UID: \"51860aef-6a79-4443-ac26-b90d019d7b14\") "
Sep 17 22:17:45.209939 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:45.209842    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51860aef-6a79-4443-ac26-b90d019d7b14-dshm\") pod \"51860aef-6a79-4443-ac26-b90d019d7b14\" (UID: \"51860aef-6a79-4443-ac26-b90d019d7b14\") "
Sep 17 22:17:45.209939 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:45.209877    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hg6d8\" (UniqueName: \"kubernetes.io/projected/51860aef-6a79-4443-ac26-b90d019d7b14-kube-api-access-hg6d8\") pod \"51860aef-6a79-4443-ac26-b90d019d7b14\" (UID: \"51860aef-6a79-4443-ac26-b90d019d7b14\") "
Sep 17 22:17:45.209939 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:45.209906    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51860aef-6a79-4443-ac26-b90d019d7b14-home\") pod \"51860aef-6a79-4443-ac26-b90d019d7b14\" (UID: \"51860aef-6a79-4443-ac26-b90d019d7b14\") "
Sep 17 22:17:45.209939 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:45.209928    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51860aef-6a79-4443-ac26-b90d019d7b14-model-cache\") pod \"51860aef-6a79-4443-ac26-b90d019d7b14\" (UID: \"51860aef-6a79-4443-ac26-b90d019d7b14\") "
Sep 17 22:17:45.210258 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:45.210020    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51860aef-6a79-4443-ac26-b90d019d7b14-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "51860aef-6a79-4443-ac26-b90d019d7b14" (UID: "51860aef-6a79-4443-ac26-b90d019d7b14"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:17:45.210258 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:45.210165    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51860aef-6a79-4443-ac26-b90d019d7b14-home" (OuterVolumeSpecName: "home") pod "51860aef-6a79-4443-ac26-b90d019d7b14" (UID: "51860aef-6a79-4443-ac26-b90d019d7b14"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:17:45.210258 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:45.210234    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51860aef-6a79-4443-ac26-b90d019d7b14-model-cache" (OuterVolumeSpecName: "model-cache") pod "51860aef-6a79-4443-ac26-b90d019d7b14" (UID: "51860aef-6a79-4443-ac26-b90d019d7b14"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:17:45.210388 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:45.210351    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51860aef-6a79-4443-ac26-b90d019d7b14-home\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:17:45.210388 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:45.210381    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51860aef-6a79-4443-ac26-b90d019d7b14-tmp-dir\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:17:45.212277 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:45.212239    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51860aef-6a79-4443-ac26-b90d019d7b14-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "51860aef-6a79-4443-ac26-b90d019d7b14" (UID: "51860aef-6a79-4443-ac26-b90d019d7b14"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:17:45.212427 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:45.212406    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51860aef-6a79-4443-ac26-b90d019d7b14-dshm" (OuterVolumeSpecName: "dshm") pod "51860aef-6a79-4443-ac26-b90d019d7b14" (UID: "51860aef-6a79-4443-ac26-b90d019d7b14"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:17:45.212526 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:45.212506    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51860aef-6a79-4443-ac26-b90d019d7b14-kube-api-access-hg6d8" (OuterVolumeSpecName: "kube-api-access-hg6d8") pod "51860aef-6a79-4443-ac26-b90d019d7b14" (UID: "51860aef-6a79-4443-ac26-b90d019d7b14"). InnerVolumeSpecName "kube-api-access-hg6d8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:17:45.311595 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:45.311500    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg6d8\" (UniqueName: \"kubernetes.io/projected/51860aef-6a79-4443-ac26-b90d019d7b14-kube-api-access-hg6d8\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:17:45.311595 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:45.311524    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51860aef-6a79-4443-ac26-b90d019d7b14-model-cache\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:17:45.311595 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:45.311534    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51860aef-6a79-4443-ac26-b90d019d7b14-tls-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:17:45.311595 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:45.311542    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51860aef-6a79-4443-ac26-b90d019d7b14-dshm\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:17:45.847596 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:45.847562    2555 generic.go:356] "Generic (PLEG): container finished" podID="51860aef-6a79-4443-ac26-b90d019d7b14" containerID="f8198960942852dfd6a12d637b0bd2bcfff5da3dfc2092ed2600fb57b0416564" exitCode=0
Sep 17 22:17:45.848087 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:45.847611    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7"
Sep 17 22:17:45.848087 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:45.847653    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7" event={"ID":"51860aef-6a79-4443-ac26-b90d019d7b14","Type":"ContainerDied","Data":"f8198960942852dfd6a12d637b0bd2bcfff5da3dfc2092ed2600fb57b0416564"}
Sep 17 22:17:45.848087 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:45.847722    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7" event={"ID":"51860aef-6a79-4443-ac26-b90d019d7b14","Type":"ContainerDied","Data":"c5ecd584bc76e03f51203202a171f836d947bd09e78956750a079840cf7ad63e"}
Sep 17 22:17:45.848087 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:45.847760    2555 scope.go:117] "RemoveContainer" containerID="f8198960942852dfd6a12d637b0bd2bcfff5da3dfc2092ed2600fb57b0416564"
Sep 17 22:17:45.857622 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:45.857605    2555 scope.go:117] "RemoveContainer" containerID="f8198960942852dfd6a12d637b0bd2bcfff5da3dfc2092ed2600fb57b0416564"
Sep 17 22:17:45.857925 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:17:45.857895    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8198960942852dfd6a12d637b0bd2bcfff5da3dfc2092ed2600fb57b0416564\": container with ID starting with f8198960942852dfd6a12d637b0bd2bcfff5da3dfc2092ed2600fb57b0416564 not found: ID does not exist" containerID="f8198960942852dfd6a12d637b0bd2bcfff5da3dfc2092ed2600fb57b0416564"
Sep 17 22:17:45.857993 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:45.857938    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8198960942852dfd6a12d637b0bd2bcfff5da3dfc2092ed2600fb57b0416564"} err="failed to get container status \"f8198960942852dfd6a12d637b0bd2bcfff5da3dfc2092ed2600fb57b0416564\": rpc error: code = NotFound desc = could not find container \"f8198960942852dfd6a12d637b0bd2bcfff5da3dfc2092ed2600fb57b0416564\": container with ID starting with f8198960942852dfd6a12d637b0bd2bcfff5da3dfc2092ed2600fb57b0416564 not found: ID does not exist"
Sep 17 22:17:45.870319 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:45.870297    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7"]
Sep 17 22:17:45.873759 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:45.873737    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfcvng7"]
Sep 17 22:17:46.679386 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:46.679355    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51860aef-6a79-4443-ac26-b90d019d7b14" path="/var/lib/kubelet/pods/51860aef-6a79-4443-ac26-b90d019d7b14/volumes"
Sep 17 22:17:46.679767 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:46.679754    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebb825a1-dc44-4958-99bc-90203461d6fe" path="/var/lib/kubelet/pods/ebb825a1-dc44-4958-99bc-90203461d6fe/volumes"
Sep 17 22:17:46.876390 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:46.876362    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:17:48.784000 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:48.783969    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:17:51.071376 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:51.071291    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:17:51.171343 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:51.171308    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:17:54.185533 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:54.185494    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:17:54.920106 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:17:54.920076    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:18:00.526544 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:00.526514    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:18:06.420667 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:06.420629    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl"]
Sep 17 22:18:06.421217 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:06.421196    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51860aef-6a79-4443-ac26-b90d019d7b14" containerName="main"
Sep 17 22:18:06.421314 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:06.421221    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="51860aef-6a79-4443-ac26-b90d019d7b14" containerName="main"
Sep 17 22:18:06.421314 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:06.421238    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebb825a1-dc44-4958-99bc-90203461d6fe" containerName="main"
Sep 17 22:18:06.421314 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:06.421247    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebb825a1-dc44-4958-99bc-90203461d6fe" containerName="main"
Sep 17 22:18:06.421478 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:06.421345    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="51860aef-6a79-4443-ac26-b90d019d7b14" containerName="main"
Sep 17 22:18:06.421478 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:06.421359    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebb825a1-dc44-4958-99bc-90203461d6fe" containerName="main"
Sep 17 22:18:06.426195 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:06.426157    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl"
Sep 17 22:18:06.428893 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:06.428867    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 17 22:18:06.429037 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:06.428868    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:18:06.429037 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:06.428868    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 17 22:18:06.429199 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:06.429112    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-wwc9t\""
Sep 17 22:18:06.429276 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:06.429260    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Sep 17 22:18:06.429377 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:06.429356    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 17 22:18:06.435808 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:06.435787    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl"]
Sep 17 22:18:06.450478 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:06.450448    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:18:06.590755 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:06.590723    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68e3ddff-b35b-420e-8fb3-0b814406dc4d-dshm\") pod \"auth-enabled-test-kserve-5c48865899-cpmsl\" (UID: \"68e3ddff-b35b-420e-8fb3-0b814406dc4d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl"
Sep 17 22:18:06.590755 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:06.590762    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68e3ddff-b35b-420e-8fb3-0b814406dc4d-cabundle-cert\") pod \"auth-enabled-test-kserve-5c48865899-cpmsl\" (UID: \"68e3ddff-b35b-420e-8fb3-0b814406dc4d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl"
Sep 17 22:18:06.591029 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:06.590782    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68e3ddff-b35b-420e-8fb3-0b814406dc4d-tmp-dir\") pod \"auth-enabled-test-kserve-5c48865899-cpmsl\" (UID: \"68e3ddff-b35b-420e-8fb3-0b814406dc4d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl"
Sep 17 22:18:06.591029 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:06.590809    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68e3ddff-b35b-420e-8fb3-0b814406dc4d-home\") pod \"auth-enabled-test-kserve-5c48865899-cpmsl\" (UID: \"68e3ddff-b35b-420e-8fb3-0b814406dc4d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl"
Sep 17 22:18:06.591029 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:06.590899    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68e3ddff-b35b-420e-8fb3-0b814406dc4d-tls-certs\") pod \"auth-enabled-test-kserve-5c48865899-cpmsl\" (UID: \"68e3ddff-b35b-420e-8fb3-0b814406dc4d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl"
Sep 17 22:18:06.591029 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:06.590973    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68e3ddff-b35b-420e-8fb3-0b814406dc4d-kserve-provision-location\") pod \"auth-enabled-test-kserve-5c48865899-cpmsl\" (UID: \"68e3ddff-b35b-420e-8fb3-0b814406dc4d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl"
Sep 17 22:18:06.591029 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:06.591000    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68e3ddff-b35b-420e-8fb3-0b814406dc4d-model-cache\") pod \"auth-enabled-test-kserve-5c48865899-cpmsl\" (UID: \"68e3ddff-b35b-420e-8fb3-0b814406dc4d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl"
Sep 17 22:18:06.591029 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:06.591029    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkmgb\" (UniqueName: \"kubernetes.io/projected/68e3ddff-b35b-420e-8fb3-0b814406dc4d-kube-api-access-bkmgb\") pod \"auth-enabled-test-kserve-5c48865899-cpmsl\" (UID: \"68e3ddff-b35b-420e-8fb3-0b814406dc4d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl"
Sep 17 22:18:06.650051 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:06.650020    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:18:06.692128 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:06.692029    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68e3ddff-b35b-420e-8fb3-0b814406dc4d-dshm\") pod \"auth-enabled-test-kserve-5c48865899-cpmsl\" (UID: \"68e3ddff-b35b-420e-8fb3-0b814406dc4d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl"
Sep 17 22:18:06.692128 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:06.692075    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68e3ddff-b35b-420e-8fb3-0b814406dc4d-cabundle-cert\") pod \"auth-enabled-test-kserve-5c48865899-cpmsl\" (UID: \"68e3ddff-b35b-420e-8fb3-0b814406dc4d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl"
Sep 17 22:18:06.692128 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:06.692096    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68e3ddff-b35b-420e-8fb3-0b814406dc4d-tmp-dir\") pod \"auth-enabled-test-kserve-5c48865899-cpmsl\" (UID: \"68e3ddff-b35b-420e-8fb3-0b814406dc4d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl"
Sep 17 22:18:06.692128 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:06.692123    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68e3ddff-b35b-420e-8fb3-0b814406dc4d-home\") pod \"auth-enabled-test-kserve-5c48865899-cpmsl\" (UID: \"68e3ddff-b35b-420e-8fb3-0b814406dc4d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl"
Sep 17 22:18:06.692554 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:06.692152    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68e3ddff-b35b-420e-8fb3-0b814406dc4d-tls-certs\") pod \"auth-enabled-test-kserve-5c48865899-cpmsl\" (UID: \"68e3ddff-b35b-420e-8fb3-0b814406dc4d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl"
Sep 17 22:18:06.692554 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:06.692192    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68e3ddff-b35b-420e-8fb3-0b814406dc4d-kserve-provision-location\") pod \"auth-enabled-test-kserve-5c48865899-cpmsl\" (UID: \"68e3ddff-b35b-420e-8fb3-0b814406dc4d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl"
Sep 17 22:18:06.692554 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:06.692220    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68e3ddff-b35b-420e-8fb3-0b814406dc4d-model-cache\") pod \"auth-enabled-test-kserve-5c48865899-cpmsl\" (UID: \"68e3ddff-b35b-420e-8fb3-0b814406dc4d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl"
Sep 17 22:18:06.692554 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:06.692254    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkmgb\" (UniqueName: \"kubernetes.io/projected/68e3ddff-b35b-420e-8fb3-0b814406dc4d-kube-api-access-bkmgb\") pod \"auth-enabled-test-kserve-5c48865899-cpmsl\" (UID: \"68e3ddff-b35b-420e-8fb3-0b814406dc4d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl"
Sep 17 22:18:06.692794 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:06.692551    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68e3ddff-b35b-420e-8fb3-0b814406dc4d-tmp-dir\") pod \"auth-enabled-test-kserve-5c48865899-cpmsl\" (UID: \"68e3ddff-b35b-420e-8fb3-0b814406dc4d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl"
Sep 17 22:18:06.692794 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:06.692629    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68e3ddff-b35b-420e-8fb3-0b814406dc4d-home\") pod \"auth-enabled-test-kserve-5c48865899-cpmsl\" (UID: \"68e3ddff-b35b-420e-8fb3-0b814406dc4d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl"
Sep 17 22:18:06.692794 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:06.692776    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68e3ddff-b35b-420e-8fb3-0b814406dc4d-model-cache\") pod \"auth-enabled-test-kserve-5c48865899-cpmsl\" (UID: \"68e3ddff-b35b-420e-8fb3-0b814406dc4d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl"
Sep 17 22:18:06.692964 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:06.692808    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68e3ddff-b35b-420e-8fb3-0b814406dc4d-kserve-provision-location\") pod \"auth-enabled-test-kserve-5c48865899-cpmsl\" (UID: \"68e3ddff-b35b-420e-8fb3-0b814406dc4d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl"
Sep 17 22:18:06.692964 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:06.692865    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68e3ddff-b35b-420e-8fb3-0b814406dc4d-cabundle-cert\") pod \"auth-enabled-test-kserve-5c48865899-cpmsl\" (UID: \"68e3ddff-b35b-420e-8fb3-0b814406dc4d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl"
Sep 17 22:18:06.694528 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:06.694504    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68e3ddff-b35b-420e-8fb3-0b814406dc4d-dshm\") pod \"auth-enabled-test-kserve-5c48865899-cpmsl\" (UID: \"68e3ddff-b35b-420e-8fb3-0b814406dc4d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl"
Sep 17 22:18:06.694702 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:06.694682    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68e3ddff-b35b-420e-8fb3-0b814406dc4d-tls-certs\") pod \"auth-enabled-test-kserve-5c48865899-cpmsl\" (UID: \"68e3ddff-b35b-420e-8fb3-0b814406dc4d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl"
Sep 17 22:18:06.700246 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:06.700221    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkmgb\" (UniqueName: \"kubernetes.io/projected/68e3ddff-b35b-420e-8fb3-0b814406dc4d-kube-api-access-bkmgb\") pod \"auth-enabled-test-kserve-5c48865899-cpmsl\" (UID: \"68e3ddff-b35b-420e-8fb3-0b814406dc4d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl"
Sep 17 22:18:06.738051 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:06.738018    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl"
Sep 17 22:18:07.019825 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:07.019633    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl"]
Sep 17 22:18:07.022498 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:18:07.022468    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68e3ddff_b35b_420e_8fb3_0b814406dc4d.slice/crio-a51005d9fb3a06934b79519b37aa39cfffceb3b2f6e386006d51fd26b86c9255 WatchSource:0}: Error finding container a51005d9fb3a06934b79519b37aa39cfffceb3b2f6e386006d51fd26b86c9255: Status 404 returned error can't find the container with id a51005d9fb3a06934b79519b37aa39cfffceb3b2f6e386006d51fd26b86c9255
Sep 17 22:18:07.971494 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:07.971460    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl" event={"ID":"68e3ddff-b35b-420e-8fb3-0b814406dc4d","Type":"ContainerStarted","Data":"21cd399546a935c8af51eb9967502a32195fd5da882557cb442e8dd86ed9bced"}
Sep 17 22:18:07.971935 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:07.971517    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl" event={"ID":"68e3ddff-b35b-420e-8fb3-0b814406dc4d","Type":"ContainerStarted","Data":"a51005d9fb3a06934b79519b37aa39cfffceb3b2f6e386006d51fd26b86c9255"}
Sep 17 22:18:09.297742 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:09.297711    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:18:09.394453 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:09.394421    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:18:09.985835 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:09.985800    2555 generic.go:356] "Generic (PLEG): container finished" podID="68e3ddff-b35b-420e-8fb3-0b814406dc4d" containerID="21cd399546a935c8af51eb9967502a32195fd5da882557cb442e8dd86ed9bced" exitCode=0
Sep 17 22:18:09.986012 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:09.985882    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl" event={"ID":"68e3ddff-b35b-420e-8fb3-0b814406dc4d","Type":"ContainerDied","Data":"21cd399546a935c8af51eb9967502a32195fd5da882557cb442e8dd86ed9bced"}
Sep 17 22:18:09.987564 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:09.987549    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 22:18:22.654759 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:22.654730    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-7njvj_a324c9ab-5d42-4d31-8ff7-449f619be774/console-operator/1.log"
Sep 17 22:18:22.655234 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:22.655142    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-7njvj_a324c9ab-5d42-4d31-8ff7-449f619be774/console-operator/1.log"
Sep 17 22:18:22.659740 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:22.659719    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s7f5_4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf/ovn-acl-logging/0.log"
Sep 17 22:18:22.659992 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:22.659978    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s7f5_4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf/ovn-acl-logging/0.log"
Sep 17 22:18:25.726995 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:25.726959    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:18:32.041315 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:32.041282    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:18:32.144305 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:32.144253    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:18:38.103515 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:38.103477    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:18:40.208810 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:40.208766    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl" event={"ID":"68e3ddff-b35b-420e-8fb3-0b814406dc4d","Type":"ContainerStarted","Data":"f7ca0d6796e9679e570115aaca393c5af85ba0bf0a47fb5024dd6e82ea6b51f1"}
Sep 17 22:18:40.231927 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:40.231880    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl" podStartSLOduration=1.904379381 podStartE2EDuration="34.231864993s" totalImagesPullingTime="30.028730922s" totalInitContainerRuntime="2.29875469s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:18:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:18:40.230287869 +0000 UTC m=+1221.676333141" watchObservedRunningTime="2026-09-17 22:18:40.231864993 +0000 UTC m=+1221.677910321"
Sep 17 22:18:40.738698 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:40.738668    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl"
Sep 17 22:18:43.749065 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:43.749032    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:18:43.849057 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:43.849023    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:18:44.498920 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:44.498885    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7"]
Sep 17 22:18:44.538200 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:44.538143    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:18:44.732542 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:44.732508    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7"
Sep 17 22:18:44.732762 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:44.732510    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7"]
Sep 17 22:18:44.735893 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:44.735870    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 17 22:18:44.736118 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:44.735898    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 17 22:18:44.736217 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:44.735913    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 17 22:18:44.736217 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:44.735974    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-d5jld\""
Sep 17 22:18:44.816891 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:44.816797    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfkglf"]
Sep 17 22:18:44.832054 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:44.832025    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfkglf"]
Sep 17 22:18:44.832234 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:44.832142    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfkglf"
Sep 17 22:18:44.834766 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:44.834743    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-rdg7p\""
Sep 17 22:18:44.849010 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:44.848982    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:18:44.859966 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:44.859941    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b935adf9-1a52-4e12-b842-f9d4bb4eee52-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7\" (UID: \"b935adf9-1a52-4e12-b842-f9d4bb4eee52\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7"
Sep 17 22:18:44.860077 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:44.859979    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b935adf9-1a52-4e12-b842-f9d4bb4eee52-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7\" (UID: \"b935adf9-1a52-4e12-b842-f9d4bb4eee52\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7"
Sep 17 22:18:44.860140 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:44.860074    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b935adf9-1a52-4e12-b842-f9d4bb4eee52-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7\" (UID: \"b935adf9-1a52-4e12-b842-f9d4bb4eee52\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7"
Sep 17 22:18:44.860321 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:44.860148    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b935adf9-1a52-4e12-b842-f9d4bb4eee52-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7\" (UID: \"b935adf9-1a52-4e12-b842-f9d4bb4eee52\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7"
Sep 17 22:18:44.860321 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:44.860173    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxb2q\" (UniqueName: \"kubernetes.io/projected/b935adf9-1a52-4e12-b842-f9d4bb4eee52-kube-api-access-cxb2q\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7\" (UID: \"b935adf9-1a52-4e12-b842-f9d4bb4eee52\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7"
Sep 17 22:18:44.860321 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:44.860223    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b935adf9-1a52-4e12-b842-f9d4bb4eee52-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7\" (UID: \"b935adf9-1a52-4e12-b842-f9d4bb4eee52\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7"
Sep 17 22:18:44.961547 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:44.961510    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b935adf9-1a52-4e12-b842-f9d4bb4eee52-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7\" (UID: \"b935adf9-1a52-4e12-b842-f9d4bb4eee52\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7"
Sep 17 22:18:44.961724 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:44.961560    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b935adf9-1a52-4e12-b842-f9d4bb4eee52-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7\" (UID: \"b935adf9-1a52-4e12-b842-f9d4bb4eee52\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7"
Sep 17 22:18:44.961724 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:44.961581    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxb2q\" (UniqueName: \"kubernetes.io/projected/b935adf9-1a52-4e12-b842-f9d4bb4eee52-kube-api-access-cxb2q\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7\" (UID: \"b935adf9-1a52-4e12-b842-f9d4bb4eee52\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7"
Sep 17 22:18:44.961724 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:44.961690    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b935adf9-1a52-4e12-b842-f9d4bb4eee52-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7\" (UID: \"b935adf9-1a52-4e12-b842-f9d4bb4eee52\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7"
Sep 17 22:18:44.961903 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:44.961731    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b935adf9-1a52-4e12-b842-f9d4bb4eee52-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7\" (UID: \"b935adf9-1a52-4e12-b842-f9d4bb4eee52\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7"
Sep 17 22:18:44.961903 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:44.961759    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88f672f3-5f43-4ffa-8e54-e80f3fdcf293-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedfkglf\" (UID: \"88f672f3-5f43-4ffa-8e54-e80f3fdcf293\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfkglf"
Sep 17 22:18:44.961903 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:44.961798    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8x8n\" (UniqueName: \"kubernetes.io/projected/88f672f3-5f43-4ffa-8e54-e80f3fdcf293-kube-api-access-l8x8n\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedfkglf\" (UID: \"88f672f3-5f43-4ffa-8e54-e80f3fdcf293\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfkglf"
Sep 17 22:18:44.961903 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:44.961842    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b935adf9-1a52-4e12-b842-f9d4bb4eee52-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7\" (UID: \"b935adf9-1a52-4e12-b842-f9d4bb4eee52\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7"
Sep 17 22:18:44.962143 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:44.962015    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b935adf9-1a52-4e12-b842-f9d4bb4eee52-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7\" (UID: \"b935adf9-1a52-4e12-b842-f9d4bb4eee52\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7"
Sep 17 22:18:44.962264 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:44.962247    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b935adf9-1a52-4e12-b842-f9d4bb4eee52-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7\" (UID: \"b935adf9-1a52-4e12-b842-f9d4bb4eee52\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7"
Sep 17 22:18:44.962329 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:44.962291    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b935adf9-1a52-4e12-b842-f9d4bb4eee52-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7\" (UID: \"b935adf9-1a52-4e12-b842-f9d4bb4eee52\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7"
Sep 17 22:18:44.964596 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:44.964577    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b935adf9-1a52-4e12-b842-f9d4bb4eee52-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7\" (UID: \"b935adf9-1a52-4e12-b842-f9d4bb4eee52\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7"
Sep 17 22:18:44.964873 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:44.964852    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b935adf9-1a52-4e12-b842-f9d4bb4eee52-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7\" (UID: \"b935adf9-1a52-4e12-b842-f9d4bb4eee52\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7"
Sep 17 22:18:44.970102 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:44.970078    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxb2q\" (UniqueName: \"kubernetes.io/projected/b935adf9-1a52-4e12-b842-f9d4bb4eee52-kube-api-access-cxb2q\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7\" (UID: \"b935adf9-1a52-4e12-b842-f9d4bb4eee52\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7"
Sep 17 22:18:45.044397 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:45.044368    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7"
Sep 17 22:18:45.062749 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:45.062719    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88f672f3-5f43-4ffa-8e54-e80f3fdcf293-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedfkglf\" (UID: \"88f672f3-5f43-4ffa-8e54-e80f3fdcf293\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfkglf"
Sep 17 22:18:45.062889 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:45.062756    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8x8n\" (UniqueName: \"kubernetes.io/projected/88f672f3-5f43-4ffa-8e54-e80f3fdcf293-kube-api-access-l8x8n\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedfkglf\" (UID: \"88f672f3-5f43-4ffa-8e54-e80f3fdcf293\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfkglf"
Sep 17 22:18:45.065532 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:45.065502    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88f672f3-5f43-4ffa-8e54-e80f3fdcf293-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedfkglf\" (UID: \"88f672f3-5f43-4ffa-8e54-e80f3fdcf293\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfkglf"
Sep 17 22:18:45.071898 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:45.071827    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8x8n\" (UniqueName: \"kubernetes.io/projected/88f672f3-5f43-4ffa-8e54-e80f3fdcf293-kube-api-access-l8x8n\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedfkglf\" (UID: \"88f672f3-5f43-4ffa-8e54-e80f3fdcf293\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfkglf"
Sep 17 22:18:45.145826 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:45.145793    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfkglf"
Sep 17 22:18:45.366924 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:45.366892    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7"]
Sep 17 22:18:45.368647 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:18:45.368621    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb935adf9_1a52_4e12_b842_f9d4bb4eee52.slice/crio-707c019f64b01e4ae915dbfa0e047ae42c140eb47d38b93685ac93d01eeb3ec6 WatchSource:0}: Error finding container 707c019f64b01e4ae915dbfa0e047ae42c140eb47d38b93685ac93d01eeb3ec6: Status 404 returned error can't find the container with id 707c019f64b01e4ae915dbfa0e047ae42c140eb47d38b93685ac93d01eeb3ec6
Sep 17 22:18:45.435077 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:45.435041    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfkglf"]
Sep 17 22:18:45.437122 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:18:45.437084    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88f672f3_5f43_4ffa_8e54_e80f3fdcf293.slice/crio-79a6c3a254025f29fedb9653f6b8cadbd9118fbcbdcd6406d1055ef0a0a5d68c WatchSource:0}: Error finding container 79a6c3a254025f29fedb9653f6b8cadbd9118fbcbdcd6406d1055ef0a0a5d68c: Status 404 returned error can't find the container with id 79a6c3a254025f29fedb9653f6b8cadbd9118fbcbdcd6406d1055ef0a0a5d68c
Sep 17 22:18:46.247963 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:46.247923    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7" event={"ID":"b935adf9-1a52-4e12-b842-f9d4bb4eee52","Type":"ContainerStarted","Data":"d6ed58adbaf04764b5802637be297871516fbb592aa564e78150d36cfad49a1c"}
Sep 17 22:18:46.248538 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:46.248005    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7" event={"ID":"b935adf9-1a52-4e12-b842-f9d4bb4eee52","Type":"ContainerStarted","Data":"707c019f64b01e4ae915dbfa0e047ae42c140eb47d38b93685ac93d01eeb3ec6"}
Sep 17 22:18:46.249808 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:46.249779    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfkglf" event={"ID":"88f672f3-5f43-4ffa-8e54-e80f3fdcf293","Type":"ContainerStarted","Data":"40c44062df8b2f664eb8a75112bab7f6defbe25685c7588896dafc7be29895d0"}
Sep 17 22:18:46.249917 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:46.249832    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfkglf" event={"ID":"88f672f3-5f43-4ffa-8e54-e80f3fdcf293","Type":"ContainerStarted","Data":"79a6c3a254025f29fedb9653f6b8cadbd9118fbcbdcd6406d1055ef0a0a5d68c"}
Sep 17 22:18:46.249917 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:46.249881    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfkglf"
Sep 17 22:18:46.267751 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:46.267709    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7" podStartSLOduration=2.267695441 podStartE2EDuration="2.267695441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:18:46.265848385 +0000 UTC m=+1227.711893658" watchObservedRunningTime="2026-09-17 22:18:46.267695441 +0000 UTC m=+1227.713740759"
Sep 17 22:18:46.286576 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:46.286534    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfkglf" podStartSLOduration=2.286521364 podStartE2EDuration="2.286521364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:18:46.284371547 +0000 UTC m=+1227.730416820" watchObservedRunningTime="2026-09-17 22:18:46.286521364 +0000 UTC m=+1227.732566636"
Sep 17 22:18:46.738574 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:46.738538    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl"
Sep 17 22:18:46.740405 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:46.740370    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl" podUID="68e3ddff-b35b-420e-8fb3-0b814406dc4d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 17 22:18:47.045332 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:47.045248    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7"
Sep 17 22:18:55.045194 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:55.045129    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7"
Sep 17 22:18:55.057707 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:55.057679    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7"
Sep 17 22:18:55.317485 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:55.317400    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7"
Sep 17 22:18:56.739391 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:56.739352    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl" podUID="68e3ddff-b35b-420e-8fb3-0b814406dc4d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 17 22:18:59.043872 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:18:59.043775    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:19:06.738619 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:06.738521    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl" podUID="68e3ddff-b35b-420e-8fb3-0b814406dc4d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 17 22:19:16.738434 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:16.738387    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl" podUID="68e3ddff-b35b-420e-8fb3-0b814406dc4d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 17 22:19:17.257602 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:17.257573    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfkglf"
Sep 17 22:19:21.361002 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:21.360968    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:19:21.362620 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:21.362593    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfkglf"]
Sep 17 22:19:21.371956 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:21.371927    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7"]
Sep 17 22:19:21.386658 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:19:21.386620    2555 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 22:19:21.386658 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:19:21.386658 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:19:21.386658 ip-10-0-140-157 kubenswrapper[2555]:  > containerID="d6ed58adbaf04764b5802637be297871516fbb592aa564e78150d36cfad49a1c" cmd=["/bin/sleep","15"]
Sep 17 22:19:21.386987 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:21.386683    2555 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7" message="" err=<
Sep 17 22:19:21.386987 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:19:21.386987 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:19:21.386987 ip-10-0-140-157 kubenswrapper[2555]:  >
Sep 17 22:19:21.386987 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:19:21.386702    2555 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 22:19:21.386987 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:19:21.386987 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:19:21.386987 ip-10-0-140-157 kubenswrapper[2555]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7" podUID="b935adf9-1a52-4e12-b842-f9d4bb4eee52" containerName="main" containerID="cri-o://d6ed58adbaf04764b5802637be297871516fbb592aa564e78150d36cfad49a1c"
Sep 17 22:19:21.386987 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:21.386759    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7" podUID="b935adf9-1a52-4e12-b842-f9d4bb4eee52" containerName="main" containerID="cri-o://d6ed58adbaf04764b5802637be297871516fbb592aa564e78150d36cfad49a1c" gracePeriod=60
Sep 17 22:19:21.428430 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:21.428398    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:19:22.145921 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:22.145898    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7"
Sep 17 22:19:22.193540 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:22.193505    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b935adf9-1a52-4e12-b842-f9d4bb4eee52-model-cache\") pod \"b935adf9-1a52-4e12-b842-f9d4bb4eee52\" (UID: \"b935adf9-1a52-4e12-b842-f9d4bb4eee52\") "
Sep 17 22:19:22.193540 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:22.193553    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b935adf9-1a52-4e12-b842-f9d4bb4eee52-dshm\") pod \"b935adf9-1a52-4e12-b842-f9d4bb4eee52\" (UID: \"b935adf9-1a52-4e12-b842-f9d4bb4eee52\") "
Sep 17 22:19:22.193786 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:22.193581    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxb2q\" (UniqueName: \"kubernetes.io/projected/b935adf9-1a52-4e12-b842-f9d4bb4eee52-kube-api-access-cxb2q\") pod \"b935adf9-1a52-4e12-b842-f9d4bb4eee52\" (UID: \"b935adf9-1a52-4e12-b842-f9d4bb4eee52\") "
Sep 17 22:19:22.193786 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:22.193671    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b935adf9-1a52-4e12-b842-f9d4bb4eee52-tmp-dir\") pod \"b935adf9-1a52-4e12-b842-f9d4bb4eee52\" (UID: \"b935adf9-1a52-4e12-b842-f9d4bb4eee52\") "
Sep 17 22:19:22.193786 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:22.193703    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b935adf9-1a52-4e12-b842-f9d4bb4eee52-home\") pod \"b935adf9-1a52-4e12-b842-f9d4bb4eee52\" (UID: \"b935adf9-1a52-4e12-b842-f9d4bb4eee52\") "
Sep 17 22:19:22.193786 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:22.193734    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b935adf9-1a52-4e12-b842-f9d4bb4eee52-tls-certs\") pod \"b935adf9-1a52-4e12-b842-f9d4bb4eee52\" (UID: \"b935adf9-1a52-4e12-b842-f9d4bb4eee52\") "
Sep 17 22:19:22.193982 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:22.193788    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b935adf9-1a52-4e12-b842-f9d4bb4eee52-model-cache" (OuterVolumeSpecName: "model-cache") pod "b935adf9-1a52-4e12-b842-f9d4bb4eee52" (UID: "b935adf9-1a52-4e12-b842-f9d4bb4eee52"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:19:22.193982 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:22.193969    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b935adf9-1a52-4e12-b842-f9d4bb4eee52-model-cache\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:19:22.194217 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:22.194172    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b935adf9-1a52-4e12-b842-f9d4bb4eee52-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b935adf9-1a52-4e12-b842-f9d4bb4eee52" (UID: "b935adf9-1a52-4e12-b842-f9d4bb4eee52"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:19:22.194321 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:22.194224    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b935adf9-1a52-4e12-b842-f9d4bb4eee52-home" (OuterVolumeSpecName: "home") pod "b935adf9-1a52-4e12-b842-f9d4bb4eee52" (UID: "b935adf9-1a52-4e12-b842-f9d4bb4eee52"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:19:22.195970 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:22.195937    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b935adf9-1a52-4e12-b842-f9d4bb4eee52-dshm" (OuterVolumeSpecName: "dshm") pod "b935adf9-1a52-4e12-b842-f9d4bb4eee52" (UID: "b935adf9-1a52-4e12-b842-f9d4bb4eee52"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:19:22.196505 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:22.196471    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b935adf9-1a52-4e12-b842-f9d4bb4eee52-kube-api-access-cxb2q" (OuterVolumeSpecName: "kube-api-access-cxb2q") pod "b935adf9-1a52-4e12-b842-f9d4bb4eee52" (UID: "b935adf9-1a52-4e12-b842-f9d4bb4eee52"). InnerVolumeSpecName "kube-api-access-cxb2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:19:22.196505 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:22.196497    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b935adf9-1a52-4e12-b842-f9d4bb4eee52-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b935adf9-1a52-4e12-b842-f9d4bb4eee52" (UID: "b935adf9-1a52-4e12-b842-f9d4bb4eee52"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:19:22.295415 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:22.295326    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b935adf9-1a52-4e12-b842-f9d4bb4eee52-tmp-dir\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:19:22.295415 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:22.295364    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b935adf9-1a52-4e12-b842-f9d4bb4eee52-home\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:19:22.295415 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:22.295378    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b935adf9-1a52-4e12-b842-f9d4bb4eee52-tls-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:19:22.295415 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:22.295390    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b935adf9-1a52-4e12-b842-f9d4bb4eee52-dshm\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:19:22.295415 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:22.295404    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxb2q\" (UniqueName: \"kubernetes.io/projected/b935adf9-1a52-4e12-b842-f9d4bb4eee52-kube-api-access-cxb2q\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:19:22.478077 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:22.478036    2555 generic.go:356] "Generic (PLEG): container finished" podID="b935adf9-1a52-4e12-b842-f9d4bb4eee52" containerID="d6ed58adbaf04764b5802637be297871516fbb592aa564e78150d36cfad49a1c" exitCode=0
Sep 17 22:19:22.478600 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:22.478094    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7"
Sep 17 22:19:22.478600 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:22.478126    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7" event={"ID":"b935adf9-1a52-4e12-b842-f9d4bb4eee52","Type":"ContainerDied","Data":"d6ed58adbaf04764b5802637be297871516fbb592aa564e78150d36cfad49a1c"}
Sep 17 22:19:22.478600 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:22.478203    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7" event={"ID":"b935adf9-1a52-4e12-b842-f9d4bb4eee52","Type":"ContainerDied","Data":"707c019f64b01e4ae915dbfa0e047ae42c140eb47d38b93685ac93d01eeb3ec6"}
Sep 17 22:19:22.478600 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:22.478227    2555 scope.go:117] "RemoveContainer" containerID="d6ed58adbaf04764b5802637be297871516fbb592aa564e78150d36cfad49a1c"
Sep 17 22:19:22.489359 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:22.489333    2555 scope.go:117] "RemoveContainer" containerID="d6ed58adbaf04764b5802637be297871516fbb592aa564e78150d36cfad49a1c"
Sep 17 22:19:22.489675 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:19:22.489651    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6ed58adbaf04764b5802637be297871516fbb592aa564e78150d36cfad49a1c\": container with ID starting with d6ed58adbaf04764b5802637be297871516fbb592aa564e78150d36cfad49a1c not found: ID does not exist" containerID="d6ed58adbaf04764b5802637be297871516fbb592aa564e78150d36cfad49a1c"
Sep 17 22:19:22.489753 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:22.489684    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6ed58adbaf04764b5802637be297871516fbb592aa564e78150d36cfad49a1c"} err="failed to get container status \"d6ed58adbaf04764b5802637be297871516fbb592aa564e78150d36cfad49a1c\": rpc error: code = NotFound desc = could not find container \"d6ed58adbaf04764b5802637be297871516fbb592aa564e78150d36cfad49a1c\": container with ID starting with d6ed58adbaf04764b5802637be297871516fbb592aa564e78150d36cfad49a1c not found: ID does not exist"
Sep 17 22:19:22.507948 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:22.507919    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7"]
Sep 17 22:19:22.508894 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:22.508874    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644487xr7"]
Sep 17 22:19:22.679268 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:22.679241    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b935adf9-1a52-4e12-b842-f9d4bb4eee52" path="/var/lib/kubelet/pods/b935adf9-1a52-4e12-b842-f9d4bb4eee52/volumes"
Sep 17 22:19:26.738841 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:26.738802    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl" podUID="68e3ddff-b35b-420e-8fb3-0b814406dc4d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 17 22:19:27.257319 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:27.257277    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfkglf" podUID="88f672f3-5f43-4ffa-8e54-e80f3fdcf293" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:19:36.387085 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:36.387011    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfkglf" podUID="88f672f3-5f43-4ffa-8e54-e80f3fdcf293" containerName="main" containerID="cri-o://40c44062df8b2f664eb8a75112bab7f6defbe25685c7588896dafc7be29895d0" gracePeriod=45
Sep 17 22:19:36.739114 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:36.739005    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl" podUID="68e3ddff-b35b-420e-8fb3-0b814406dc4d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 17 22:19:37.257488 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:37.257446    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfkglf" podUID="88f672f3-5f43-4ffa-8e54-e80f3fdcf293" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:19:46.738775 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:46.738730    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl" podUID="68e3ddff-b35b-420e-8fb3-0b814406dc4d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 17 22:19:47.257240 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:47.257198    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfkglf" podUID="88f672f3-5f43-4ffa-8e54-e80f3fdcf293" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:19:47.257447 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:47.257337    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfkglf"
Sep 17 22:19:56.739076 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:56.739022    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl" podUID="68e3ddff-b35b-420e-8fb3-0b814406dc4d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 17 22:19:57.257299 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:19:57.257251    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfkglf" podUID="88f672f3-5f43-4ffa-8e54-e80f3fdcf293" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:20:06.651377 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:06.651308    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfkglf"
Sep 17 22:20:06.704599 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:06.704567    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8x8n\" (UniqueName: \"kubernetes.io/projected/88f672f3-5f43-4ffa-8e54-e80f3fdcf293-kube-api-access-l8x8n\") pod \"88f672f3-5f43-4ffa-8e54-e80f3fdcf293\" (UID: \"88f672f3-5f43-4ffa-8e54-e80f3fdcf293\") "
Sep 17 22:20:06.704832 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:06.704682    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88f672f3-5f43-4ffa-8e54-e80f3fdcf293-tls-certs\") pod \"88f672f3-5f43-4ffa-8e54-e80f3fdcf293\" (UID: \"88f672f3-5f43-4ffa-8e54-e80f3fdcf293\") "
Sep 17 22:20:06.706865 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:06.706830    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88f672f3-5f43-4ffa-8e54-e80f3fdcf293-kube-api-access-l8x8n" (OuterVolumeSpecName: "kube-api-access-l8x8n") pod "88f672f3-5f43-4ffa-8e54-e80f3fdcf293" (UID: "88f672f3-5f43-4ffa-8e54-e80f3fdcf293"). InnerVolumeSpecName "kube-api-access-l8x8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:20:06.706991 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:06.706897    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88f672f3-5f43-4ffa-8e54-e80f3fdcf293-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "88f672f3-5f43-4ffa-8e54-e80f3fdcf293" (UID: "88f672f3-5f43-4ffa-8e54-e80f3fdcf293"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:20:06.738810 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:06.738762    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl" podUID="68e3ddff-b35b-420e-8fb3-0b814406dc4d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 17 22:20:06.762380 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:06.762344    2555 generic.go:356] "Generic (PLEG): container finished" podID="88f672f3-5f43-4ffa-8e54-e80f3fdcf293" containerID="40c44062df8b2f664eb8a75112bab7f6defbe25685c7588896dafc7be29895d0" exitCode=0
Sep 17 22:20:06.762581 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:06.762405    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfkglf" event={"ID":"88f672f3-5f43-4ffa-8e54-e80f3fdcf293","Type":"ContainerDied","Data":"40c44062df8b2f664eb8a75112bab7f6defbe25685c7588896dafc7be29895d0"}
Sep 17 22:20:06.762581 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:06.762465    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfkglf" event={"ID":"88f672f3-5f43-4ffa-8e54-e80f3fdcf293","Type":"ContainerDied","Data":"79a6c3a254025f29fedb9653f6b8cadbd9118fbcbdcd6406d1055ef0a0a5d68c"}
Sep 17 22:20:06.762581 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:06.762476    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfkglf"
Sep 17 22:20:06.762581 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:06.762492    2555 scope.go:117] "RemoveContainer" containerID="40c44062df8b2f664eb8a75112bab7f6defbe25685c7588896dafc7be29895d0"
Sep 17 22:20:06.772041 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:06.772015    2555 scope.go:117] "RemoveContainer" containerID="40c44062df8b2f664eb8a75112bab7f6defbe25685c7588896dafc7be29895d0"
Sep 17 22:20:06.772427 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:20:06.772393    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40c44062df8b2f664eb8a75112bab7f6defbe25685c7588896dafc7be29895d0\": container with ID starting with 40c44062df8b2f664eb8a75112bab7f6defbe25685c7588896dafc7be29895d0 not found: ID does not exist" containerID="40c44062df8b2f664eb8a75112bab7f6defbe25685c7588896dafc7be29895d0"
Sep 17 22:20:06.772530 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:06.772440    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40c44062df8b2f664eb8a75112bab7f6defbe25685c7588896dafc7be29895d0"} err="failed to get container status \"40c44062df8b2f664eb8a75112bab7f6defbe25685c7588896dafc7be29895d0\": rpc error: code = NotFound desc = could not find container \"40c44062df8b2f664eb8a75112bab7f6defbe25685c7588896dafc7be29895d0\": container with ID starting with 40c44062df8b2f664eb8a75112bab7f6defbe25685c7588896dafc7be29895d0 not found: ID does not exist"
Sep 17 22:20:06.790543 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:06.790512    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfkglf"]
Sep 17 22:20:06.794273 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:06.794249    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedfkglf"]
Sep 17 22:20:06.805562 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:06.805525    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88f672f3-5f43-4ffa-8e54-e80f3fdcf293-tls-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:20:06.805562 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:06.805561    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8x8n\" (UniqueName: \"kubernetes.io/projected/88f672f3-5f43-4ffa-8e54-e80f3fdcf293-kube-api-access-l8x8n\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:20:08.680483 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:08.680447    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88f672f3-5f43-4ffa-8e54-e80f3fdcf293" path="/var/lib/kubelet/pods/88f672f3-5f43-4ffa-8e54-e80f3fdcf293/volumes"
Sep 17 22:20:11.493489 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:11.493454    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:20:11.591321 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:11.591286    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:20:16.739455 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:16.739408    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl" podUID="68e3ddff-b35b-420e-8fb3-0b814406dc4d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 17 22:20:17.606455 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:17.606394    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:20:20.734310 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:20.734247    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:20:23.136395 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.136363    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:20:23.238586 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.238550    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:20:23.253469 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.253440    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp"]
Sep 17 22:20:23.253829 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.253815    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b935adf9-1a52-4e12-b842-f9d4bb4eee52" containerName="main"
Sep 17 22:20:23.253875 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.253831    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b935adf9-1a52-4e12-b842-f9d4bb4eee52" containerName="main"
Sep 17 22:20:23.253875 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.253847    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88f672f3-5f43-4ffa-8e54-e80f3fdcf293" containerName="main"
Sep 17 22:20:23.253875 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.253853    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="88f672f3-5f43-4ffa-8e54-e80f3fdcf293" containerName="main"
Sep 17 22:20:23.253973 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.253922    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b935adf9-1a52-4e12-b842-f9d4bb4eee52" containerName="main"
Sep 17 22:20:23.253973 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.253932    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="88f672f3-5f43-4ffa-8e54-e80f3fdcf293" containerName="main"
Sep 17 22:20:23.259830 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.259811    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp"
Sep 17 22:20:23.262135 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.262108    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 17 22:20:23.262135 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.262126    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 17 22:20:23.262327 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.262140    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 17 22:20:23.262327 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.262126    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:20:23.262948 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.262929    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 17 22:20:23.263041 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.262956    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-tn5p7\""
Sep 17 22:20:23.270105 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.270084    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp"]
Sep 17 22:20:23.344305 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.344274    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:20:23.352864 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.352825    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c355d94d-a394-49ae-861a-6cca6185c5b7-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-28wrp\" (UID: \"c355d94d-a394-49ae-861a-6cca6185c5b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp"
Sep 17 22:20:23.353044 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.352871    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g49lm\" (UniqueName: \"kubernetes.io/projected/c355d94d-a394-49ae-861a-6cca6185c5b7-kube-api-access-g49lm\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-28wrp\" (UID: \"c355d94d-a394-49ae-861a-6cca6185c5b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp"
Sep 17 22:20:23.353044 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.352926    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c355d94d-a394-49ae-861a-6cca6185c5b7-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-28wrp\" (UID: \"c355d94d-a394-49ae-861a-6cca6185c5b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp"
Sep 17 22:20:23.353044 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.352974    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c355d94d-a394-49ae-861a-6cca6185c5b7-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-28wrp\" (UID: \"c355d94d-a394-49ae-861a-6cca6185c5b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp"
Sep 17 22:20:23.353044 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.352999    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c355d94d-a394-49ae-861a-6cca6185c5b7-home\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-28wrp\" (UID: \"c355d94d-a394-49ae-861a-6cca6185c5b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp"
Sep 17 22:20:23.353044 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.353041    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c355d94d-a394-49ae-861a-6cca6185c5b7-model-cache\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-28wrp\" (UID: \"c355d94d-a394-49ae-861a-6cca6185c5b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp"
Sep 17 22:20:23.353352 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.353082    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c355d94d-a394-49ae-861a-6cca6185c5b7-dshm\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-28wrp\" (UID: \"c355d94d-a394-49ae-861a-6cca6185c5b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp"
Sep 17 22:20:23.353352 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.353105    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c355d94d-a394-49ae-861a-6cca6185c5b7-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-28wrp\" (UID: \"c355d94d-a394-49ae-861a-6cca6185c5b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp"
Sep 17 22:20:23.454224 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.454108    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c355d94d-a394-49ae-861a-6cca6185c5b7-dshm\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-28wrp\" (UID: \"c355d94d-a394-49ae-861a-6cca6185c5b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp"
Sep 17 22:20:23.454224 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.454150    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c355d94d-a394-49ae-861a-6cca6185c5b7-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-28wrp\" (UID: \"c355d94d-a394-49ae-861a-6cca6185c5b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp"
Sep 17 22:20:23.454455 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.454244    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c355d94d-a394-49ae-861a-6cca6185c5b7-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-28wrp\" (UID: \"c355d94d-a394-49ae-861a-6cca6185c5b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp"
Sep 17 22:20:23.454455 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.454271    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g49lm\" (UniqueName: \"kubernetes.io/projected/c355d94d-a394-49ae-861a-6cca6185c5b7-kube-api-access-g49lm\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-28wrp\" (UID: \"c355d94d-a394-49ae-861a-6cca6185c5b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp"
Sep 17 22:20:23.454455 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.454320    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c355d94d-a394-49ae-861a-6cca6185c5b7-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-28wrp\" (UID: \"c355d94d-a394-49ae-861a-6cca6185c5b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp"
Sep 17 22:20:23.454620 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.454506    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c355d94d-a394-49ae-861a-6cca6185c5b7-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-28wrp\" (UID: \"c355d94d-a394-49ae-861a-6cca6185c5b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp"
Sep 17 22:20:23.454620 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.454556    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c355d94d-a394-49ae-861a-6cca6185c5b7-home\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-28wrp\" (UID: \"c355d94d-a394-49ae-861a-6cca6185c5b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp"
Sep 17 22:20:23.454731 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.454662    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c355d94d-a394-49ae-861a-6cca6185c5b7-model-cache\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-28wrp\" (UID: \"c355d94d-a394-49ae-861a-6cca6185c5b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp"
Sep 17 22:20:23.454731 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.454699    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c355d94d-a394-49ae-861a-6cca6185c5b7-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-28wrp\" (UID: \"c355d94d-a394-49ae-861a-6cca6185c5b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp"
Sep 17 22:20:23.455027 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.455000    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c355d94d-a394-49ae-861a-6cca6185c5b7-home\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-28wrp\" (UID: \"c355d94d-a394-49ae-861a-6cca6185c5b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp"
Sep 17 22:20:23.455133 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.455055    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c355d94d-a394-49ae-861a-6cca6185c5b7-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-28wrp\" (UID: \"c355d94d-a394-49ae-861a-6cca6185c5b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp"
Sep 17 22:20:23.455224 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.455154    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c355d94d-a394-49ae-861a-6cca6185c5b7-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-28wrp\" (UID: \"c355d94d-a394-49ae-861a-6cca6185c5b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp"
Sep 17 22:20:23.455281 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.455222    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c355d94d-a394-49ae-861a-6cca6185c5b7-model-cache\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-28wrp\" (UID: \"c355d94d-a394-49ae-861a-6cca6185c5b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp"
Sep 17 22:20:23.456632 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.456606    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c355d94d-a394-49ae-861a-6cca6185c5b7-dshm\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-28wrp\" (UID: \"c355d94d-a394-49ae-861a-6cca6185c5b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp"
Sep 17 22:20:23.456974 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.456956    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c355d94d-a394-49ae-861a-6cca6185c5b7-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-28wrp\" (UID: \"c355d94d-a394-49ae-861a-6cca6185c5b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp"
Sep 17 22:20:23.462820 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.462796    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g49lm\" (UniqueName: \"kubernetes.io/projected/c355d94d-a394-49ae-861a-6cca6185c5b7-kube-api-access-g49lm\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-28wrp\" (UID: \"c355d94d-a394-49ae-861a-6cca6185c5b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp"
Sep 17 22:20:23.518012 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.517590    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7"]
Sep 17 22:20:23.524677 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.524646    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7"
Sep 17 22:20:23.540467 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.540421    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7"]
Sep 17 22:20:23.557995 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.557957    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:20:23.571957 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.571922    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp"
Sep 17 22:20:23.631561 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.631531    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:20:23.656357 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.656255    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db557a94-0209-4d58-a467-fa7eb2241308-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7\" (UID: \"db557a94-0209-4d58-a467-fa7eb2241308\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7"
Sep 17 22:20:23.656531 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.656368    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db557a94-0209-4d58-a467-fa7eb2241308-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7\" (UID: \"db557a94-0209-4d58-a467-fa7eb2241308\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7"
Sep 17 22:20:23.656531 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.656419    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db557a94-0209-4d58-a467-fa7eb2241308-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7\" (UID: \"db557a94-0209-4d58-a467-fa7eb2241308\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7"
Sep 17 22:20:23.656531 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.656470    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db557a94-0209-4d58-a467-fa7eb2241308-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7\" (UID: \"db557a94-0209-4d58-a467-fa7eb2241308\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7"
Sep 17 22:20:23.757193 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.757082    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db557a94-0209-4d58-a467-fa7eb2241308-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7\" (UID: \"db557a94-0209-4d58-a467-fa7eb2241308\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7"
Sep 17 22:20:23.757193 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.757160    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db557a94-0209-4d58-a467-fa7eb2241308-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7\" (UID: \"db557a94-0209-4d58-a467-fa7eb2241308\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7"
Sep 17 22:20:23.757439 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.757225    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db557a94-0209-4d58-a467-fa7eb2241308-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7\" (UID: \"db557a94-0209-4d58-a467-fa7eb2241308\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7"
Sep 17 22:20:23.757439 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.757286    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db557a94-0209-4d58-a467-fa7eb2241308-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7\" (UID: \"db557a94-0209-4d58-a467-fa7eb2241308\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7"
Sep 17 22:20:23.757630 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.757606    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db557a94-0209-4d58-a467-fa7eb2241308-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7\" (UID: \"db557a94-0209-4d58-a467-fa7eb2241308\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7"
Sep 17 22:20:23.757707 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.757670    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db557a94-0209-4d58-a467-fa7eb2241308-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7\" (UID: \"db557a94-0209-4d58-a467-fa7eb2241308\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7"
Sep 17 22:20:23.757975 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.757954    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db557a94-0209-4d58-a467-fa7eb2241308-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7\" (UID: \"db557a94-0209-4d58-a467-fa7eb2241308\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7"
Sep 17 22:20:23.760026 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.760003    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db557a94-0209-4d58-a467-fa7eb2241308-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7\" (UID: \"db557a94-0209-4d58-a467-fa7eb2241308\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7"
Sep 17 22:20:23.849468 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.849435    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp"]
Sep 17 22:20:23.849770 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.849473    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7"
Sep 17 22:20:23.852096 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:20:23.852062    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc355d94d_a394_49ae_861a_6cca6185c5b7.slice/crio-a83d6145fb11515e76febe5e3af20cd285178fc61d77d51befde338793e8dc5b WatchSource:0}: Error finding container a83d6145fb11515e76febe5e3af20cd285178fc61d77d51befde338793e8dc5b: Status 404 returned error can't find the container with id a83d6145fb11515e76febe5e3af20cd285178fc61d77d51befde338793e8dc5b
Sep 17 22:20:23.864246 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:23.864217    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp" event={"ID":"c355d94d-a394-49ae-861a-6cca6185c5b7","Type":"ContainerStarted","Data":"a83d6145fb11515e76febe5e3af20cd285178fc61d77d51befde338793e8dc5b"}
Sep 17 22:20:24.130831 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:24.130800    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7"]
Sep 17 22:20:24.131980 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:20:24.131949    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb557a94_0209_4d58_a467_fa7eb2241308.slice/crio-ee75bc478e615cf769b5d2a4122db78dfae1ef3ad4b42ea688894f9f29f00ca0 WatchSource:0}: Error finding container ee75bc478e615cf769b5d2a4122db78dfae1ef3ad4b42ea688894f9f29f00ca0: Status 404 returned error can't find the container with id ee75bc478e615cf769b5d2a4122db78dfae1ef3ad4b42ea688894f9f29f00ca0
Sep 17 22:20:24.872949 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:24.872909    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7" event={"ID":"db557a94-0209-4d58-a467-fa7eb2241308","Type":"ContainerStarted","Data":"8c1c93327558e5f3e2e8c032a63d886cf8ad987d279992c498262c402b7adaed"}
Sep 17 22:20:24.873464 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:24.872986    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7" event={"ID":"db557a94-0209-4d58-a467-fa7eb2241308","Type":"ContainerStarted","Data":"ee75bc478e615cf769b5d2a4122db78dfae1ef3ad4b42ea688894f9f29f00ca0"}
Sep 17 22:20:24.874887 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:24.874852    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp" event={"ID":"c355d94d-a394-49ae-861a-6cca6185c5b7","Type":"ContainerStarted","Data":"150af9ca80d4d9abe08fadc62bee9e16fba0506abec38df98da4957ecd25523d"}
Sep 17 22:20:26.738829 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:26.738782    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl" podUID="68e3ddff-b35b-420e-8fb3-0b814406dc4d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 17 22:20:26.891693 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:26.891653    2555 generic.go:356] "Generic (PLEG): container finished" podID="db557a94-0209-4d58-a467-fa7eb2241308" containerID="8c1c93327558e5f3e2e8c032a63d886cf8ad987d279992c498262c402b7adaed" exitCode=0
Sep 17 22:20:26.891902 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:26.891727    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7" event={"ID":"db557a94-0209-4d58-a467-fa7eb2241308","Type":"ContainerDied","Data":"8c1c93327558e5f3e2e8c032a63d886cf8ad987d279992c498262c402b7adaed"}
Sep 17 22:20:26.893678 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:26.893652    2555 generic.go:356] "Generic (PLEG): container finished" podID="c355d94d-a394-49ae-861a-6cca6185c5b7" containerID="150af9ca80d4d9abe08fadc62bee9e16fba0506abec38df98da4957ecd25523d" exitCode=0
Sep 17 22:20:26.893781 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:26.893710    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp" event={"ID":"c355d94d-a394-49ae-861a-6cca6185c5b7","Type":"ContainerDied","Data":"150af9ca80d4d9abe08fadc62bee9e16fba0506abec38df98da4957ecd25523d"}
Sep 17 22:20:27.906213 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:27.905645    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp" event={"ID":"c355d94d-a394-49ae-861a-6cca6185c5b7","Type":"ContainerStarted","Data":"f7b726f2580a395fcbaeb1da5f5cea739196c663dc50ea0569efbd3122359974"}
Sep 17 22:20:27.928277 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:27.928220    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp" podStartSLOduration=2.361906601 podStartE2EDuration="4.928199739s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.566293138s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:20:27.924743483 +0000 UTC m=+1329.370788758" watchObservedRunningTime="2026-09-17 22:20:27.928199739 +0000 UTC m=+1329.374245008"
Sep 17 22:20:28.572496 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:28.572457    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp"
Sep 17 22:20:33.572571 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:33.572481    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp"
Sep 17 22:20:33.600811 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:33.600548    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp"
Sep 17 22:20:33.971132 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:33.971100    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp"
Sep 17 22:20:36.755236 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:36.755168    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl"
Sep 17 22:20:36.767985 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:20:36.767955    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl"
Sep 17 22:21:01.185702 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:01.185665    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:21:16.272927 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:16.272891    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7" event={"ID":"db557a94-0209-4d58-a467-fa7eb2241308","Type":"ContainerStarted","Data":"cd415ebc39b400dfcc9ea13edb88e6c9cbe1090d1aa763061fb7573b55280280"}
Sep 17 22:21:16.273372 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:16.273102    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7"
Sep 17 22:21:16.292597 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:16.292546    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7" podStartSLOduration=2.234365875 podStartE2EDuration="53.292531623s" totalImagesPullingTime="48.46843752s" totalInitContainerRuntime="2.589728228s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:20:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:21:16.291331128 +0000 UTC m=+1377.737376401" watchObservedRunningTime="2026-09-17 22:21:16.292531623 +0000 UTC m=+1377.738576894"
Sep 17 22:21:26.364016 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:26.363968    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:21:26.367422 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:26.367391    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl"]
Sep 17 22:21:26.473733 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:26.473696    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:21:41.394454 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:41.394413    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl" podUID="68e3ddff-b35b-420e-8fb3-0b814406dc4d" containerName="main" containerID="cri-o://f7ca0d6796e9679e570115aaca393c5af85ba0bf0a47fb5024dd6e82ea6b51f1" gracePeriod=45
Sep 17 22:21:41.756496 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:41.756405    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl" podUID="68e3ddff-b35b-420e-8fb3-0b814406dc4d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 17 22:21:42.215127 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:42.215097    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl"
Sep 17 22:21:42.323068 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:42.322982    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68e3ddff-b35b-420e-8fb3-0b814406dc4d-home\") pod \"68e3ddff-b35b-420e-8fb3-0b814406dc4d\" (UID: \"68e3ddff-b35b-420e-8fb3-0b814406dc4d\") "
Sep 17 22:21:42.323068 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:42.323026    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68e3ddff-b35b-420e-8fb3-0b814406dc4d-kserve-provision-location\") pod \"68e3ddff-b35b-420e-8fb3-0b814406dc4d\" (UID: \"68e3ddff-b35b-420e-8fb3-0b814406dc4d\") "
Sep 17 22:21:42.323316 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:42.323088    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68e3ddff-b35b-420e-8fb3-0b814406dc4d-model-cache\") pod \"68e3ddff-b35b-420e-8fb3-0b814406dc4d\" (UID: \"68e3ddff-b35b-420e-8fb3-0b814406dc4d\") "
Sep 17 22:21:42.323316 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:42.323112    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68e3ddff-b35b-420e-8fb3-0b814406dc4d-dshm\") pod \"68e3ddff-b35b-420e-8fb3-0b814406dc4d\" (UID: \"68e3ddff-b35b-420e-8fb3-0b814406dc4d\") "
Sep 17 22:21:42.323316 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:42.323136    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68e3ddff-b35b-420e-8fb3-0b814406dc4d-tmp-dir\") pod \"68e3ddff-b35b-420e-8fb3-0b814406dc4d\" (UID: \"68e3ddff-b35b-420e-8fb3-0b814406dc4d\") "
Sep 17 22:21:42.323316 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:42.323163    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68e3ddff-b35b-420e-8fb3-0b814406dc4d-tls-certs\") pod \"68e3ddff-b35b-420e-8fb3-0b814406dc4d\" (UID: \"68e3ddff-b35b-420e-8fb3-0b814406dc4d\") "
Sep 17 22:21:42.323316 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:42.323216    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68e3ddff-b35b-420e-8fb3-0b814406dc4d-cabundle-cert\") pod \"68e3ddff-b35b-420e-8fb3-0b814406dc4d\" (UID: \"68e3ddff-b35b-420e-8fb3-0b814406dc4d\") "
Sep 17 22:21:42.323316 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:42.323252    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkmgb\" (UniqueName: \"kubernetes.io/projected/68e3ddff-b35b-420e-8fb3-0b814406dc4d-kube-api-access-bkmgb\") pod \"68e3ddff-b35b-420e-8fb3-0b814406dc4d\" (UID: \"68e3ddff-b35b-420e-8fb3-0b814406dc4d\") "
Sep 17 22:21:42.323585 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:42.323423    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68e3ddff-b35b-420e-8fb3-0b814406dc4d-model-cache" (OuterVolumeSpecName: "model-cache") pod "68e3ddff-b35b-420e-8fb3-0b814406dc4d" (UID: "68e3ddff-b35b-420e-8fb3-0b814406dc4d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:21:42.323649 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:42.323636    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68e3ddff-b35b-420e-8fb3-0b814406dc4d-model-cache\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:21:42.323815 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:42.323782    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68e3ddff-b35b-420e-8fb3-0b814406dc4d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "68e3ddff-b35b-420e-8fb3-0b814406dc4d" (UID: "68e3ddff-b35b-420e-8fb3-0b814406dc4d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:21:42.323815 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:42.323808    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68e3ddff-b35b-420e-8fb3-0b814406dc4d-home" (OuterVolumeSpecName: "home") pod "68e3ddff-b35b-420e-8fb3-0b814406dc4d" (UID: "68e3ddff-b35b-420e-8fb3-0b814406dc4d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:21:42.329608 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:42.329585    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68e3ddff-b35b-420e-8fb3-0b814406dc4d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "68e3ddff-b35b-420e-8fb3-0b814406dc4d" (UID: "68e3ddff-b35b-420e-8fb3-0b814406dc4d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:21:42.329984 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:42.329965    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68e3ddff-b35b-420e-8fb3-0b814406dc4d-dshm" (OuterVolumeSpecName: "dshm") pod "68e3ddff-b35b-420e-8fb3-0b814406dc4d" (UID: "68e3ddff-b35b-420e-8fb3-0b814406dc4d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:21:42.330057 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:42.329983    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68e3ddff-b35b-420e-8fb3-0b814406dc4d-kube-api-access-bkmgb" (OuterVolumeSpecName: "kube-api-access-bkmgb") pod "68e3ddff-b35b-420e-8fb3-0b814406dc4d" (UID: "68e3ddff-b35b-420e-8fb3-0b814406dc4d"). InnerVolumeSpecName "kube-api-access-bkmgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:21:42.334601 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:42.334571    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68e3ddff-b35b-420e-8fb3-0b814406dc4d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "68e3ddff-b35b-420e-8fb3-0b814406dc4d" (UID: "68e3ddff-b35b-420e-8fb3-0b814406dc4d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:21:42.335671 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:42.335643    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68e3ddff-b35b-420e-8fb3-0b814406dc4d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "68e3ddff-b35b-420e-8fb3-0b814406dc4d" (UID: "68e3ddff-b35b-420e-8fb3-0b814406dc4d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:21:42.424913 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:42.424875    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68e3ddff-b35b-420e-8fb3-0b814406dc4d-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:21:42.424913 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:42.424902    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkmgb\" (UniqueName: \"kubernetes.io/projected/68e3ddff-b35b-420e-8fb3-0b814406dc4d-kube-api-access-bkmgb\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:21:42.424913 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:42.424913    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68e3ddff-b35b-420e-8fb3-0b814406dc4d-home\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:21:42.424913 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:42.424923    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68e3ddff-b35b-420e-8fb3-0b814406dc4d-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:21:42.425579 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:42.424931    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68e3ddff-b35b-420e-8fb3-0b814406dc4d-dshm\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:21:42.425579 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:42.424939    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68e3ddff-b35b-420e-8fb3-0b814406dc4d-tmp-dir\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:21:42.425579 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:42.424946    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68e3ddff-b35b-420e-8fb3-0b814406dc4d-tls-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:21:42.430294 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:42.430265    2555 generic.go:356] "Generic (PLEG): container finished" podID="68e3ddff-b35b-420e-8fb3-0b814406dc4d" containerID="f7ca0d6796e9679e570115aaca393c5af85ba0bf0a47fb5024dd6e82ea6b51f1" exitCode=0
Sep 17 22:21:42.430433 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:42.430330    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl"
Sep 17 22:21:42.430433 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:42.430350    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl" event={"ID":"68e3ddff-b35b-420e-8fb3-0b814406dc4d","Type":"ContainerDied","Data":"f7ca0d6796e9679e570115aaca393c5af85ba0bf0a47fb5024dd6e82ea6b51f1"}
Sep 17 22:21:42.430523 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:42.430440    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl" event={"ID":"68e3ddff-b35b-420e-8fb3-0b814406dc4d","Type":"ContainerDied","Data":"a51005d9fb3a06934b79519b37aa39cfffceb3b2f6e386006d51fd26b86c9255"}
Sep 17 22:21:42.430523 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:42.430471    2555 scope.go:117] "RemoveContainer" containerID="f7ca0d6796e9679e570115aaca393c5af85ba0bf0a47fb5024dd6e82ea6b51f1"
Sep 17 22:21:42.440443 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:42.440428    2555 scope.go:117] "RemoveContainer" containerID="21cd399546a935c8af51eb9967502a32195fd5da882557cb442e8dd86ed9bced"
Sep 17 22:21:42.448999 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:42.448975    2555 scope.go:117] "RemoveContainer" containerID="f7ca0d6796e9679e570115aaca393c5af85ba0bf0a47fb5024dd6e82ea6b51f1"
Sep 17 22:21:42.449285 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:21:42.449265    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7ca0d6796e9679e570115aaca393c5af85ba0bf0a47fb5024dd6e82ea6b51f1\": container with ID starting with f7ca0d6796e9679e570115aaca393c5af85ba0bf0a47fb5024dd6e82ea6b51f1 not found: ID does not exist" containerID="f7ca0d6796e9679e570115aaca393c5af85ba0bf0a47fb5024dd6e82ea6b51f1"
Sep 17 22:21:42.449357 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:42.449293    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7ca0d6796e9679e570115aaca393c5af85ba0bf0a47fb5024dd6e82ea6b51f1"} err="failed to get container status \"f7ca0d6796e9679e570115aaca393c5af85ba0bf0a47fb5024dd6e82ea6b51f1\": rpc error: code = NotFound desc = could not find container \"f7ca0d6796e9679e570115aaca393c5af85ba0bf0a47fb5024dd6e82ea6b51f1\": container with ID starting with f7ca0d6796e9679e570115aaca393c5af85ba0bf0a47fb5024dd6e82ea6b51f1 not found: ID does not exist"
Sep 17 22:21:42.449357 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:42.449308    2555 scope.go:117] "RemoveContainer" containerID="21cd399546a935c8af51eb9967502a32195fd5da882557cb442e8dd86ed9bced"
Sep 17 22:21:42.449532 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:21:42.449515    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21cd399546a935c8af51eb9967502a32195fd5da882557cb442e8dd86ed9bced\": container with ID starting with 21cd399546a935c8af51eb9967502a32195fd5da882557cb442e8dd86ed9bced not found: ID does not exist" containerID="21cd399546a935c8af51eb9967502a32195fd5da882557cb442e8dd86ed9bced"
Sep 17 22:21:42.449569 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:42.449536    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21cd399546a935c8af51eb9967502a32195fd5da882557cb442e8dd86ed9bced"} err="failed to get container status \"21cd399546a935c8af51eb9967502a32195fd5da882557cb442e8dd86ed9bced\": rpc error: code = NotFound desc = could not find container \"21cd399546a935c8af51eb9967502a32195fd5da882557cb442e8dd86ed9bced\": container with ID starting with 21cd399546a935c8af51eb9967502a32195fd5da882557cb442e8dd86ed9bced not found: ID does not exist"
Sep 17 22:21:42.454562 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:42.454542    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl"]
Sep 17 22:21:42.460778 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:42.460756    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5c48865899-cpmsl"]
Sep 17 22:21:42.679072 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:42.679041    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68e3ddff-b35b-420e-8fb3-0b814406dc4d" path="/var/lib/kubelet/pods/68e3ddff-b35b-420e-8fb3-0b814406dc4d/volumes"
Sep 17 22:21:47.289848 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:47.289814    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7"
Sep 17 22:21:48.271645 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:48.271604    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7_db557a94-0209-4d58-a467-fa7eb2241308/main/0.log"
Sep 17 22:21:49.068597 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.068565    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp"]
Sep 17 22:21:49.073475 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.073451    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7"]
Sep 17 22:21:49.074047 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.074016    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7" podUID="db557a94-0209-4d58-a467-fa7eb2241308" containerName="main" containerID="cri-o://cd415ebc39b400dfcc9ea13edb88e6c9cbe1090d1aa763061fb7573b55280280" gracePeriod=30
Sep 17 22:21:49.077353 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.077279    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:21:49.081821 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:21:49.081785    2555 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 22:21:49.081821 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:21:49.081821 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:21:49.081821 ip-10-0-140-157 kubenswrapper[2555]:  > containerID="f7b726f2580a395fcbaeb1da5f5cea739196c663dc50ea0569efbd3122359974" cmd=["/bin/sleep","15"]
Sep 17 22:21:49.082058 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.081834    2555 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp" message="" err=<
Sep 17 22:21:49.082058 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:21:49.082058 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:21:49.082058 ip-10-0-140-157 kubenswrapper[2555]:  >
Sep 17 22:21:49.082058 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:21:49.081847    2555 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 22:21:49.082058 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:21:49.082058 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:21:49.082058 ip-10-0-140-157 kubenswrapper[2555]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp" podUID="c355d94d-a394-49ae-861a-6cca6185c5b7" containerName="main" containerID="cri-o://f7b726f2580a395fcbaeb1da5f5cea739196c663dc50ea0569efbd3122359974"
Sep 17 22:21:49.082058 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.081893    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp" podUID="c355d94d-a394-49ae-861a-6cca6185c5b7" containerName="main" containerID="cri-o://f7b726f2580a395fcbaeb1da5f5cea739196c663dc50ea0569efbd3122359974" gracePeriod=60
Sep 17 22:21:49.137688 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.137660    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:21:49.457209 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.457154    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp"
Sep 17 22:21:49.475165 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.475132    2555 generic.go:356] "Generic (PLEG): container finished" podID="c355d94d-a394-49ae-861a-6cca6185c5b7" containerID="f7b726f2580a395fcbaeb1da5f5cea739196c663dc50ea0569efbd3122359974" exitCode=0
Sep 17 22:21:49.475458 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.475219    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp"
Sep 17 22:21:49.475928 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.475240    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp" event={"ID":"c355d94d-a394-49ae-861a-6cca6185c5b7","Type":"ContainerDied","Data":"f7b726f2580a395fcbaeb1da5f5cea739196c663dc50ea0569efbd3122359974"}
Sep 17 22:21:49.475928 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.475891    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp" event={"ID":"c355d94d-a394-49ae-861a-6cca6185c5b7","Type":"ContainerDied","Data":"a83d6145fb11515e76febe5e3af20cd285178fc61d77d51befde338793e8dc5b"}
Sep 17 22:21:49.475928 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.475929    2555 scope.go:117] "RemoveContainer" containerID="f7b726f2580a395fcbaeb1da5f5cea739196c663dc50ea0569efbd3122359974"
Sep 17 22:21:49.487901 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.487880    2555 scope.go:117] "RemoveContainer" containerID="150af9ca80d4d9abe08fadc62bee9e16fba0506abec38df98da4957ecd25523d"
Sep 17 22:21:49.497483 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.497455    2555 scope.go:117] "RemoveContainer" containerID="f7b726f2580a395fcbaeb1da5f5cea739196c663dc50ea0569efbd3122359974"
Sep 17 22:21:49.497810 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:21:49.497790    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7b726f2580a395fcbaeb1da5f5cea739196c663dc50ea0569efbd3122359974\": container with ID starting with f7b726f2580a395fcbaeb1da5f5cea739196c663dc50ea0569efbd3122359974 not found: ID does not exist" containerID="f7b726f2580a395fcbaeb1da5f5cea739196c663dc50ea0569efbd3122359974"
Sep 17 22:21:49.497907 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.497824    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7b726f2580a395fcbaeb1da5f5cea739196c663dc50ea0569efbd3122359974"} err="failed to get container status \"f7b726f2580a395fcbaeb1da5f5cea739196c663dc50ea0569efbd3122359974\": rpc error: code = NotFound desc = could not find container \"f7b726f2580a395fcbaeb1da5f5cea739196c663dc50ea0569efbd3122359974\": container with ID starting with f7b726f2580a395fcbaeb1da5f5cea739196c663dc50ea0569efbd3122359974 not found: ID does not exist"
Sep 17 22:21:49.497907 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.497850    2555 scope.go:117] "RemoveContainer" containerID="150af9ca80d4d9abe08fadc62bee9e16fba0506abec38df98da4957ecd25523d"
Sep 17 22:21:49.498159 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:21:49.498143    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"150af9ca80d4d9abe08fadc62bee9e16fba0506abec38df98da4957ecd25523d\": container with ID starting with 150af9ca80d4d9abe08fadc62bee9e16fba0506abec38df98da4957ecd25523d not found: ID does not exist" containerID="150af9ca80d4d9abe08fadc62bee9e16fba0506abec38df98da4957ecd25523d"
Sep 17 22:21:49.498257 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.498166    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"150af9ca80d4d9abe08fadc62bee9e16fba0506abec38df98da4957ecd25523d"} err="failed to get container status \"150af9ca80d4d9abe08fadc62bee9e16fba0506abec38df98da4957ecd25523d\": rpc error: code = NotFound desc = could not find container \"150af9ca80d4d9abe08fadc62bee9e16fba0506abec38df98da4957ecd25523d\": container with ID starting with 150af9ca80d4d9abe08fadc62bee9e16fba0506abec38df98da4957ecd25523d not found: ID does not exist"
Sep 17 22:21:49.600615 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.600588    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c355d94d-a394-49ae-861a-6cca6185c5b7-home\") pod \"c355d94d-a394-49ae-861a-6cca6185c5b7\" (UID: \"c355d94d-a394-49ae-861a-6cca6185c5b7\") "
Sep 17 22:21:49.600748 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.600649    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c355d94d-a394-49ae-861a-6cca6185c5b7-kserve-provision-location\") pod \"c355d94d-a394-49ae-861a-6cca6185c5b7\" (UID: \"c355d94d-a394-49ae-861a-6cca6185c5b7\") "
Sep 17 22:21:49.600748 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.600671    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c355d94d-a394-49ae-861a-6cca6185c5b7-dshm\") pod \"c355d94d-a394-49ae-861a-6cca6185c5b7\" (UID: \"c355d94d-a394-49ae-861a-6cca6185c5b7\") "
Sep 17 22:21:49.600748 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.600733    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c355d94d-a394-49ae-861a-6cca6185c5b7-tmp-dir\") pod \"c355d94d-a394-49ae-861a-6cca6185c5b7\" (UID: \"c355d94d-a394-49ae-861a-6cca6185c5b7\") "
Sep 17 22:21:49.600984 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.600766    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g49lm\" (UniqueName: \"kubernetes.io/projected/c355d94d-a394-49ae-861a-6cca6185c5b7-kube-api-access-g49lm\") pod \"c355d94d-a394-49ae-861a-6cca6185c5b7\" (UID: \"c355d94d-a394-49ae-861a-6cca6185c5b7\") "
Sep 17 22:21:49.600984 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.600817    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c355d94d-a394-49ae-861a-6cca6185c5b7-model-cache\") pod \"c355d94d-a394-49ae-861a-6cca6185c5b7\" (UID: \"c355d94d-a394-49ae-861a-6cca6185c5b7\") "
Sep 17 22:21:49.600984 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.600825    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c355d94d-a394-49ae-861a-6cca6185c5b7-home" (OuterVolumeSpecName: "home") pod "c355d94d-a394-49ae-861a-6cca6185c5b7" (UID: "c355d94d-a394-49ae-861a-6cca6185c5b7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:21:49.600984 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.600867    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c355d94d-a394-49ae-861a-6cca6185c5b7-tls-certs\") pod \"c355d94d-a394-49ae-861a-6cca6185c5b7\" (UID: \"c355d94d-a394-49ae-861a-6cca6185c5b7\") "
Sep 17 22:21:49.600984 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.600910    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c355d94d-a394-49ae-861a-6cca6185c5b7-cabundle-cert\") pod \"c355d94d-a394-49ae-861a-6cca6185c5b7\" (UID: \"c355d94d-a394-49ae-861a-6cca6185c5b7\") "
Sep 17 22:21:49.601278 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.601008    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c355d94d-a394-49ae-861a-6cca6185c5b7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c355d94d-a394-49ae-861a-6cca6185c5b7" (UID: "c355d94d-a394-49ae-861a-6cca6185c5b7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:21:49.601278 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.601073    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c355d94d-a394-49ae-861a-6cca6185c5b7-model-cache" (OuterVolumeSpecName: "model-cache") pod "c355d94d-a394-49ae-861a-6cca6185c5b7" (UID: "c355d94d-a394-49ae-861a-6cca6185c5b7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:21:49.601278 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.601225    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c355d94d-a394-49ae-861a-6cca6185c5b7-tmp-dir\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:21:49.601278 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.601245    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c355d94d-a394-49ae-861a-6cca6185c5b7-model-cache\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:21:49.601278 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.601258    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c355d94d-a394-49ae-861a-6cca6185c5b7-home\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:21:49.601548 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.601403    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c355d94d-a394-49ae-861a-6cca6185c5b7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c355d94d-a394-49ae-861a-6cca6185c5b7" (UID: "c355d94d-a394-49ae-861a-6cca6185c5b7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:21:49.603066 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.603015    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c355d94d-a394-49ae-861a-6cca6185c5b7-dshm" (OuterVolumeSpecName: "dshm") pod "c355d94d-a394-49ae-861a-6cca6185c5b7" (UID: "c355d94d-a394-49ae-861a-6cca6185c5b7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:21:49.603201 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.603165    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c355d94d-a394-49ae-861a-6cca6185c5b7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c355d94d-a394-49ae-861a-6cca6185c5b7" (UID: "c355d94d-a394-49ae-861a-6cca6185c5b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:21:49.603458 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.603438    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c355d94d-a394-49ae-861a-6cca6185c5b7-kube-api-access-g49lm" (OuterVolumeSpecName: "kube-api-access-g49lm") pod "c355d94d-a394-49ae-861a-6cca6185c5b7" (UID: "c355d94d-a394-49ae-861a-6cca6185c5b7"). InnerVolumeSpecName "kube-api-access-g49lm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:21:49.606569 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.606543    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c355d94d-a394-49ae-861a-6cca6185c5b7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c355d94d-a394-49ae-861a-6cca6185c5b7" (UID: "c355d94d-a394-49ae-861a-6cca6185c5b7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:21:49.702136 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.702107    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c355d94d-a394-49ae-861a-6cca6185c5b7-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:21:49.702136 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.702135    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c355d94d-a394-49ae-861a-6cca6185c5b7-dshm\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:21:49.702374 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.702148    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g49lm\" (UniqueName: \"kubernetes.io/projected/c355d94d-a394-49ae-861a-6cca6185c5b7-kube-api-access-g49lm\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:21:49.702374 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.702161    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c355d94d-a394-49ae-861a-6cca6185c5b7-tls-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:21:49.702374 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.702172    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c355d94d-a394-49ae-861a-6cca6185c5b7-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:21:49.727527 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.727488    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7"
Sep 17 22:21:49.797979 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.797954    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp"]
Sep 17 22:21:49.801416 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.801390    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-28wrp"]
Sep 17 22:21:49.802914 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.802889    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db557a94-0209-4d58-a467-fa7eb2241308-tmp-dir\") pod \"db557a94-0209-4d58-a467-fa7eb2241308\" (UID: \"db557a94-0209-4d58-a467-fa7eb2241308\") "
Sep 17 22:21:49.803032 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.802945    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db557a94-0209-4d58-a467-fa7eb2241308-cabundle-cert\") pod \"db557a94-0209-4d58-a467-fa7eb2241308\" (UID: \"db557a94-0209-4d58-a467-fa7eb2241308\") "
Sep 17 22:21:49.803032 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.802985    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db557a94-0209-4d58-a467-fa7eb2241308-tls-certs\") pod \"db557a94-0209-4d58-a467-fa7eb2241308\" (UID: \"db557a94-0209-4d58-a467-fa7eb2241308\") "
Sep 17 22:21:49.803032 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.803009    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db557a94-0209-4d58-a467-fa7eb2241308-kserve-provision-location\") pod \"db557a94-0209-4d58-a467-fa7eb2241308\" (UID: \"db557a94-0209-4d58-a467-fa7eb2241308\") "
Sep 17 22:21:49.803351 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.803328    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db557a94-0209-4d58-a467-fa7eb2241308-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "db557a94-0209-4d58-a467-fa7eb2241308" (UID: "db557a94-0209-4d58-a467-fa7eb2241308"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:21:49.803477 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.803392    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db557a94-0209-4d58-a467-fa7eb2241308-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "db557a94-0209-4d58-a467-fa7eb2241308" (UID: "db557a94-0209-4d58-a467-fa7eb2241308"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:21:49.805148 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.805130    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db557a94-0209-4d58-a467-fa7eb2241308-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "db557a94-0209-4d58-a467-fa7eb2241308" (UID: "db557a94-0209-4d58-a467-fa7eb2241308"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:21:49.809341 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.809319    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db557a94-0209-4d58-a467-fa7eb2241308-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "db557a94-0209-4d58-a467-fa7eb2241308" (UID: "db557a94-0209-4d58-a467-fa7eb2241308"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:21:49.904151 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.904118    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db557a94-0209-4d58-a467-fa7eb2241308-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:21:49.904151 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.904145    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db557a94-0209-4d58-a467-fa7eb2241308-tmp-dir\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:21:49.904151 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.904155    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db557a94-0209-4d58-a467-fa7eb2241308-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:21:49.904151 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:49.904165    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db557a94-0209-4d58-a467-fa7eb2241308-tls-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:21:50.482645 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:50.482553    2555 generic.go:356] "Generic (PLEG): container finished" podID="db557a94-0209-4d58-a467-fa7eb2241308" containerID="cd415ebc39b400dfcc9ea13edb88e6c9cbe1090d1aa763061fb7573b55280280" exitCode=0
Sep 17 22:21:50.482645 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:50.482605    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7"
Sep 17 22:21:50.483142 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:50.482602    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7" event={"ID":"db557a94-0209-4d58-a467-fa7eb2241308","Type":"ContainerDied","Data":"cd415ebc39b400dfcc9ea13edb88e6c9cbe1090d1aa763061fb7573b55280280"}
Sep 17 22:21:50.483142 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:50.482727    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7" event={"ID":"db557a94-0209-4d58-a467-fa7eb2241308","Type":"ContainerDied","Data":"ee75bc478e615cf769b5d2a4122db78dfae1ef3ad4b42ea688894f9f29f00ca0"}
Sep 17 22:21:50.483142 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:50.482781    2555 scope.go:117] "RemoveContainer" containerID="cd415ebc39b400dfcc9ea13edb88e6c9cbe1090d1aa763061fb7573b55280280"
Sep 17 22:21:50.492461 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:50.492444    2555 scope.go:117] "RemoveContainer" containerID="8c1c93327558e5f3e2e8c032a63d886cf8ad987d279992c498262c402b7adaed"
Sep 17 22:21:50.500606 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:50.500590    2555 scope.go:117] "RemoveContainer" containerID="cd415ebc39b400dfcc9ea13edb88e6c9cbe1090d1aa763061fb7573b55280280"
Sep 17 22:21:50.500882 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:21:50.500863    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd415ebc39b400dfcc9ea13edb88e6c9cbe1090d1aa763061fb7573b55280280\": container with ID starting with cd415ebc39b400dfcc9ea13edb88e6c9cbe1090d1aa763061fb7573b55280280 not found: ID does not exist" containerID="cd415ebc39b400dfcc9ea13edb88e6c9cbe1090d1aa763061fb7573b55280280"
Sep 17 22:21:50.500921 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:50.500890    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd415ebc39b400dfcc9ea13edb88e6c9cbe1090d1aa763061fb7573b55280280"} err="failed to get container status \"cd415ebc39b400dfcc9ea13edb88e6c9cbe1090d1aa763061fb7573b55280280\": rpc error: code = NotFound desc = could not find container \"cd415ebc39b400dfcc9ea13edb88e6c9cbe1090d1aa763061fb7573b55280280\": container with ID starting with cd415ebc39b400dfcc9ea13edb88e6c9cbe1090d1aa763061fb7573b55280280 not found: ID does not exist"
Sep 17 22:21:50.500921 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:50.500906    2555 scope.go:117] "RemoveContainer" containerID="8c1c93327558e5f3e2e8c032a63d886cf8ad987d279992c498262c402b7adaed"
Sep 17 22:21:50.501127 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:21:50.501109    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c1c93327558e5f3e2e8c032a63d886cf8ad987d279992c498262c402b7adaed\": container with ID starting with 8c1c93327558e5f3e2e8c032a63d886cf8ad987d279992c498262c402b7adaed not found: ID does not exist" containerID="8c1c93327558e5f3e2e8c032a63d886cf8ad987d279992c498262c402b7adaed"
Sep 17 22:21:50.501167 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:50.501132    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c1c93327558e5f3e2e8c032a63d886cf8ad987d279992c498262c402b7adaed"} err="failed to get container status \"8c1c93327558e5f3e2e8c032a63d886cf8ad987d279992c498262c402b7adaed\": rpc error: code = NotFound desc = could not find container \"8c1c93327558e5f3e2e8c032a63d886cf8ad987d279992c498262c402b7adaed\": container with ID starting with 8c1c93327558e5f3e2e8c032a63d886cf8ad987d279992c498262c402b7adaed not found: ID does not exist"
Sep 17 22:21:50.504867 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:50.504843    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7"]
Sep 17 22:21:50.507071 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:50.507039    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54c6f4f758-2wrt7"]
Sep 17 22:21:50.679609 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:50.679573    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c355d94d-a394-49ae-861a-6cca6185c5b7" path="/var/lib/kubelet/pods/c355d94d-a394-49ae-861a-6cca6185c5b7/volumes"
Sep 17 22:21:50.680008 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:21:50.679995    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db557a94-0209-4d58-a467-fa7eb2241308" path="/var/lib/kubelet/pods/db557a94-0209-4d58-a467-fa7eb2241308/volumes"
Sep 17 22:22:15.000998 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:15.000916    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:22:15.099859 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:15.099828    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:22:21.139192 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:21.139157    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:22:26.730546 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.730515    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd"]
Sep 17 22:22:26.730921 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.730892    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68e3ddff-b35b-420e-8fb3-0b814406dc4d" containerName="main"
Sep 17 22:22:26.730921 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.730907    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="68e3ddff-b35b-420e-8fb3-0b814406dc4d" containerName="main"
Sep 17 22:22:26.730994 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.730926    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c355d94d-a394-49ae-861a-6cca6185c5b7" containerName="storage-initializer"
Sep 17 22:22:26.730994 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.730932    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c355d94d-a394-49ae-861a-6cca6185c5b7" containerName="storage-initializer"
Sep 17 22:22:26.730994 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.730941    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68e3ddff-b35b-420e-8fb3-0b814406dc4d" containerName="storage-initializer"
Sep 17 22:22:26.730994 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.730948    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="68e3ddff-b35b-420e-8fb3-0b814406dc4d" containerName="storage-initializer"
Sep 17 22:22:26.730994 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.730954    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db557a94-0209-4d58-a467-fa7eb2241308" containerName="storage-initializer"
Sep 17 22:22:26.730994 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.730960    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="db557a94-0209-4d58-a467-fa7eb2241308" containerName="storage-initializer"
Sep 17 22:22:26.730994 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.730965    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c355d94d-a394-49ae-861a-6cca6185c5b7" containerName="main"
Sep 17 22:22:26.730994 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.730970    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c355d94d-a394-49ae-861a-6cca6185c5b7" containerName="main"
Sep 17 22:22:26.730994 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.730984    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db557a94-0209-4d58-a467-fa7eb2241308" containerName="main"
Sep 17 22:22:26.730994 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.730989    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="db557a94-0209-4d58-a467-fa7eb2241308" containerName="main"
Sep 17 22:22:26.731324 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.731049    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="68e3ddff-b35b-420e-8fb3-0b814406dc4d" containerName="main"
Sep 17 22:22:26.731324 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.731057    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c355d94d-a394-49ae-861a-6cca6185c5b7" containerName="main"
Sep 17 22:22:26.731324 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.731063    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="db557a94-0209-4d58-a467-fa7eb2241308" containerName="main"
Sep 17 22:22:26.735801 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.735780    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd"
Sep 17 22:22:26.738120 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.738098    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 17 22:22:26.738239 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.738120    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 17 22:22:26.738239 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.738226    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 17 22:22:26.738368 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.738227    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-wj8t9\""
Sep 17 22:22:26.738368 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.738228    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:22:26.738368 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.738227    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Sep 17 22:22:26.744917 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.744896    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd"]
Sep 17 22:22:26.750055 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.750029    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:22:26.817137 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.817111    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e99380b-b274-458a-b655-6df2a18a8e49-home\") pod \"auth-invalid-token-test-kserve-585fcb4787-d6kbd\" (UID: \"0e99380b-b274-458a-b655-6df2a18a8e49\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd"
Sep 17 22:22:26.817137 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.817138    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e99380b-b274-458a-b655-6df2a18a8e49-dshm\") pod \"auth-invalid-token-test-kserve-585fcb4787-d6kbd\" (UID: \"0e99380b-b274-458a-b655-6df2a18a8e49\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd"
Sep 17 22:22:26.817336 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.817172    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e99380b-b274-458a-b655-6df2a18a8e49-model-cache\") pod \"auth-invalid-token-test-kserve-585fcb4787-d6kbd\" (UID: \"0e99380b-b274-458a-b655-6df2a18a8e49\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd"
Sep 17 22:22:26.817336 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.817284    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e99380b-b274-458a-b655-6df2a18a8e49-tmp-dir\") pod \"auth-invalid-token-test-kserve-585fcb4787-d6kbd\" (UID: \"0e99380b-b274-458a-b655-6df2a18a8e49\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd"
Sep 17 22:22:26.817336 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.817322    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e99380b-b274-458a-b655-6df2a18a8e49-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-585fcb4787-d6kbd\" (UID: \"0e99380b-b274-458a-b655-6df2a18a8e49\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd"
Sep 17 22:22:26.817433 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.817369    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e99380b-b274-458a-b655-6df2a18a8e49-cabundle-cert\") pod \"auth-invalid-token-test-kserve-585fcb4787-d6kbd\" (UID: \"0e99380b-b274-458a-b655-6df2a18a8e49\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd"
Sep 17 22:22:26.817433 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.817399    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e99380b-b274-458a-b655-6df2a18a8e49-tls-certs\") pod \"auth-invalid-token-test-kserve-585fcb4787-d6kbd\" (UID: \"0e99380b-b274-458a-b655-6df2a18a8e49\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd"
Sep 17 22:22:26.817433 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.817413    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp87l\" (UniqueName: \"kubernetes.io/projected/0e99380b-b274-458a-b655-6df2a18a8e49-kube-api-access-mp87l\") pod \"auth-invalid-token-test-kserve-585fcb4787-d6kbd\" (UID: \"0e99380b-b274-458a-b655-6df2a18a8e49\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd"
Sep 17 22:22:26.912092 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.912060    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:22:26.918127 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.918103    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e99380b-b274-458a-b655-6df2a18a8e49-home\") pod \"auth-invalid-token-test-kserve-585fcb4787-d6kbd\" (UID: \"0e99380b-b274-458a-b655-6df2a18a8e49\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd"
Sep 17 22:22:26.918247 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.918133    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e99380b-b274-458a-b655-6df2a18a8e49-dshm\") pod \"auth-invalid-token-test-kserve-585fcb4787-d6kbd\" (UID: \"0e99380b-b274-458a-b655-6df2a18a8e49\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd"
Sep 17 22:22:26.918247 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.918178    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e99380b-b274-458a-b655-6df2a18a8e49-model-cache\") pod \"auth-invalid-token-test-kserve-585fcb4787-d6kbd\" (UID: \"0e99380b-b274-458a-b655-6df2a18a8e49\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd"
Sep 17 22:22:26.918247 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.918226    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e99380b-b274-458a-b655-6df2a18a8e49-tmp-dir\") pod \"auth-invalid-token-test-kserve-585fcb4787-d6kbd\" (UID: \"0e99380b-b274-458a-b655-6df2a18a8e49\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd"
Sep 17 22:22:26.918423 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.918261    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e99380b-b274-458a-b655-6df2a18a8e49-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-585fcb4787-d6kbd\" (UID: \"0e99380b-b274-458a-b655-6df2a18a8e49\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd"
Sep 17 22:22:26.918423 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.918303    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e99380b-b274-458a-b655-6df2a18a8e49-cabundle-cert\") pod \"auth-invalid-token-test-kserve-585fcb4787-d6kbd\" (UID: \"0e99380b-b274-458a-b655-6df2a18a8e49\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd"
Sep 17 22:22:26.918423 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.918333    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e99380b-b274-458a-b655-6df2a18a8e49-tls-certs\") pod \"auth-invalid-token-test-kserve-585fcb4787-d6kbd\" (UID: \"0e99380b-b274-458a-b655-6df2a18a8e49\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd"
Sep 17 22:22:26.918423 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.918358    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp87l\" (UniqueName: \"kubernetes.io/projected/0e99380b-b274-458a-b655-6df2a18a8e49-kube-api-access-mp87l\") pod \"auth-invalid-token-test-kserve-585fcb4787-d6kbd\" (UID: \"0e99380b-b274-458a-b655-6df2a18a8e49\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd"
Sep 17 22:22:26.918645 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.918569    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e99380b-b274-458a-b655-6df2a18a8e49-home\") pod \"auth-invalid-token-test-kserve-585fcb4787-d6kbd\" (UID: \"0e99380b-b274-458a-b655-6df2a18a8e49\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd"
Sep 17 22:22:26.918720 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.918688    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e99380b-b274-458a-b655-6df2a18a8e49-tmp-dir\") pod \"auth-invalid-token-test-kserve-585fcb4787-d6kbd\" (UID: \"0e99380b-b274-458a-b655-6df2a18a8e49\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd"
Sep 17 22:22:26.918941 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.918920    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e99380b-b274-458a-b655-6df2a18a8e49-model-cache\") pod \"auth-invalid-token-test-kserve-585fcb4787-d6kbd\" (UID: \"0e99380b-b274-458a-b655-6df2a18a8e49\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd"
Sep 17 22:22:26.919037 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.918965    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e99380b-b274-458a-b655-6df2a18a8e49-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-585fcb4787-d6kbd\" (UID: \"0e99380b-b274-458a-b655-6df2a18a8e49\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd"
Sep 17 22:22:26.919308 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.919293    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e99380b-b274-458a-b655-6df2a18a8e49-cabundle-cert\") pod \"auth-invalid-token-test-kserve-585fcb4787-d6kbd\" (UID: \"0e99380b-b274-458a-b655-6df2a18a8e49\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd"
Sep 17 22:22:26.921062 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.921019    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e99380b-b274-458a-b655-6df2a18a8e49-dshm\") pod \"auth-invalid-token-test-kserve-585fcb4787-d6kbd\" (UID: \"0e99380b-b274-458a-b655-6df2a18a8e49\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd"
Sep 17 22:22:26.921821 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.921797    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e99380b-b274-458a-b655-6df2a18a8e49-tls-certs\") pod \"auth-invalid-token-test-kserve-585fcb4787-d6kbd\" (UID: \"0e99380b-b274-458a-b655-6df2a18a8e49\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd"
Sep 17 22:22:26.926850 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:26.926829    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp87l\" (UniqueName: \"kubernetes.io/projected/0e99380b-b274-458a-b655-6df2a18a8e49-kube-api-access-mp87l\") pod \"auth-invalid-token-test-kserve-585fcb4787-d6kbd\" (UID: \"0e99380b-b274-458a-b655-6df2a18a8e49\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd"
Sep 17 22:22:27.007880 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:27.007803    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:22:27.046951 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:27.046925    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd"
Sep 17 22:22:27.329703 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:27.329680    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd"]
Sep 17 22:22:27.331522 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:22:27.331487    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e99380b_b274_458a_b655_6df2a18a8e49.slice/crio-2f79318e90a6386348a013095f8ee4ad97cf6d8791cad2595d75ce6effd5730c WatchSource:0}: Error finding container 2f79318e90a6386348a013095f8ee4ad97cf6d8791cad2595d75ce6effd5730c: Status 404 returned error can't find the container with id 2f79318e90a6386348a013095f8ee4ad97cf6d8791cad2595d75ce6effd5730c
Sep 17 22:22:27.691256 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:27.691215    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd" event={"ID":"0e99380b-b274-458a-b655-6df2a18a8e49","Type":"ContainerStarted","Data":"5a82ea6811ab480fe5599b77edbea1f58e0706d370c2fd539794373f97d2039a"}
Sep 17 22:22:27.691460 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:27.691284    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd" event={"ID":"0e99380b-b274-458a-b655-6df2a18a8e49","Type":"ContainerStarted","Data":"2f79318e90a6386348a013095f8ee4ad97cf6d8791cad2595d75ce6effd5730c"}
Sep 17 22:22:29.705585 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:29.705558    2555 generic.go:356] "Generic (PLEG): container finished" podID="0e99380b-b274-458a-b655-6df2a18a8e49" containerID="5a82ea6811ab480fe5599b77edbea1f58e0706d370c2fd539794373f97d2039a" exitCode=0
Sep 17 22:22:29.705926 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:29.705630    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd" event={"ID":"0e99380b-b274-458a-b655-6df2a18a8e49","Type":"ContainerDied","Data":"5a82ea6811ab480fe5599b77edbea1f58e0706d370c2fd539794373f97d2039a"}
Sep 17 22:22:30.714867 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:30.714827    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd" event={"ID":"0e99380b-b274-458a-b655-6df2a18a8e49","Type":"ContainerStarted","Data":"82f821198a58403f7fb1bfb296136c8bda64d8b0b080b24673959a7b20c72e0a"}
Sep 17 22:22:30.736345 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:30.736295    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd" podStartSLOduration=2.524985177 podStartE2EDuration="4.736281084s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.211295907s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:22:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:22:30.734208365 +0000 UTC m=+1452.180253637" watchObservedRunningTime="2026-09-17 22:22:30.736281084 +0000 UTC m=+1452.182326355"
Sep 17 22:22:31.047637 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:31.047554    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd"
Sep 17 22:22:37.047900 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:37.047857    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd"
Sep 17 22:22:37.051778 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:37.049728    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd" podUID="0e99380b-b274-458a-b655-6df2a18a8e49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 17 22:22:38.596935 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:38.596890    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:22:38.693339 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:38.693308    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:22:42.324490 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:42.324448    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:22:44.813099 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:44.813052    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:22:47.047578 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:47.047537    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd" podUID="0e99380b-b274-458a-b655-6df2a18a8e49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 17 22:22:50.407514 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:50.407480    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:22:50.507523 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:50.507490    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:22:57.047839 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:22:57.047789    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd" podUID="0e99380b-b274-458a-b655-6df2a18a8e49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 17 22:23:01.960872 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:01.960839    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq"]
Sep 17 22:23:01.964844 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:01.964819    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq"
Sep 17 22:23:01.967153 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:01.967127    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 17 22:23:01.967304 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:01.967166    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 17 22:23:01.967304 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:01.967197    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:23:01.967304 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:01.967207    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 17 22:23:01.967972 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:01.967953    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 17 22:23:01.968079 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:01.967991    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-qg2mv\""
Sep 17 22:23:01.974973 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:01.974943    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq"]
Sep 17 22:23:01.977328 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:01.977305    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:23:02.042458 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.042409    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faa98f61-fa4c-4787-a67b-1976e40b46de-model-cache\") pod \"tokenizer-migration-path-test-kserve-867557464b-n5xtq\" (UID: \"faa98f61-fa4c-4787-a67b-1976e40b46de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq"
Sep 17 22:23:02.042659 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.042463    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pggzc\" (UniqueName: \"kubernetes.io/projected/faa98f61-fa4c-4787-a67b-1976e40b46de-kube-api-access-pggzc\") pod \"tokenizer-migration-path-test-kserve-867557464b-n5xtq\" (UID: \"faa98f61-fa4c-4787-a67b-1976e40b46de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq"
Sep 17 22:23:02.042659 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.042537    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faa98f61-fa4c-4787-a67b-1976e40b46de-tls-certs\") pod \"tokenizer-migration-path-test-kserve-867557464b-n5xtq\" (UID: \"faa98f61-fa4c-4787-a67b-1976e40b46de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq"
Sep 17 22:23:02.042659 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.042573    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/faa98f61-fa4c-4787-a67b-1976e40b46de-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-867557464b-n5xtq\" (UID: \"faa98f61-fa4c-4787-a67b-1976e40b46de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq"
Sep 17 22:23:02.042659 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.042622    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/faa98f61-fa4c-4787-a67b-1976e40b46de-dshm\") pod \"tokenizer-migration-path-test-kserve-867557464b-n5xtq\" (UID: \"faa98f61-fa4c-4787-a67b-1976e40b46de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq"
Sep 17 22:23:02.042659 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.042657    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/faa98f61-fa4c-4787-a67b-1976e40b46de-home\") pod \"tokenizer-migration-path-test-kserve-867557464b-n5xtq\" (UID: \"faa98f61-fa4c-4787-a67b-1976e40b46de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq"
Sep 17 22:23:02.042873 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.042725    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faa98f61-fa4c-4787-a67b-1976e40b46de-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-867557464b-n5xtq\" (UID: \"faa98f61-fa4c-4787-a67b-1976e40b46de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq"
Sep 17 22:23:02.042873 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.042771    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/faa98f61-fa4c-4787-a67b-1976e40b46de-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-867557464b-n5xtq\" (UID: \"faa98f61-fa4c-4787-a67b-1976e40b46de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq"
Sep 17 22:23:02.143294 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.143253    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faa98f61-fa4c-4787-a67b-1976e40b46de-tls-certs\") pod \"tokenizer-migration-path-test-kserve-867557464b-n5xtq\" (UID: \"faa98f61-fa4c-4787-a67b-1976e40b46de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq"
Sep 17 22:23:02.143294 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.143295    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/faa98f61-fa4c-4787-a67b-1976e40b46de-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-867557464b-n5xtq\" (UID: \"faa98f61-fa4c-4787-a67b-1976e40b46de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq"
Sep 17 22:23:02.143567 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.143318    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/faa98f61-fa4c-4787-a67b-1976e40b46de-dshm\") pod \"tokenizer-migration-path-test-kserve-867557464b-n5xtq\" (UID: \"faa98f61-fa4c-4787-a67b-1976e40b46de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq"
Sep 17 22:23:02.143567 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.143339    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/faa98f61-fa4c-4787-a67b-1976e40b46de-home\") pod \"tokenizer-migration-path-test-kserve-867557464b-n5xtq\" (UID: \"faa98f61-fa4c-4787-a67b-1976e40b46de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq"
Sep 17 22:23:02.143567 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.143364    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faa98f61-fa4c-4787-a67b-1976e40b46de-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-867557464b-n5xtq\" (UID: \"faa98f61-fa4c-4787-a67b-1976e40b46de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq"
Sep 17 22:23:02.143567 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.143397    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/faa98f61-fa4c-4787-a67b-1976e40b46de-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-867557464b-n5xtq\" (UID: \"faa98f61-fa4c-4787-a67b-1976e40b46de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq"
Sep 17 22:23:02.143567 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.143470    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faa98f61-fa4c-4787-a67b-1976e40b46de-model-cache\") pod \"tokenizer-migration-path-test-kserve-867557464b-n5xtq\" (UID: \"faa98f61-fa4c-4787-a67b-1976e40b46de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq"
Sep 17 22:23:02.143567 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.143491    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pggzc\" (UniqueName: \"kubernetes.io/projected/faa98f61-fa4c-4787-a67b-1976e40b46de-kube-api-access-pggzc\") pod \"tokenizer-migration-path-test-kserve-867557464b-n5xtq\" (UID: \"faa98f61-fa4c-4787-a67b-1976e40b46de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq"
Sep 17 22:23:02.143890 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.143793    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/faa98f61-fa4c-4787-a67b-1976e40b46de-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-867557464b-n5xtq\" (UID: \"faa98f61-fa4c-4787-a67b-1976e40b46de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq"
Sep 17 22:23:02.143890 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.143817    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/faa98f61-fa4c-4787-a67b-1976e40b46de-home\") pod \"tokenizer-migration-path-test-kserve-867557464b-n5xtq\" (UID: \"faa98f61-fa4c-4787-a67b-1976e40b46de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq"
Sep 17 22:23:02.143890 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.143872    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faa98f61-fa4c-4787-a67b-1976e40b46de-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-867557464b-n5xtq\" (UID: \"faa98f61-fa4c-4787-a67b-1976e40b46de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq"
Sep 17 22:23:02.144098 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.144071    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faa98f61-fa4c-4787-a67b-1976e40b46de-model-cache\") pod \"tokenizer-migration-path-test-kserve-867557464b-n5xtq\" (UID: \"faa98f61-fa4c-4787-a67b-1976e40b46de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq"
Sep 17 22:23:02.144452 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.144432    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/faa98f61-fa4c-4787-a67b-1976e40b46de-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-867557464b-n5xtq\" (UID: \"faa98f61-fa4c-4787-a67b-1976e40b46de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq"
Sep 17 22:23:02.145844 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.145827    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/faa98f61-fa4c-4787-a67b-1976e40b46de-dshm\") pod \"tokenizer-migration-path-test-kserve-867557464b-n5xtq\" (UID: \"faa98f61-fa4c-4787-a67b-1976e40b46de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq"
Sep 17 22:23:02.146223 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.146200    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faa98f61-fa4c-4787-a67b-1976e40b46de-tls-certs\") pod \"tokenizer-migration-path-test-kserve-867557464b-n5xtq\" (UID: \"faa98f61-fa4c-4787-a67b-1976e40b46de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq"
Sep 17 22:23:02.152747 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.152720    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pggzc\" (UniqueName: \"kubernetes.io/projected/faa98f61-fa4c-4787-a67b-1976e40b46de-kube-api-access-pggzc\") pod \"tokenizer-migration-path-test-kserve-867557464b-n5xtq\" (UID: \"faa98f61-fa4c-4787-a67b-1976e40b46de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq"
Sep 17 22:23:02.206072 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.206037    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w"]
Sep 17 22:23:02.211016 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.210948    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w"
Sep 17 22:23:02.222609 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.221576    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w"]
Sep 17 22:23:02.233644 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.233612    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:23:02.243848 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.243802    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53693869-5914-4830-b952-04e2f6faed42-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w\" (UID: \"53693869-5914-4830-b952-04e2f6faed42\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w"
Sep 17 22:23:02.243848 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.243847    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53693869-5914-4830-b952-04e2f6faed42-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w\" (UID: \"53693869-5914-4830-b952-04e2f6faed42\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w"
Sep 17 22:23:02.244113 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.243880    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53693869-5914-4830-b952-04e2f6faed42-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w\" (UID: \"53693869-5914-4830-b952-04e2f6faed42\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w"
Sep 17 22:23:02.244113 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.243930    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53693869-5914-4830-b952-04e2f6faed42-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w\" (UID: \"53693869-5914-4830-b952-04e2f6faed42\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w"
Sep 17 22:23:02.276606 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.276573    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq"
Sep 17 22:23:02.331473 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.331444    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:23:02.346015 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.345715    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53693869-5914-4830-b952-04e2f6faed42-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w\" (UID: \"53693869-5914-4830-b952-04e2f6faed42\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w"
Sep 17 22:23:02.346015 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.345823    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53693869-5914-4830-b952-04e2f6faed42-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w\" (UID: \"53693869-5914-4830-b952-04e2f6faed42\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w"
Sep 17 22:23:02.346015 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.345893    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53693869-5914-4830-b952-04e2f6faed42-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w\" (UID: \"53693869-5914-4830-b952-04e2f6faed42\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w"
Sep 17 22:23:02.347151 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.346115    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53693869-5914-4830-b952-04e2f6faed42-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w\" (UID: \"53693869-5914-4830-b952-04e2f6faed42\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w"
Sep 17 22:23:02.347151 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.346529    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53693869-5914-4830-b952-04e2f6faed42-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w\" (UID: \"53693869-5914-4830-b952-04e2f6faed42\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w"
Sep 17 22:23:02.347151 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.346681    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53693869-5914-4830-b952-04e2f6faed42-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w\" (UID: \"53693869-5914-4830-b952-04e2f6faed42\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w"
Sep 17 22:23:02.347151 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.346926    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53693869-5914-4830-b952-04e2f6faed42-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w\" (UID: \"53693869-5914-4830-b952-04e2f6faed42\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w"
Sep 17 22:23:02.349748 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.349725    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53693869-5914-4830-b952-04e2f6faed42-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w\" (UID: \"53693869-5914-4830-b952-04e2f6faed42\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w"
Sep 17 22:23:02.526229 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.526117    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w"
Sep 17 22:23:02.552278 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.552251    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq"]
Sep 17 22:23:02.553478 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:23:02.553455    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaa98f61_fa4c_4787_a67b_1976e40b46de.slice/crio-641698bed5197d1c31d561f0d5a8bec2337a543fe7dbd770a790d91af570f770 WatchSource:0}: Error finding container 641698bed5197d1c31d561f0d5a8bec2337a543fe7dbd770a790d91af570f770: Status 404 returned error can't find the container with id 641698bed5197d1c31d561f0d5a8bec2337a543fe7dbd770a790d91af570f770
Sep 17 22:23:02.812956 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.812923    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w"]
Sep 17 22:23:02.814170 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:23:02.814132    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53693869_5914_4830_b952_04e2f6faed42.slice/crio-a941608f7939be6d29bbbf406d27bc8c5f922f3a86c8f2e8b9510472492ad929 WatchSource:0}: Error finding container a941608f7939be6d29bbbf406d27bc8c5f922f3a86c8f2e8b9510472492ad929: Status 404 returned error can't find the container with id a941608f7939be6d29bbbf406d27bc8c5f922f3a86c8f2e8b9510472492ad929
Sep 17 22:23:02.906579 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.906512    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w" event={"ID":"53693869-5914-4830-b952-04e2f6faed42","Type":"ContainerStarted","Data":"ea6b4fee5ff2d8ff61699a5141a1a6314484c88924a6923244e0b3c0559d6c60"}
Sep 17 22:23:02.906786 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.906584    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w" event={"ID":"53693869-5914-4830-b952-04e2f6faed42","Type":"ContainerStarted","Data":"a941608f7939be6d29bbbf406d27bc8c5f922f3a86c8f2e8b9510472492ad929"}
Sep 17 22:23:02.909134 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.909097    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq" event={"ID":"faa98f61-fa4c-4787-a67b-1976e40b46de","Type":"ContainerStarted","Data":"e8063f64970f4a86cc33193a582418b857678e7a44b966afae1d9b1ec5c3e9b1"}
Sep 17 22:23:02.909328 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:02.909156    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq" event={"ID":"faa98f61-fa4c-4787-a67b-1976e40b46de","Type":"ContainerStarted","Data":"641698bed5197d1c31d561f0d5a8bec2337a543fe7dbd770a790d91af570f770"}
Sep 17 22:23:05.932586 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:05.932548    2555 generic.go:356] "Generic (PLEG): container finished" podID="faa98f61-fa4c-4787-a67b-1976e40b46de" containerID="e8063f64970f4a86cc33193a582418b857678e7a44b966afae1d9b1ec5c3e9b1" exitCode=0
Sep 17 22:23:05.933030 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:05.932620    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq" event={"ID":"faa98f61-fa4c-4787-a67b-1976e40b46de","Type":"ContainerDied","Data":"e8063f64970f4a86cc33193a582418b857678e7a44b966afae1d9b1ec5c3e9b1"}
Sep 17 22:23:05.934478 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:05.934452    2555 generic.go:356] "Generic (PLEG): container finished" podID="53693869-5914-4830-b952-04e2f6faed42" containerID="ea6b4fee5ff2d8ff61699a5141a1a6314484c88924a6923244e0b3c0559d6c60" exitCode=0
Sep 17 22:23:05.934606 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:05.934484    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w" event={"ID":"53693869-5914-4830-b952-04e2f6faed42","Type":"ContainerDied","Data":"ea6b4fee5ff2d8ff61699a5141a1a6314484c88924a6923244e0b3c0559d6c60"}
Sep 17 22:23:06.942141 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:06.942108    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w" event={"ID":"53693869-5914-4830-b952-04e2f6faed42","Type":"ContainerStarted","Data":"c9a7067500399cccb354a461ac5276ef2f168c4e673cf8d6ccfcdd119edcdaac"}
Sep 17 22:23:06.942637 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:06.942351    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w"
Sep 17 22:23:06.944363 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:06.944341    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq" event={"ID":"faa98f61-fa4c-4787-a67b-1976e40b46de","Type":"ContainerStarted","Data":"34725cd88898571cb9af27fe944d71fbd29a24b55a5d978aa203c17c482c8480"}
Sep 17 22:23:06.963506 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:06.963448    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w" podStartSLOduration=2.280817492 podStartE2EDuration="4.9634272s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.682609708s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:23:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:23:06.959257338 +0000 UTC m=+1488.405302622" watchObservedRunningTime="2026-09-17 22:23:06.9634272 +0000 UTC m=+1488.409472475"
Sep 17 22:23:06.978647 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:06.978595    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq" podStartSLOduration=3.354527014 podStartE2EDuration="5.978577358s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.624050344s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:23:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:23:06.97678971 +0000 UTC m=+1488.422834966" watchObservedRunningTime="2026-09-17 22:23:06.978577358 +0000 UTC m=+1488.424622631"
Sep 17 22:23:07.047793 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:07.047751    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd" podUID="0e99380b-b274-458a-b655-6df2a18a8e49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 17 22:23:07.277128 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:07.277038    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq"
Sep 17 22:23:12.277194 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:12.277137    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq"
Sep 17 22:23:12.291176 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:12.291153    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq"
Sep 17 22:23:12.994768 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:12.994732    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq"
Sep 17 22:23:15.228761 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:15.228731    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:23:17.047944 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:17.047900    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd" podUID="0e99380b-b274-458a-b655-6df2a18a8e49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 17 22:23:22.702965 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:22.702934    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-7njvj_a324c9ab-5d42-4d31-8ff7-449f619be774/console-operator/1.log"
Sep 17 22:23:22.706097 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:22.706076    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-7njvj_a324c9ab-5d42-4d31-8ff7-449f619be774/console-operator/1.log"
Sep 17 22:23:22.707964 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:22.707942    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s7f5_4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf/ovn-acl-logging/0.log"
Sep 17 22:23:22.710773 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:22.710749    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s7f5_4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf/ovn-acl-logging/0.log"
Sep 17 22:23:27.047557 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:27.047505    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd" podUID="0e99380b-b274-458a-b655-6df2a18a8e49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 17 22:23:37.047799 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:37.047700    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd" podUID="0e99380b-b274-458a-b655-6df2a18a8e49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 17 22:23:37.962702 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:37.962668    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w"
Sep 17 22:23:39.825701 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:39.825664    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w_53693869-5914-4830-b952-04e2f6faed42/main/0.log"
Sep 17 22:23:40.154342 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:40.154290    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq"]
Sep 17 22:23:40.156056 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:40.156024    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w"]
Sep 17 22:23:40.156329 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:40.156292    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w" podUID="53693869-5914-4830-b952-04e2f6faed42" containerName="main" containerID="cri-o://c9a7067500399cccb354a461ac5276ef2f168c4e673cf8d6ccfcdd119edcdaac" gracePeriod=30
Sep 17 22:23:40.166730 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:23:40.166692    2555 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 22:23:40.166730 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:23:40.166730 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:23:40.166730 ip-10-0-140-157 kubenswrapper[2555]:  > containerID="34725cd88898571cb9af27fe944d71fbd29a24b55a5d978aa203c17c482c8480" cmd=["/bin/sleep","15"]
Sep 17 22:23:40.166989 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:40.166748    2555 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq" message="" err=<
Sep 17 22:23:40.166989 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:23:40.166989 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:23:40.166989 ip-10-0-140-157 kubenswrapper[2555]:  >
Sep 17 22:23:40.166989 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:23:40.166760    2555 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 22:23:40.166989 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:23:40.166989 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:23:40.166989 ip-10-0-140-157 kubenswrapper[2555]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq" podUID="faa98f61-fa4c-4787-a67b-1976e40b46de" containerName="main" containerID="cri-o://34725cd88898571cb9af27fe944d71fbd29a24b55a5d978aa203c17c482c8480"
Sep 17 22:23:40.166989 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:40.166796    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq" podUID="faa98f61-fa4c-4787-a67b-1976e40b46de" containerName="main" containerID="cri-o://34725cd88898571cb9af27fe944d71fbd29a24b55a5d978aa203c17c482c8480" gracePeriod=60
Sep 17 22:23:40.534009 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:40.533982    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq"
Sep 17 22:23:40.667607 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:40.667574    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:23:40.713538 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:40.713459    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/faa98f61-fa4c-4787-a67b-1976e40b46de-kserve-provision-location\") pod \"faa98f61-fa4c-4787-a67b-1976e40b46de\" (UID: \"faa98f61-fa4c-4787-a67b-1976e40b46de\") "
Sep 17 22:23:40.713538 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:40.713497    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faa98f61-fa4c-4787-a67b-1976e40b46de-tmp-dir\") pod \"faa98f61-fa4c-4787-a67b-1976e40b46de\" (UID: \"faa98f61-fa4c-4787-a67b-1976e40b46de\") "
Sep 17 22:23:40.713538 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:40.713526    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/faa98f61-fa4c-4787-a67b-1976e40b46de-dshm\") pod \"faa98f61-fa4c-4787-a67b-1976e40b46de\" (UID: \"faa98f61-fa4c-4787-a67b-1976e40b46de\") "
Sep 17 22:23:40.713849 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:40.713555    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/faa98f61-fa4c-4787-a67b-1976e40b46de-cabundle-cert\") pod \"faa98f61-fa4c-4787-a67b-1976e40b46de\" (UID: \"faa98f61-fa4c-4787-a67b-1976e40b46de\") "
Sep 17 22:23:40.713849 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:40.713604    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pggzc\" (UniqueName: \"kubernetes.io/projected/faa98f61-fa4c-4787-a67b-1976e40b46de-kube-api-access-pggzc\") pod \"faa98f61-fa4c-4787-a67b-1976e40b46de\" (UID: \"faa98f61-fa4c-4787-a67b-1976e40b46de\") "
Sep 17 22:23:40.713849 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:40.713641    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faa98f61-fa4c-4787-a67b-1976e40b46de-tls-certs\") pod \"faa98f61-fa4c-4787-a67b-1976e40b46de\" (UID: \"faa98f61-fa4c-4787-a67b-1976e40b46de\") "
Sep 17 22:23:40.713849 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:40.713675    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faa98f61-fa4c-4787-a67b-1976e40b46de-model-cache\") pod \"faa98f61-fa4c-4787-a67b-1976e40b46de\" (UID: \"faa98f61-fa4c-4787-a67b-1976e40b46de\") "
Sep 17 22:23:40.713849 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:40.713742    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/faa98f61-fa4c-4787-a67b-1976e40b46de-home\") pod \"faa98f61-fa4c-4787-a67b-1976e40b46de\" (UID: \"faa98f61-fa4c-4787-a67b-1976e40b46de\") "
Sep 17 22:23:40.713849 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:40.713782    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa98f61-fa4c-4787-a67b-1976e40b46de-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "faa98f61-fa4c-4787-a67b-1976e40b46de" (UID: "faa98f61-fa4c-4787-a67b-1976e40b46de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:23:40.714127 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:40.713972    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faa98f61-fa4c-4787-a67b-1976e40b46de-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "faa98f61-fa4c-4787-a67b-1976e40b46de" (UID: "faa98f61-fa4c-4787-a67b-1976e40b46de"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:23:40.714127 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:40.714096    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faa98f61-fa4c-4787-a67b-1976e40b46de-tmp-dir\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:23:40.714127 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:40.714118    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/faa98f61-fa4c-4787-a67b-1976e40b46de-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:23:40.714332 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:40.714128    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa98f61-fa4c-4787-a67b-1976e40b46de-model-cache" (OuterVolumeSpecName: "model-cache") pod "faa98f61-fa4c-4787-a67b-1976e40b46de" (UID: "faa98f61-fa4c-4787-a67b-1976e40b46de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:23:40.714332 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:40.714325    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa98f61-fa4c-4787-a67b-1976e40b46de-home" (OuterVolumeSpecName: "home") pod "faa98f61-fa4c-4787-a67b-1976e40b46de" (UID: "faa98f61-fa4c-4787-a67b-1976e40b46de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:23:40.717020 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:40.716991    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faa98f61-fa4c-4787-a67b-1976e40b46de-kube-api-access-pggzc" (OuterVolumeSpecName: "kube-api-access-pggzc") pod "faa98f61-fa4c-4787-a67b-1976e40b46de" (UID: "faa98f61-fa4c-4787-a67b-1976e40b46de"). InnerVolumeSpecName "kube-api-access-pggzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:23:40.717475 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:40.717451    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faa98f61-fa4c-4787-a67b-1976e40b46de-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "faa98f61-fa4c-4787-a67b-1976e40b46de" (UID: "faa98f61-fa4c-4787-a67b-1976e40b46de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:23:40.717562 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:40.717537    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa98f61-fa4c-4787-a67b-1976e40b46de-dshm" (OuterVolumeSpecName: "dshm") pod "faa98f61-fa4c-4787-a67b-1976e40b46de" (UID: "faa98f61-fa4c-4787-a67b-1976e40b46de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:23:40.735680 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:40.735645    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa98f61-fa4c-4787-a67b-1976e40b46de-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "faa98f61-fa4c-4787-a67b-1976e40b46de" (UID: "faa98f61-fa4c-4787-a67b-1976e40b46de"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:23:40.768131 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:40.768084    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:23:40.814536 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:40.814515    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w"
Sep 17 22:23:40.815117 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:40.815096    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/faa98f61-fa4c-4787-a67b-1976e40b46de-home\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:23:40.815266 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:40.815121    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/faa98f61-fa4c-4787-a67b-1976e40b46de-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:23:40.815266 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:40.815135    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/faa98f61-fa4c-4787-a67b-1976e40b46de-dshm\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:23:40.815266 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:40.815150    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pggzc\" (UniqueName: \"kubernetes.io/projected/faa98f61-fa4c-4787-a67b-1976e40b46de-kube-api-access-pggzc\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:23:40.815266 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:40.815163    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faa98f61-fa4c-4787-a67b-1976e40b46de-tls-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:23:40.815266 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:40.815175    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faa98f61-fa4c-4787-a67b-1976e40b46de-model-cache\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:23:40.916093 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:40.916060    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53693869-5914-4830-b952-04e2f6faed42-cabundle-cert\") pod \"53693869-5914-4830-b952-04e2f6faed42\" (UID: \"53693869-5914-4830-b952-04e2f6faed42\") "
Sep 17 22:23:40.916480 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:40.916107    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53693869-5914-4830-b952-04e2f6faed42-tls-certs\") pod \"53693869-5914-4830-b952-04e2f6faed42\" (UID: \"53693869-5914-4830-b952-04e2f6faed42\") "
Sep 17 22:23:40.916480 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:40.916154    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53693869-5914-4830-b952-04e2f6faed42-tmp-dir\") pod \"53693869-5914-4830-b952-04e2f6faed42\" (UID: \"53693869-5914-4830-b952-04e2f6faed42\") "
Sep 17 22:23:40.916480 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:40.916175    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53693869-5914-4830-b952-04e2f6faed42-kserve-provision-location\") pod \"53693869-5914-4830-b952-04e2f6faed42\" (UID: \"53693869-5914-4830-b952-04e2f6faed42\") "
Sep 17 22:23:40.916593 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:40.916477    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53693869-5914-4830-b952-04e2f6faed42-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "53693869-5914-4830-b952-04e2f6faed42" (UID: "53693869-5914-4830-b952-04e2f6faed42"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:23:40.916593 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:40.916566    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53693869-5914-4830-b952-04e2f6faed42-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "53693869-5914-4830-b952-04e2f6faed42" (UID: "53693869-5914-4830-b952-04e2f6faed42"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:23:40.918275 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:40.918247    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53693869-5914-4830-b952-04e2f6faed42-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "53693869-5914-4830-b952-04e2f6faed42" (UID: "53693869-5914-4830-b952-04e2f6faed42"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:23:40.936352 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:40.936326    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53693869-5914-4830-b952-04e2f6faed42-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "53693869-5914-4830-b952-04e2f6faed42" (UID: "53693869-5914-4830-b952-04e2f6faed42"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:23:41.017165 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:41.017076    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53693869-5914-4830-b952-04e2f6faed42-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:23:41.017165 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:41.017107    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53693869-5914-4830-b952-04e2f6faed42-tls-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:23:41.017165 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:41.017118    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53693869-5914-4830-b952-04e2f6faed42-tmp-dir\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:23:41.017165 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:41.017131    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53693869-5914-4830-b952-04e2f6faed42-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:23:41.164042 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:41.164011    2555 generic.go:356] "Generic (PLEG): container finished" podID="faa98f61-fa4c-4787-a67b-1976e40b46de" containerID="34725cd88898571cb9af27fe944d71fbd29a24b55a5d978aa203c17c482c8480" exitCode=0
Sep 17 22:23:41.164256 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:41.164073    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq"
Sep 17 22:23:41.164256 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:41.164109    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq" event={"ID":"faa98f61-fa4c-4787-a67b-1976e40b46de","Type":"ContainerDied","Data":"34725cd88898571cb9af27fe944d71fbd29a24b55a5d978aa203c17c482c8480"}
Sep 17 22:23:41.164256 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:41.164197    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq" event={"ID":"faa98f61-fa4c-4787-a67b-1976e40b46de","Type":"ContainerDied","Data":"641698bed5197d1c31d561f0d5a8bec2337a543fe7dbd770a790d91af570f770"}
Sep 17 22:23:41.164256 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:41.164207    2555 scope.go:117] "RemoveContainer" containerID="34725cd88898571cb9af27fe944d71fbd29a24b55a5d978aa203c17c482c8480"
Sep 17 22:23:41.165870 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:41.165843    2555 generic.go:356] "Generic (PLEG): container finished" podID="53693869-5914-4830-b952-04e2f6faed42" containerID="c9a7067500399cccb354a461ac5276ef2f168c4e673cf8d6ccfcdd119edcdaac" exitCode=0
Sep 17 22:23:41.165984 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:41.165906    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w"
Sep 17 22:23:41.165984 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:41.165917    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w" event={"ID":"53693869-5914-4830-b952-04e2f6faed42","Type":"ContainerDied","Data":"c9a7067500399cccb354a461ac5276ef2f168c4e673cf8d6ccfcdd119edcdaac"}
Sep 17 22:23:41.165984 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:41.165952    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w" event={"ID":"53693869-5914-4830-b952-04e2f6faed42","Type":"ContainerDied","Data":"a941608f7939be6d29bbbf406d27bc8c5f922f3a86c8f2e8b9510472492ad929"}
Sep 17 22:23:41.184403 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:41.184377    2555 scope.go:117] "RemoveContainer" containerID="e8063f64970f4a86cc33193a582418b857678e7a44b966afae1d9b1ec5c3e9b1"
Sep 17 22:23:41.193202 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:41.193166    2555 scope.go:117] "RemoveContainer" containerID="34725cd88898571cb9af27fe944d71fbd29a24b55a5d978aa203c17c482c8480"
Sep 17 22:23:41.193519 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:23:41.193493    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34725cd88898571cb9af27fe944d71fbd29a24b55a5d978aa203c17c482c8480\": container with ID starting with 34725cd88898571cb9af27fe944d71fbd29a24b55a5d978aa203c17c482c8480 not found: ID does not exist" containerID="34725cd88898571cb9af27fe944d71fbd29a24b55a5d978aa203c17c482c8480"
Sep 17 22:23:41.193578 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:41.193534    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34725cd88898571cb9af27fe944d71fbd29a24b55a5d978aa203c17c482c8480"} err="failed to get container status \"34725cd88898571cb9af27fe944d71fbd29a24b55a5d978aa203c17c482c8480\": rpc error: code = NotFound desc = could not find container \"34725cd88898571cb9af27fe944d71fbd29a24b55a5d978aa203c17c482c8480\": container with ID starting with 34725cd88898571cb9af27fe944d71fbd29a24b55a5d978aa203c17c482c8480 not found: ID does not exist"
Sep 17 22:23:41.193578 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:41.193562    2555 scope.go:117] "RemoveContainer" containerID="e8063f64970f4a86cc33193a582418b857678e7a44b966afae1d9b1ec5c3e9b1"
Sep 17 22:23:41.193933 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:23:41.193912    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8063f64970f4a86cc33193a582418b857678e7a44b966afae1d9b1ec5c3e9b1\": container with ID starting with e8063f64970f4a86cc33193a582418b857678e7a44b966afae1d9b1ec5c3e9b1 not found: ID does not exist" containerID="e8063f64970f4a86cc33193a582418b857678e7a44b966afae1d9b1ec5c3e9b1"
Sep 17 22:23:41.194095 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:41.193935    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8063f64970f4a86cc33193a582418b857678e7a44b966afae1d9b1ec5c3e9b1"} err="failed to get container status \"e8063f64970f4a86cc33193a582418b857678e7a44b966afae1d9b1ec5c3e9b1\": rpc error: code = NotFound desc = could not find container \"e8063f64970f4a86cc33193a582418b857678e7a44b966afae1d9b1ec5c3e9b1\": container with ID starting with e8063f64970f4a86cc33193a582418b857678e7a44b966afae1d9b1ec5c3e9b1 not found: ID does not exist"
Sep 17 22:23:41.194095 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:41.193950    2555 scope.go:117] "RemoveContainer" containerID="c9a7067500399cccb354a461ac5276ef2f168c4e673cf8d6ccfcdd119edcdaac"
Sep 17 22:23:41.197438 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:41.197418    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq"]
Sep 17 22:23:41.201804 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:41.201779    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-n5xtq"]
Sep 17 22:23:41.203640 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:41.203621    2555 scope.go:117] "RemoveContainer" containerID="ea6b4fee5ff2d8ff61699a5141a1a6314484c88924a6923244e0b3c0559d6c60"
Sep 17 22:23:41.210856 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:41.210836    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w"]
Sep 17 22:23:41.214027 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:41.214008    2555 scope.go:117] "RemoveContainer" containerID="c9a7067500399cccb354a461ac5276ef2f168c4e673cf8d6ccfcdd119edcdaac"
Sep 17 22:23:41.214380 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:23:41.214349    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9a7067500399cccb354a461ac5276ef2f168c4e673cf8d6ccfcdd119edcdaac\": container with ID starting with c9a7067500399cccb354a461ac5276ef2f168c4e673cf8d6ccfcdd119edcdaac not found: ID does not exist" containerID="c9a7067500399cccb354a461ac5276ef2f168c4e673cf8d6ccfcdd119edcdaac"
Sep 17 22:23:41.214495 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:41.214386    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5df5fd8568-7jd9w"]
Sep 17 22:23:41.214495 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:41.214388    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9a7067500399cccb354a461ac5276ef2f168c4e673cf8d6ccfcdd119edcdaac"} err="failed to get container status \"c9a7067500399cccb354a461ac5276ef2f168c4e673cf8d6ccfcdd119edcdaac\": rpc error: code = NotFound desc = could not find container \"c9a7067500399cccb354a461ac5276ef2f168c4e673cf8d6ccfcdd119edcdaac\": container with ID starting with c9a7067500399cccb354a461ac5276ef2f168c4e673cf8d6ccfcdd119edcdaac not found: ID does not exist"
Sep 17 22:23:41.214495 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:41.214411    2555 scope.go:117] "RemoveContainer" containerID="ea6b4fee5ff2d8ff61699a5141a1a6314484c88924a6923244e0b3c0559d6c60"
Sep 17 22:23:41.214755 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:23:41.214723    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea6b4fee5ff2d8ff61699a5141a1a6314484c88924a6923244e0b3c0559d6c60\": container with ID starting with ea6b4fee5ff2d8ff61699a5141a1a6314484c88924a6923244e0b3c0559d6c60 not found: ID does not exist" containerID="ea6b4fee5ff2d8ff61699a5141a1a6314484c88924a6923244e0b3c0559d6c60"
Sep 17 22:23:41.214806 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:41.214762    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea6b4fee5ff2d8ff61699a5141a1a6314484c88924a6923244e0b3c0559d6c60"} err="failed to get container status \"ea6b4fee5ff2d8ff61699a5141a1a6314484c88924a6923244e0b3c0559d6c60\": rpc error: code = NotFound desc = could not find container \"ea6b4fee5ff2d8ff61699a5141a1a6314484c88924a6923244e0b3c0559d6c60\": container with ID starting with ea6b4fee5ff2d8ff61699a5141a1a6314484c88924a6923244e0b3c0559d6c60 not found: ID does not exist"
Sep 17 22:23:42.679039 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:42.679004    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53693869-5914-4830-b952-04e2f6faed42" path="/var/lib/kubelet/pods/53693869-5914-4830-b952-04e2f6faed42/volumes"
Sep 17 22:23:42.679527 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:42.679511    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="faa98f61-fa4c-4787-a67b-1976e40b46de" path="/var/lib/kubelet/pods/faa98f61-fa4c-4787-a67b-1976e40b46de/volumes"
Sep 17 22:23:47.047870 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:47.047832    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd" podUID="0e99380b-b274-458a-b655-6df2a18a8e49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 17 22:23:57.047567 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:23:57.047523    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd" podUID="0e99380b-b274-458a-b655-6df2a18a8e49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 17 22:24:00.539062 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:00.539024    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:24:00.639873 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:00.639841    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:24:07.048255 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:07.048209    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd" podUID="0e99380b-b274-458a-b655-6df2a18a8e49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 17 22:24:17.048426 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:17.048376    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd" podUID="0e99380b-b274-458a-b655-6df2a18a8e49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 17 22:24:17.475009 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:17.474975    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:24:17.873329 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:17.873236    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:24:27.057623 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:27.057593    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd"
Sep 17 22:24:27.066161 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:27.066133    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd"
Sep 17 22:24:28.578508 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:28.578472    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd"]
Sep 17 22:24:28.579469 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:28.579437    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:24:28.645402 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:28.645370    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:24:30.138417 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:30.138381    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:24:30.234424 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:30.234394    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:24:33.752345 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:33.752310    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd"]
Sep 17 22:24:33.752826 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:33.752695    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="faa98f61-fa4c-4787-a67b-1976e40b46de" containerName="storage-initializer"
Sep 17 22:24:33.752826 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:33.752707    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="faa98f61-fa4c-4787-a67b-1976e40b46de" containerName="storage-initializer"
Sep 17 22:24:33.752826 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:33.752723    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="faa98f61-fa4c-4787-a67b-1976e40b46de" containerName="main"
Sep 17 22:24:33.752826 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:33.752729    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="faa98f61-fa4c-4787-a67b-1976e40b46de" containerName="main"
Sep 17 22:24:33.752826 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:33.752739    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53693869-5914-4830-b952-04e2f6faed42" containerName="storage-initializer"
Sep 17 22:24:33.752826 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:33.752746    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="53693869-5914-4830-b952-04e2f6faed42" containerName="storage-initializer"
Sep 17 22:24:33.752826 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:33.752760    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53693869-5914-4830-b952-04e2f6faed42" containerName="main"
Sep 17 22:24:33.752826 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:33.752765    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="53693869-5914-4830-b952-04e2f6faed42" containerName="main"
Sep 17 22:24:33.752826 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:33.752829    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="53693869-5914-4830-b952-04e2f6faed42" containerName="main"
Sep 17 22:24:33.753331 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:33.752840    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="faa98f61-fa4c-4787-a67b-1976e40b46de" containerName="main"
Sep 17 22:24:33.758384 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:33.758363    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd"
Sep 17 22:24:33.760904 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:33.760875    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:24:33.761022 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:33.760912    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 17 22:24:33.761088 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:33.761055    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 17 22:24:33.761145 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:33.761097    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 17 22:24:33.762029 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:33.762010    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-z26p2\""
Sep 17 22:24:33.762029 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:33.762027    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 17 22:24:33.771600 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:33.771574    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd"]
Sep 17 22:24:33.781873 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:33.781851    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:24:33.878095 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:33.878049    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d514c40c-943e-4f85-a98a-82970c2e75f9-tls-certs\") pod \"tokenizer-template-args-test-kserve-5b44b88946-4zjkd\" (UID: \"d514c40c-943e-4f85-a98a-82970c2e75f9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd"
Sep 17 22:24:33.878095 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:33.878103    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d514c40c-943e-4f85-a98a-82970c2e75f9-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-5b44b88946-4zjkd\" (UID: \"d514c40c-943e-4f85-a98a-82970c2e75f9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd"
Sep 17 22:24:33.878354 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:33.878123    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d514c40c-943e-4f85-a98a-82970c2e75f9-model-cache\") pod \"tokenizer-template-args-test-kserve-5b44b88946-4zjkd\" (UID: \"d514c40c-943e-4f85-a98a-82970c2e75f9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd"
Sep 17 22:24:33.878354 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:33.878166    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d514c40c-943e-4f85-a98a-82970c2e75f9-home\") pod \"tokenizer-template-args-test-kserve-5b44b88946-4zjkd\" (UID: \"d514c40c-943e-4f85-a98a-82970c2e75f9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd"
Sep 17 22:24:33.878354 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:33.878210    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d514c40c-943e-4f85-a98a-82970c2e75f9-tmp-dir\") pod \"tokenizer-template-args-test-kserve-5b44b88946-4zjkd\" (UID: \"d514c40c-943e-4f85-a98a-82970c2e75f9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd"
Sep 17 22:24:33.878354 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:33.878252    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d514c40c-943e-4f85-a98a-82970c2e75f9-dshm\") pod \"tokenizer-template-args-test-kserve-5b44b88946-4zjkd\" (UID: \"d514c40c-943e-4f85-a98a-82970c2e75f9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd"
Sep 17 22:24:33.878354 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:33.878274    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d514c40c-943e-4f85-a98a-82970c2e75f9-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-5b44b88946-4zjkd\" (UID: \"d514c40c-943e-4f85-a98a-82970c2e75f9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd"
Sep 17 22:24:33.878354 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:33.878310    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhw54\" (UniqueName: \"kubernetes.io/projected/d514c40c-943e-4f85-a98a-82970c2e75f9-kube-api-access-mhw54\") pod \"tokenizer-template-args-test-kserve-5b44b88946-4zjkd\" (UID: \"d514c40c-943e-4f85-a98a-82970c2e75f9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd"
Sep 17 22:24:33.979252 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:33.979214    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d514c40c-943e-4f85-a98a-82970c2e75f9-tls-certs\") pod \"tokenizer-template-args-test-kserve-5b44b88946-4zjkd\" (UID: \"d514c40c-943e-4f85-a98a-82970c2e75f9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd"
Sep 17 22:24:33.979453 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:33.979261    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d514c40c-943e-4f85-a98a-82970c2e75f9-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-5b44b88946-4zjkd\" (UID: \"d514c40c-943e-4f85-a98a-82970c2e75f9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd"
Sep 17 22:24:33.979453 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:33.979282    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d514c40c-943e-4f85-a98a-82970c2e75f9-model-cache\") pod \"tokenizer-template-args-test-kserve-5b44b88946-4zjkd\" (UID: \"d514c40c-943e-4f85-a98a-82970c2e75f9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd"
Sep 17 22:24:33.979453 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:33.979309    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d514c40c-943e-4f85-a98a-82970c2e75f9-home\") pod \"tokenizer-template-args-test-kserve-5b44b88946-4zjkd\" (UID: \"d514c40c-943e-4f85-a98a-82970c2e75f9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd"
Sep 17 22:24:33.979453 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:33.979323    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d514c40c-943e-4f85-a98a-82970c2e75f9-tmp-dir\") pod \"tokenizer-template-args-test-kserve-5b44b88946-4zjkd\" (UID: \"d514c40c-943e-4f85-a98a-82970c2e75f9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd"
Sep 17 22:24:33.979453 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:33.979364    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d514c40c-943e-4f85-a98a-82970c2e75f9-dshm\") pod \"tokenizer-template-args-test-kserve-5b44b88946-4zjkd\" (UID: \"d514c40c-943e-4f85-a98a-82970c2e75f9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd"
Sep 17 22:24:33.979453 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:33.979391    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d514c40c-943e-4f85-a98a-82970c2e75f9-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-5b44b88946-4zjkd\" (UID: \"d514c40c-943e-4f85-a98a-82970c2e75f9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd"
Sep 17 22:24:33.979453 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:33.979415    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhw54\" (UniqueName: \"kubernetes.io/projected/d514c40c-943e-4f85-a98a-82970c2e75f9-kube-api-access-mhw54\") pod \"tokenizer-template-args-test-kserve-5b44b88946-4zjkd\" (UID: \"d514c40c-943e-4f85-a98a-82970c2e75f9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd"
Sep 17 22:24:33.979845 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:33.979713    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d514c40c-943e-4f85-a98a-82970c2e75f9-home\") pod \"tokenizer-template-args-test-kserve-5b44b88946-4zjkd\" (UID: \"d514c40c-943e-4f85-a98a-82970c2e75f9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd"
Sep 17 22:24:33.979845 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:33.979768    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d514c40c-943e-4f85-a98a-82970c2e75f9-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-5b44b88946-4zjkd\" (UID: \"d514c40c-943e-4f85-a98a-82970c2e75f9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd"
Sep 17 22:24:33.979962 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:33.979860    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d514c40c-943e-4f85-a98a-82970c2e75f9-tmp-dir\") pod \"tokenizer-template-args-test-kserve-5b44b88946-4zjkd\" (UID: \"d514c40c-943e-4f85-a98a-82970c2e75f9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd"
Sep 17 22:24:33.980048 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:33.980022    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d514c40c-943e-4f85-a98a-82970c2e75f9-model-cache\") pod \"tokenizer-template-args-test-kserve-5b44b88946-4zjkd\" (UID: \"d514c40c-943e-4f85-a98a-82970c2e75f9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd"
Sep 17 22:24:33.980356 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:33.980320    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d514c40c-943e-4f85-a98a-82970c2e75f9-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-5b44b88946-4zjkd\" (UID: \"d514c40c-943e-4f85-a98a-82970c2e75f9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd"
Sep 17 22:24:33.981890 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:33.981858    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d514c40c-943e-4f85-a98a-82970c2e75f9-dshm\") pod \"tokenizer-template-args-test-kserve-5b44b88946-4zjkd\" (UID: \"d514c40c-943e-4f85-a98a-82970c2e75f9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd"
Sep 17 22:24:33.982218 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:33.982168    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d514c40c-943e-4f85-a98a-82970c2e75f9-tls-certs\") pod \"tokenizer-template-args-test-kserve-5b44b88946-4zjkd\" (UID: \"d514c40c-943e-4f85-a98a-82970c2e75f9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd"
Sep 17 22:24:33.986905 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:33.986886    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhw54\" (UniqueName: \"kubernetes.io/projected/d514c40c-943e-4f85-a98a-82970c2e75f9-kube-api-access-mhw54\") pod \"tokenizer-template-args-test-kserve-5b44b88946-4zjkd\" (UID: \"d514c40c-943e-4f85-a98a-82970c2e75f9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd"
Sep 17 22:24:34.024279 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:34.024162    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm"]
Sep 17 22:24:34.029979 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:34.029957    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm"
Sep 17 22:24:34.038170 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:34.037924    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm"]
Sep 17 22:24:34.054609 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:34.054583    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:24:34.076716 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:34.076689    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd"
Sep 17 22:24:34.151738 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:34.151704    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:24:34.180849 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:34.180815    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f57bea8-04ff-4766-9594-7bb5d0ae58b1-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm\" (UID: \"9f57bea8-04ff-4766-9594-7bb5d0ae58b1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm"
Sep 17 22:24:34.180990 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:34.180857    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f57bea8-04ff-4766-9594-7bb5d0ae58b1-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm\" (UID: \"9f57bea8-04ff-4766-9594-7bb5d0ae58b1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm"
Sep 17 22:24:34.180990 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:34.180927    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f57bea8-04ff-4766-9594-7bb5d0ae58b1-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm\" (UID: \"9f57bea8-04ff-4766-9594-7bb5d0ae58b1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm"
Sep 17 22:24:34.180990 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:34.180947    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f57bea8-04ff-4766-9594-7bb5d0ae58b1-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm\" (UID: \"9f57bea8-04ff-4766-9594-7bb5d0ae58b1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm"
Sep 17 22:24:34.281562 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:34.281472    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f57bea8-04ff-4766-9594-7bb5d0ae58b1-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm\" (UID: \"9f57bea8-04ff-4766-9594-7bb5d0ae58b1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm"
Sep 17 22:24:34.281562 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:34.281515    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f57bea8-04ff-4766-9594-7bb5d0ae58b1-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm\" (UID: \"9f57bea8-04ff-4766-9594-7bb5d0ae58b1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm"
Sep 17 22:24:34.281562 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:34.281559    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f57bea8-04ff-4766-9594-7bb5d0ae58b1-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm\" (UID: \"9f57bea8-04ff-4766-9594-7bb5d0ae58b1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm"
Sep 17 22:24:34.281792 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:34.281585    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f57bea8-04ff-4766-9594-7bb5d0ae58b1-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm\" (UID: \"9f57bea8-04ff-4766-9594-7bb5d0ae58b1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm"
Sep 17 22:24:34.281920 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:34.281901    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f57bea8-04ff-4766-9594-7bb5d0ae58b1-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm\" (UID: \"9f57bea8-04ff-4766-9594-7bb5d0ae58b1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm"
Sep 17 22:24:34.281957 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:34.281944    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f57bea8-04ff-4766-9594-7bb5d0ae58b1-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm\" (UID: \"9f57bea8-04ff-4766-9594-7bb5d0ae58b1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm"
Sep 17 22:24:34.282323 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:34.282304    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f57bea8-04ff-4766-9594-7bb5d0ae58b1-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm\" (UID: \"9f57bea8-04ff-4766-9594-7bb5d0ae58b1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm"
Sep 17 22:24:34.284355 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:34.284339    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f57bea8-04ff-4766-9594-7bb5d0ae58b1-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm\" (UID: \"9f57bea8-04ff-4766-9594-7bb5d0ae58b1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm"
Sep 17 22:24:34.342434 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:34.342401    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm"
Sep 17 22:24:34.363500 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:34.363471    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd"]
Sep 17 22:24:34.365836 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:24:34.365810    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd514c40c_943e_4f85_a98a_82970c2e75f9.slice/crio-6b5742227e158a7d8c6469b4f414f7b715e0f5e2104110c26b5ef903ea5c37aa WatchSource:0}: Error finding container 6b5742227e158a7d8c6469b4f414f7b715e0f5e2104110c26b5ef903ea5c37aa: Status 404 returned error can't find the container with id 6b5742227e158a7d8c6469b4f414f7b715e0f5e2104110c26b5ef903ea5c37aa
Sep 17 22:24:34.486852 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:34.486809    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd" event={"ID":"d514c40c-943e-4f85-a98a-82970c2e75f9","Type":"ContainerStarted","Data":"844c8e4c6ef006db1563403c1e9a0ef8ea80d6ea9c3635f38b1fd25fbd671884"}
Sep 17 22:24:34.487010 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:34.486873    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd" event={"ID":"d514c40c-943e-4f85-a98a-82970c2e75f9","Type":"ContainerStarted","Data":"6b5742227e158a7d8c6469b4f414f7b715e0f5e2104110c26b5ef903ea5c37aa"}
Sep 17 22:24:34.620319 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:34.620283    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm"]
Sep 17 22:24:34.621503 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:24:34.621468    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f57bea8_04ff_4766_9594_7bb5d0ae58b1.slice/crio-87c8cb5678e68ae5f64ebd4f9f350ca523b4c0908204d036b695acc86379fd50 WatchSource:0}: Error finding container 87c8cb5678e68ae5f64ebd4f9f350ca523b4c0908204d036b695acc86379fd50: Status 404 returned error can't find the container with id 87c8cb5678e68ae5f64ebd4f9f350ca523b4c0908204d036b695acc86379fd50
Sep 17 22:24:35.498302 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:35.498266    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm" event={"ID":"9f57bea8-04ff-4766-9594-7bb5d0ae58b1","Type":"ContainerStarted","Data":"06bed17aba7d166d6d064b6bcbe7153262226d79e3bf1bd6f3ee8f2e745a8373"}
Sep 17 22:24:35.498756 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:35.498325    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm" event={"ID":"9f57bea8-04ff-4766-9594-7bb5d0ae58b1","Type":"ContainerStarted","Data":"87c8cb5678e68ae5f64ebd4f9f350ca523b4c0908204d036b695acc86379fd50"}
Sep 17 22:24:37.064251 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:37.064207    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:24:37.515854 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:37.515822    2555 generic.go:356] "Generic (PLEG): container finished" podID="d514c40c-943e-4f85-a98a-82970c2e75f9" containerID="844c8e4c6ef006db1563403c1e9a0ef8ea80d6ea9c3635f38b1fd25fbd671884" exitCode=0
Sep 17 22:24:37.516069 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:37.515899    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd" event={"ID":"d514c40c-943e-4f85-a98a-82970c2e75f9","Type":"ContainerDied","Data":"844c8e4c6ef006db1563403c1e9a0ef8ea80d6ea9c3635f38b1fd25fbd671884"}
Sep 17 22:24:37.517573 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:37.517549    2555 generic.go:356] "Generic (PLEG): container finished" podID="9f57bea8-04ff-4766-9594-7bb5d0ae58b1" containerID="06bed17aba7d166d6d064b6bcbe7153262226d79e3bf1bd6f3ee8f2e745a8373" exitCode=0
Sep 17 22:24:37.517693 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:37.517581    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm" event={"ID":"9f57bea8-04ff-4766-9594-7bb5d0ae58b1","Type":"ContainerDied","Data":"06bed17aba7d166d6d064b6bcbe7153262226d79e3bf1bd6f3ee8f2e745a8373"}
Sep 17 22:24:38.526209 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:38.526136    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm" event={"ID":"9f57bea8-04ff-4766-9594-7bb5d0ae58b1","Type":"ContainerStarted","Data":"b3597bb69fa0255eb6063f90ce4d2212f0da81c11477372dbf3c5da207425524"}
Sep 17 22:24:38.526673 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:38.526480    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm"
Sep 17 22:24:38.528161 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:38.528132    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd" event={"ID":"d514c40c-943e-4f85-a98a-82970c2e75f9","Type":"ContainerStarted","Data":"59e4e70cd642116f00889e43dcef0f0e42bd1c58c3fe798301ef49dd7bd8ba43"}
Sep 17 22:24:38.543779 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:38.543728    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm" podStartSLOduration=1.9216884749999998 podStartE2EDuration="4.543713045s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.62202457s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:24:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:24:38.542225787 +0000 UTC m=+1579.988271056" watchObservedRunningTime="2026-09-17 22:24:38.543713045 +0000 UTC m=+1579.989758317"
Sep 17 22:24:38.560740 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:38.560691    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd" podStartSLOduration=2.996396298 podStartE2EDuration="5.560676072s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.564279774s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:24:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:24:38.558629746 +0000 UTC m=+1580.004675056" watchObservedRunningTime="2026-09-17 22:24:38.560676072 +0000 UTC m=+1580.006721343"
Sep 17 22:24:39.077132 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:39.077086    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd"
Sep 17 22:24:42.534569 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:42.534533    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:24:42.637060 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:42.637030    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:24:43.602196 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:43.602143    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd" podUID="0e99380b-b274-458a-b655-6df2a18a8e49" containerName="main" containerID="cri-o://82f821198a58403f7fb1bfb296136c8bda64d8b0b080b24673959a7b20c72e0a" gracePeriod=45
Sep 17 22:24:43.676679 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:43.676629    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:24:44.058511 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:44.058467    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd" podUID="0e99380b-b274-458a-b655-6df2a18a8e49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 17 22:24:44.077540 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:44.077501    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd"
Sep 17 22:24:44.091081 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:44.091043    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd"
Sep 17 22:24:44.365754 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:44.365729    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd"
Sep 17 22:24:44.381362 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:44.381337    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e99380b-b274-458a-b655-6df2a18a8e49-cabundle-cert\") pod \"0e99380b-b274-458a-b655-6df2a18a8e49\" (UID: \"0e99380b-b274-458a-b655-6df2a18a8e49\") "
Sep 17 22:24:44.381510 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:44.381403    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e99380b-b274-458a-b655-6df2a18a8e49-tls-certs\") pod \"0e99380b-b274-458a-b655-6df2a18a8e49\" (UID: \"0e99380b-b274-458a-b655-6df2a18a8e49\") "
Sep 17 22:24:44.381510 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:44.381428    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e99380b-b274-458a-b655-6df2a18a8e49-model-cache\") pod \"0e99380b-b274-458a-b655-6df2a18a8e49\" (UID: \"0e99380b-b274-458a-b655-6df2a18a8e49\") "
Sep 17 22:24:44.381510 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:44.381458    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e99380b-b274-458a-b655-6df2a18a8e49-tmp-dir\") pod \"0e99380b-b274-458a-b655-6df2a18a8e49\" (UID: \"0e99380b-b274-458a-b655-6df2a18a8e49\") "
Sep 17 22:24:44.381510 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:44.381491    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e99380b-b274-458a-b655-6df2a18a8e49-home\") pod \"0e99380b-b274-458a-b655-6df2a18a8e49\" (UID: \"0e99380b-b274-458a-b655-6df2a18a8e49\") "
Sep 17 22:24:44.381759 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:44.381530    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mp87l\" (UniqueName: \"kubernetes.io/projected/0e99380b-b274-458a-b655-6df2a18a8e49-kube-api-access-mp87l\") pod \"0e99380b-b274-458a-b655-6df2a18a8e49\" (UID: \"0e99380b-b274-458a-b655-6df2a18a8e49\") "
Sep 17 22:24:44.381759 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:44.381571    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e99380b-b274-458a-b655-6df2a18a8e49-kserve-provision-location\") pod \"0e99380b-b274-458a-b655-6df2a18a8e49\" (UID: \"0e99380b-b274-458a-b655-6df2a18a8e49\") "
Sep 17 22:24:44.381759 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:44.381610    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e99380b-b274-458a-b655-6df2a18a8e49-dshm\") pod \"0e99380b-b274-458a-b655-6df2a18a8e49\" (UID: \"0e99380b-b274-458a-b655-6df2a18a8e49\") "
Sep 17 22:24:44.381759 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:44.381646    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e99380b-b274-458a-b655-6df2a18a8e49-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0e99380b-b274-458a-b655-6df2a18a8e49" (UID: "0e99380b-b274-458a-b655-6df2a18a8e49"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:24:44.382045 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:44.381929    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e99380b-b274-458a-b655-6df2a18a8e49-model-cache" (OuterVolumeSpecName: "model-cache") pod "0e99380b-b274-458a-b655-6df2a18a8e49" (UID: "0e99380b-b274-458a-b655-6df2a18a8e49"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:24:44.382045 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:44.381973    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e99380b-b274-458a-b655-6df2a18a8e49-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:24:44.382312 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:44.382282    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e99380b-b274-458a-b655-6df2a18a8e49-home" (OuterVolumeSpecName: "home") pod "0e99380b-b274-458a-b655-6df2a18a8e49" (UID: "0e99380b-b274-458a-b655-6df2a18a8e49"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:24:44.383905 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:44.383874    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e99380b-b274-458a-b655-6df2a18a8e49-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0e99380b-b274-458a-b655-6df2a18a8e49" (UID: "0e99380b-b274-458a-b655-6df2a18a8e49"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:24:44.384017 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:44.383885    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e99380b-b274-458a-b655-6df2a18a8e49-kube-api-access-mp87l" (OuterVolumeSpecName: "kube-api-access-mp87l") pod "0e99380b-b274-458a-b655-6df2a18a8e49" (UID: "0e99380b-b274-458a-b655-6df2a18a8e49"). InnerVolumeSpecName "kube-api-access-mp87l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:24:44.384749 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:44.384702    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e99380b-b274-458a-b655-6df2a18a8e49-dshm" (OuterVolumeSpecName: "dshm") pod "0e99380b-b274-458a-b655-6df2a18a8e49" (UID: "0e99380b-b274-458a-b655-6df2a18a8e49"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:24:44.402415 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:44.402374    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e99380b-b274-458a-b655-6df2a18a8e49-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0e99380b-b274-458a-b655-6df2a18a8e49" (UID: "0e99380b-b274-458a-b655-6df2a18a8e49"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:24:44.422851 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:44.422784    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e99380b-b274-458a-b655-6df2a18a8e49-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0e99380b-b274-458a-b655-6df2a18a8e49" (UID: "0e99380b-b274-458a-b655-6df2a18a8e49"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:24:44.482659 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:44.482627    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e99380b-b274-458a-b655-6df2a18a8e49-tls-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:24:44.482823 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:44.482674    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e99380b-b274-458a-b655-6df2a18a8e49-model-cache\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:24:44.482823 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:44.482683    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e99380b-b274-458a-b655-6df2a18a8e49-tmp-dir\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:24:44.482823 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:44.482691    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e99380b-b274-458a-b655-6df2a18a8e49-home\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:24:44.482823 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:44.482700    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp87l\" (UniqueName: \"kubernetes.io/projected/0e99380b-b274-458a-b655-6df2a18a8e49-kube-api-access-mp87l\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:24:44.482823 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:44.482709    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e99380b-b274-458a-b655-6df2a18a8e49-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:24:44.482823 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:44.482717    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e99380b-b274-458a-b655-6df2a18a8e49-dshm\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:24:44.567400 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:44.567359    2555 generic.go:356] "Generic (PLEG): container finished" podID="0e99380b-b274-458a-b655-6df2a18a8e49" containerID="82f821198a58403f7fb1bfb296136c8bda64d8b0b080b24673959a7b20c72e0a" exitCode=0
Sep 17 22:24:44.567582 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:44.567434    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd"
Sep 17 22:24:44.567582 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:44.567444    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd" event={"ID":"0e99380b-b274-458a-b655-6df2a18a8e49","Type":"ContainerDied","Data":"82f821198a58403f7fb1bfb296136c8bda64d8b0b080b24673959a7b20c72e0a"}
Sep 17 22:24:44.567582 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:44.567537    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd" event={"ID":"0e99380b-b274-458a-b655-6df2a18a8e49","Type":"ContainerDied","Data":"2f79318e90a6386348a013095f8ee4ad97cf6d8791cad2595d75ce6effd5730c"}
Sep 17 22:24:44.567582 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:44.567561    2555 scope.go:117] "RemoveContainer" containerID="82f821198a58403f7fb1bfb296136c8bda64d8b0b080b24673959a7b20c72e0a"
Sep 17 22:24:44.579483 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:44.579460    2555 scope.go:117] "RemoveContainer" containerID="5a82ea6811ab480fe5599b77edbea1f58e0706d370c2fd539794373f97d2039a"
Sep 17 22:24:44.581554 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:44.581522    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd"
Sep 17 22:24:44.588708 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:44.588691    2555 scope.go:117] "RemoveContainer" containerID="82f821198a58403f7fb1bfb296136c8bda64d8b0b080b24673959a7b20c72e0a"
Sep 17 22:24:44.589048 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:24:44.589024    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82f821198a58403f7fb1bfb296136c8bda64d8b0b080b24673959a7b20c72e0a\": container with ID starting with 82f821198a58403f7fb1bfb296136c8bda64d8b0b080b24673959a7b20c72e0a not found: ID does not exist" containerID="82f821198a58403f7fb1bfb296136c8bda64d8b0b080b24673959a7b20c72e0a"
Sep 17 22:24:44.589151 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:44.589060    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82f821198a58403f7fb1bfb296136c8bda64d8b0b080b24673959a7b20c72e0a"} err="failed to get container status \"82f821198a58403f7fb1bfb296136c8bda64d8b0b080b24673959a7b20c72e0a\": rpc error: code = NotFound desc = could not find container \"82f821198a58403f7fb1bfb296136c8bda64d8b0b080b24673959a7b20c72e0a\": container with ID starting with 82f821198a58403f7fb1bfb296136c8bda64d8b0b080b24673959a7b20c72e0a not found: ID does not exist"
Sep 17 22:24:44.589151 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:44.589084    2555 scope.go:117] "RemoveContainer" containerID="5a82ea6811ab480fe5599b77edbea1f58e0706d370c2fd539794373f97d2039a"
Sep 17 22:24:44.589423 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:24:44.589400    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a82ea6811ab480fe5599b77edbea1f58e0706d370c2fd539794373f97d2039a\": container with ID starting with 5a82ea6811ab480fe5599b77edbea1f58e0706d370c2fd539794373f97d2039a not found: ID does not exist" containerID="5a82ea6811ab480fe5599b77edbea1f58e0706d370c2fd539794373f97d2039a"
Sep 17 22:24:44.589473 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:44.589433    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a82ea6811ab480fe5599b77edbea1f58e0706d370c2fd539794373f97d2039a"} err="failed to get container status \"5a82ea6811ab480fe5599b77edbea1f58e0706d370c2fd539794373f97d2039a\": rpc error: code = NotFound desc = could not find container \"5a82ea6811ab480fe5599b77edbea1f58e0706d370c2fd539794373f97d2039a\": container with ID starting with 5a82ea6811ab480fe5599b77edbea1f58e0706d370c2fd539794373f97d2039a not found: ID does not exist"
Sep 17 22:24:44.594253 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:44.594167    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd"]
Sep 17 22:24:44.597472 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:44.597451    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-585fcb4787-d6kbd"]
Sep 17 22:24:44.680786 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:24:44.680749    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e99380b-b274-458a-b655-6df2a18a8e49" path="/var/lib/kubelet/pods/0e99380b-b274-458a-b655-6df2a18a8e49/volumes"
Sep 17 22:25:09.547132 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:09.547045    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm"
Sep 17 22:25:10.551474 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:10.551444    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-0eb24714_tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm_9f57bea8-04ff-4766-9594-7bb5d0ae58b1/main/0.log"
Sep 17 22:25:11.758948 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:11.758918    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:25:11.765974 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:11.765946    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm"]
Sep 17 22:25:11.766232 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:11.766199    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm" podUID="9f57bea8-04ff-4766-9594-7bb5d0ae58b1" containerName="main" containerID="cri-o://b3597bb69fa0255eb6063f90ce4d2212f0da81c11477372dbf3c5da207425524" gracePeriod=30
Sep 17 22:25:11.773726 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:11.773693    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd"]
Sep 17 22:25:11.784207 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:25:11.784137    2555 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 22:25:11.784207 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:25:11.784207 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:25:11.784207 ip-10-0-140-157 kubenswrapper[2555]:  > containerID="59e4e70cd642116f00889e43dcef0f0e42bd1c58c3fe798301ef49dd7bd8ba43" cmd=["/bin/sleep","15"]
Sep 17 22:25:11.784516 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:11.784222    2555 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd" message="" err=<
Sep 17 22:25:11.784516 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:25:11.784516 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:25:11.784516 ip-10-0-140-157 kubenswrapper[2555]:  >
Sep 17 22:25:11.784516 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:25:11.784236    2555 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 22:25:11.784516 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:25:11.784516 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:25:11.784516 ip-10-0-140-157 kubenswrapper[2555]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd" podUID="d514c40c-943e-4f85-a98a-82970c2e75f9" containerName="main" containerID="cri-o://59e4e70cd642116f00889e43dcef0f0e42bd1c58c3fe798301ef49dd7bd8ba43"
Sep 17 22:25:11.784516 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:11.784278    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd" podUID="d514c40c-943e-4f85-a98a-82970c2e75f9" containerName="main" containerID="cri-o://59e4e70cd642116f00889e43dcef0f0e42bd1c58c3fe798301ef49dd7bd8ba43" gracePeriod=60
Sep 17 22:25:11.831221 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:11.831191    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:25:12.136586 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.136563    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd"
Sep 17 22:25:12.225455 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.225415    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d514c40c-943e-4f85-a98a-82970c2e75f9-tmp-dir\") pod \"d514c40c-943e-4f85-a98a-82970c2e75f9\" (UID: \"d514c40c-943e-4f85-a98a-82970c2e75f9\") "
Sep 17 22:25:12.225693 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.225471    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d514c40c-943e-4f85-a98a-82970c2e75f9-tls-certs\") pod \"d514c40c-943e-4f85-a98a-82970c2e75f9\" (UID: \"d514c40c-943e-4f85-a98a-82970c2e75f9\") "
Sep 17 22:25:12.225693 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.225513    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d514c40c-943e-4f85-a98a-82970c2e75f9-kserve-provision-location\") pod \"d514c40c-943e-4f85-a98a-82970c2e75f9\" (UID: \"d514c40c-943e-4f85-a98a-82970c2e75f9\") "
Sep 17 22:25:12.225693 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.225533    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d514c40c-943e-4f85-a98a-82970c2e75f9-dshm\") pod \"d514c40c-943e-4f85-a98a-82970c2e75f9\" (UID: \"d514c40c-943e-4f85-a98a-82970c2e75f9\") "
Sep 17 22:25:12.225693 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.225598    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d514c40c-943e-4f85-a98a-82970c2e75f9-home\") pod \"d514c40c-943e-4f85-a98a-82970c2e75f9\" (UID: \"d514c40c-943e-4f85-a98a-82970c2e75f9\") "
Sep 17 22:25:12.225693 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.225634    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhw54\" (UniqueName: \"kubernetes.io/projected/d514c40c-943e-4f85-a98a-82970c2e75f9-kube-api-access-mhw54\") pod \"d514c40c-943e-4f85-a98a-82970c2e75f9\" (UID: \"d514c40c-943e-4f85-a98a-82970c2e75f9\") "
Sep 17 22:25:12.225693 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.225689    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d514c40c-943e-4f85-a98a-82970c2e75f9-model-cache\") pod \"d514c40c-943e-4f85-a98a-82970c2e75f9\" (UID: \"d514c40c-943e-4f85-a98a-82970c2e75f9\") "
Sep 17 22:25:12.226106 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.225729    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d514c40c-943e-4f85-a98a-82970c2e75f9-cabundle-cert\") pod \"d514c40c-943e-4f85-a98a-82970c2e75f9\" (UID: \"d514c40c-943e-4f85-a98a-82970c2e75f9\") "
Sep 17 22:25:12.226106 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.225725    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d514c40c-943e-4f85-a98a-82970c2e75f9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d514c40c-943e-4f85-a98a-82970c2e75f9" (UID: "d514c40c-943e-4f85-a98a-82970c2e75f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:25:12.226106 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.225968    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d514c40c-943e-4f85-a98a-82970c2e75f9-tmp-dir\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:25:12.226297 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.226104    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d514c40c-943e-4f85-a98a-82970c2e75f9-model-cache" (OuterVolumeSpecName: "model-cache") pod "d514c40c-943e-4f85-a98a-82970c2e75f9" (UID: "d514c40c-943e-4f85-a98a-82970c2e75f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:25:12.226297 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.226227    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d514c40c-943e-4f85-a98a-82970c2e75f9-home" (OuterVolumeSpecName: "home") pod "d514c40c-943e-4f85-a98a-82970c2e75f9" (UID: "d514c40c-943e-4f85-a98a-82970c2e75f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:25:12.226391 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.226322    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d514c40c-943e-4f85-a98a-82970c2e75f9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d514c40c-943e-4f85-a98a-82970c2e75f9" (UID: "d514c40c-943e-4f85-a98a-82970c2e75f9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:25:12.228453 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.228418    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d514c40c-943e-4f85-a98a-82970c2e75f9-dshm" (OuterVolumeSpecName: "dshm") pod "d514c40c-943e-4f85-a98a-82970c2e75f9" (UID: "d514c40c-943e-4f85-a98a-82970c2e75f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:25:12.228568 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.228497    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d514c40c-943e-4f85-a98a-82970c2e75f9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d514c40c-943e-4f85-a98a-82970c2e75f9" (UID: "d514c40c-943e-4f85-a98a-82970c2e75f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:25:12.228635 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.228611    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d514c40c-943e-4f85-a98a-82970c2e75f9-kube-api-access-mhw54" (OuterVolumeSpecName: "kube-api-access-mhw54") pod "d514c40c-943e-4f85-a98a-82970c2e75f9" (UID: "d514c40c-943e-4f85-a98a-82970c2e75f9"). InnerVolumeSpecName "kube-api-access-mhw54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:25:12.247091 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.247068    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d514c40c-943e-4f85-a98a-82970c2e75f9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d514c40c-943e-4f85-a98a-82970c2e75f9" (UID: "d514c40c-943e-4f85-a98a-82970c2e75f9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:25:12.327120 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.327045    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d514c40c-943e-4f85-a98a-82970c2e75f9-home\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:25:12.327120 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.327073    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhw54\" (UniqueName: \"kubernetes.io/projected/d514c40c-943e-4f85-a98a-82970c2e75f9-kube-api-access-mhw54\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:25:12.327120 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.327083    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d514c40c-943e-4f85-a98a-82970c2e75f9-model-cache\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:25:12.327120 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.327091    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d514c40c-943e-4f85-a98a-82970c2e75f9-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:25:12.327120 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.327100    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d514c40c-943e-4f85-a98a-82970c2e75f9-tls-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:25:12.327120 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.327108    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d514c40c-943e-4f85-a98a-82970c2e75f9-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:25:12.327120 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.327118    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d514c40c-943e-4f85-a98a-82970c2e75f9-dshm\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:25:12.405658 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.405632    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm"
Sep 17 22:25:12.529724 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.529690    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f57bea8-04ff-4766-9594-7bb5d0ae58b1-cabundle-cert\") pod \"9f57bea8-04ff-4766-9594-7bb5d0ae58b1\" (UID: \"9f57bea8-04ff-4766-9594-7bb5d0ae58b1\") "
Sep 17 22:25:12.529887 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.529737    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f57bea8-04ff-4766-9594-7bb5d0ae58b1-kserve-provision-location\") pod \"9f57bea8-04ff-4766-9594-7bb5d0ae58b1\" (UID: \"9f57bea8-04ff-4766-9594-7bb5d0ae58b1\") "
Sep 17 22:25:12.529887 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.529796    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f57bea8-04ff-4766-9594-7bb5d0ae58b1-tmp-dir\") pod \"9f57bea8-04ff-4766-9594-7bb5d0ae58b1\" (UID: \"9f57bea8-04ff-4766-9594-7bb5d0ae58b1\") "
Sep 17 22:25:12.529887 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.529882    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f57bea8-04ff-4766-9594-7bb5d0ae58b1-tls-certs\") pod \"9f57bea8-04ff-4766-9594-7bb5d0ae58b1\" (UID: \"9f57bea8-04ff-4766-9594-7bb5d0ae58b1\") "
Sep 17 22:25:12.530205 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.530171    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f57bea8-04ff-4766-9594-7bb5d0ae58b1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9f57bea8-04ff-4766-9594-7bb5d0ae58b1" (UID: "9f57bea8-04ff-4766-9594-7bb5d0ae58b1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:25:12.530409 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.530384    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f57bea8-04ff-4766-9594-7bb5d0ae58b1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9f57bea8-04ff-4766-9594-7bb5d0ae58b1" (UID: "9f57bea8-04ff-4766-9594-7bb5d0ae58b1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:25:12.532044 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.532025    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f57bea8-04ff-4766-9594-7bb5d0ae58b1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9f57bea8-04ff-4766-9594-7bb5d0ae58b1" (UID: "9f57bea8-04ff-4766-9594-7bb5d0ae58b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:25:12.552374 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.552337    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f57bea8-04ff-4766-9594-7bb5d0ae58b1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9f57bea8-04ff-4766-9594-7bb5d0ae58b1" (UID: "9f57bea8-04ff-4766-9594-7bb5d0ae58b1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:25:12.631458 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.631430    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f57bea8-04ff-4766-9594-7bb5d0ae58b1-tls-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:25:12.631458 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.631457    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f57bea8-04ff-4766-9594-7bb5d0ae58b1-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:25:12.631621 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.631470    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f57bea8-04ff-4766-9594-7bb5d0ae58b1-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:25:12.631621 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.631484    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f57bea8-04ff-4766-9594-7bb5d0ae58b1-tmp-dir\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:25:12.749155 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.749121    2555 generic.go:356] "Generic (PLEG): container finished" podID="d514c40c-943e-4f85-a98a-82970c2e75f9" containerID="59e4e70cd642116f00889e43dcef0f0e42bd1c58c3fe798301ef49dd7bd8ba43" exitCode=0
Sep 17 22:25:12.749312 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.749209    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd"
Sep 17 22:25:12.749312 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.749208    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd" event={"ID":"d514c40c-943e-4f85-a98a-82970c2e75f9","Type":"ContainerDied","Data":"59e4e70cd642116f00889e43dcef0f0e42bd1c58c3fe798301ef49dd7bd8ba43"}
Sep 17 22:25:12.749312 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.749268    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd" event={"ID":"d514c40c-943e-4f85-a98a-82970c2e75f9","Type":"ContainerDied","Data":"6b5742227e158a7d8c6469b4f414f7b715e0f5e2104110c26b5ef903ea5c37aa"}
Sep 17 22:25:12.749312 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.749291    2555 scope.go:117] "RemoveContainer" containerID="59e4e70cd642116f00889e43dcef0f0e42bd1c58c3fe798301ef49dd7bd8ba43"
Sep 17 22:25:12.751525 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.751492    2555 generic.go:356] "Generic (PLEG): container finished" podID="9f57bea8-04ff-4766-9594-7bb5d0ae58b1" containerID="b3597bb69fa0255eb6063f90ce4d2212f0da81c11477372dbf3c5da207425524" exitCode=0
Sep 17 22:25:12.751635 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.751551    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm" event={"ID":"9f57bea8-04ff-4766-9594-7bb5d0ae58b1","Type":"ContainerDied","Data":"b3597bb69fa0255eb6063f90ce4d2212f0da81c11477372dbf3c5da207425524"}
Sep 17 22:25:12.751635 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.751595    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm" event={"ID":"9f57bea8-04ff-4766-9594-7bb5d0ae58b1","Type":"ContainerDied","Data":"87c8cb5678e68ae5f64ebd4f9f350ca523b4c0908204d036b695acc86379fd50"}
Sep 17 22:25:12.751635 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.751592    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm"
Sep 17 22:25:12.759093 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.759077    2555 scope.go:117] "RemoveContainer" containerID="844c8e4c6ef006db1563403c1e9a0ef8ea80d6ea9c3635f38b1fd25fbd671884"
Sep 17 22:25:12.767681 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.767661    2555 scope.go:117] "RemoveContainer" containerID="59e4e70cd642116f00889e43dcef0f0e42bd1c58c3fe798301ef49dd7bd8ba43"
Sep 17 22:25:12.767950 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:25:12.767930    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59e4e70cd642116f00889e43dcef0f0e42bd1c58c3fe798301ef49dd7bd8ba43\": container with ID starting with 59e4e70cd642116f00889e43dcef0f0e42bd1c58c3fe798301ef49dd7bd8ba43 not found: ID does not exist" containerID="59e4e70cd642116f00889e43dcef0f0e42bd1c58c3fe798301ef49dd7bd8ba43"
Sep 17 22:25:12.768017 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.767956    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59e4e70cd642116f00889e43dcef0f0e42bd1c58c3fe798301ef49dd7bd8ba43"} err="failed to get container status \"59e4e70cd642116f00889e43dcef0f0e42bd1c58c3fe798301ef49dd7bd8ba43\": rpc error: code = NotFound desc = could not find container \"59e4e70cd642116f00889e43dcef0f0e42bd1c58c3fe798301ef49dd7bd8ba43\": container with ID starting with 59e4e70cd642116f00889e43dcef0f0e42bd1c58c3fe798301ef49dd7bd8ba43 not found: ID does not exist"
Sep 17 22:25:12.768017 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.767972    2555 scope.go:117] "RemoveContainer" containerID="844c8e4c6ef006db1563403c1e9a0ef8ea80d6ea9c3635f38b1fd25fbd671884"
Sep 17 22:25:12.768212 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:25:12.768174    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"844c8e4c6ef006db1563403c1e9a0ef8ea80d6ea9c3635f38b1fd25fbd671884\": container with ID starting with 844c8e4c6ef006db1563403c1e9a0ef8ea80d6ea9c3635f38b1fd25fbd671884 not found: ID does not exist" containerID="844c8e4c6ef006db1563403c1e9a0ef8ea80d6ea9c3635f38b1fd25fbd671884"
Sep 17 22:25:12.768273 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.768219    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"844c8e4c6ef006db1563403c1e9a0ef8ea80d6ea9c3635f38b1fd25fbd671884"} err="failed to get container status \"844c8e4c6ef006db1563403c1e9a0ef8ea80d6ea9c3635f38b1fd25fbd671884\": rpc error: code = NotFound desc = could not find container \"844c8e4c6ef006db1563403c1e9a0ef8ea80d6ea9c3635f38b1fd25fbd671884\": container with ID starting with 844c8e4c6ef006db1563403c1e9a0ef8ea80d6ea9c3635f38b1fd25fbd671884 not found: ID does not exist"
Sep 17 22:25:12.768273 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.768234    2555 scope.go:117] "RemoveContainer" containerID="b3597bb69fa0255eb6063f90ce4d2212f0da81c11477372dbf3c5da207425524"
Sep 17 22:25:12.770253 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.770231    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd"]
Sep 17 22:25:12.776108 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.776088    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-4zjkd"]
Sep 17 22:25:12.776628 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.776613    2555 scope.go:117] "RemoveContainer" containerID="06bed17aba7d166d6d064b6bcbe7153262226d79e3bf1bd6f3ee8f2e745a8373"
Sep 17 22:25:12.784781 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.784761    2555 scope.go:117] "RemoveContainer" containerID="b3597bb69fa0255eb6063f90ce4d2212f0da81c11477372dbf3c5da207425524"
Sep 17 22:25:12.785048 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:25:12.785028    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3597bb69fa0255eb6063f90ce4d2212f0da81c11477372dbf3c5da207425524\": container with ID starting with b3597bb69fa0255eb6063f90ce4d2212f0da81c11477372dbf3c5da207425524 not found: ID does not exist" containerID="b3597bb69fa0255eb6063f90ce4d2212f0da81c11477372dbf3c5da207425524"
Sep 17 22:25:12.785125 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.785058    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3597bb69fa0255eb6063f90ce4d2212f0da81c11477372dbf3c5da207425524"} err="failed to get container status \"b3597bb69fa0255eb6063f90ce4d2212f0da81c11477372dbf3c5da207425524\": rpc error: code = NotFound desc = could not find container \"b3597bb69fa0255eb6063f90ce4d2212f0da81c11477372dbf3c5da207425524\": container with ID starting with b3597bb69fa0255eb6063f90ce4d2212f0da81c11477372dbf3c5da207425524 not found: ID does not exist"
Sep 17 22:25:12.785125 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.785084    2555 scope.go:117] "RemoveContainer" containerID="06bed17aba7d166d6d064b6bcbe7153262226d79e3bf1bd6f3ee8f2e745a8373"
Sep 17 22:25:12.785354 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:25:12.785336    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06bed17aba7d166d6d064b6bcbe7153262226d79e3bf1bd6f3ee8f2e745a8373\": container with ID starting with 06bed17aba7d166d6d064b6bcbe7153262226d79e3bf1bd6f3ee8f2e745a8373 not found: ID does not exist" containerID="06bed17aba7d166d6d064b6bcbe7153262226d79e3bf1bd6f3ee8f2e745a8373"
Sep 17 22:25:12.785402 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.785359    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06bed17aba7d166d6d064b6bcbe7153262226d79e3bf1bd6f3ee8f2e745a8373"} err="failed to get container status \"06bed17aba7d166d6d064b6bcbe7153262226d79e3bf1bd6f3ee8f2e745a8373\": rpc error: code = NotFound desc = could not find container \"06bed17aba7d166d6d064b6bcbe7153262226d79e3bf1bd6f3ee8f2e745a8373\": container with ID starting with 06bed17aba7d166d6d064b6bcbe7153262226d79e3bf1bd6f3ee8f2e745a8373 not found: ID does not exist"
Sep 17 22:25:12.785931 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.785915    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm"]
Sep 17 22:25:12.789721 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:12.789691    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f4bc7784-7nxpm"]
Sep 17 22:25:14.679218 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:14.679166    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f57bea8-04ff-4766-9594-7bb5d0ae58b1" path="/var/lib/kubelet/pods/9f57bea8-04ff-4766-9594-7bb5d0ae58b1/volumes"
Sep 17 22:25:14.679673 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:14.679657    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d514c40c-943e-4f85-a98a-82970c2e75f9" path="/var/lib/kubelet/pods/d514c40c-943e-4f85-a98a-82970c2e75f9/volumes"
Sep 17 22:25:18.516615 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:18.516579    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:25:18.611119 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:18.611086    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:25:23.860913 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:23.860880    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x"]
Sep 17 22:25:23.861285 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:23.861273    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e99380b-b274-458a-b655-6df2a18a8e49" containerName="main"
Sep 17 22:25:23.861285 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:23.861286    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e99380b-b274-458a-b655-6df2a18a8e49" containerName="main"
Sep 17 22:25:23.861361 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:23.861296    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d514c40c-943e-4f85-a98a-82970c2e75f9" containerName="main"
Sep 17 22:25:23.861361 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:23.861302    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d514c40c-943e-4f85-a98a-82970c2e75f9" containerName="main"
Sep 17 22:25:23.861361 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:23.861310    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d514c40c-943e-4f85-a98a-82970c2e75f9" containerName="storage-initializer"
Sep 17 22:25:23.861361 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:23.861316    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d514c40c-943e-4f85-a98a-82970c2e75f9" containerName="storage-initializer"
Sep 17 22:25:23.861361 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:23.861326    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e99380b-b274-458a-b655-6df2a18a8e49" containerName="storage-initializer"
Sep 17 22:25:23.861361 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:23.861331    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e99380b-b274-458a-b655-6df2a18a8e49" containerName="storage-initializer"
Sep 17 22:25:23.861361 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:23.861341    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f57bea8-04ff-4766-9594-7bb5d0ae58b1" containerName="main"
Sep 17 22:25:23.861361 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:23.861346    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f57bea8-04ff-4766-9594-7bb5d0ae58b1" containerName="main"
Sep 17 22:25:23.861361 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:23.861358    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f57bea8-04ff-4766-9594-7bb5d0ae58b1" containerName="storage-initializer"
Sep 17 22:25:23.861361 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:23.861363    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f57bea8-04ff-4766-9594-7bb5d0ae58b1" containerName="storage-initializer"
Sep 17 22:25:23.861691 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:23.861427    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e99380b-b274-458a-b655-6df2a18a8e49" containerName="main"
Sep 17 22:25:23.861691 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:23.861433    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f57bea8-04ff-4766-9594-7bb5d0ae58b1" containerName="main"
Sep 17 22:25:23.861691 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:23.861440    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d514c40c-943e-4f85-a98a-82970c2e75f9" containerName="main"
Sep 17 22:25:23.866065 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:23.866047    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x"
Sep 17 22:25:23.868758 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:23.868737    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Sep 17 22:25:23.868852 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:23.868738    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 17 22:25:23.868852 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:23.868739    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 17 22:25:23.869453 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:23.869432    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-6lqz5\""
Sep 17 22:25:23.869569 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:23.869482    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:25:23.869569 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:23.869503    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 17 22:25:23.876997 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:23.876976    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x"]
Sep 17 22:25:23.891680 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:23.891647    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:25:24.027744 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:24.027705    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efc28864-841e-4451-bfce-5817940b27d5-dshm\") pod \"auth-disabled-test-kserve-6448cc6bb8-x7z9x\" (UID: \"efc28864-841e-4451-bfce-5817940b27d5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x"
Sep 17 22:25:24.027921 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:24.027753    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efc28864-841e-4451-bfce-5817940b27d5-tmp-dir\") pod \"auth-disabled-test-kserve-6448cc6bb8-x7z9x\" (UID: \"efc28864-841e-4451-bfce-5817940b27d5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x"
Sep 17 22:25:24.027921 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:24.027819    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efc28864-841e-4451-bfce-5817940b27d5-model-cache\") pod \"auth-disabled-test-kserve-6448cc6bb8-x7z9x\" (UID: \"efc28864-841e-4451-bfce-5817940b27d5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x"
Sep 17 22:25:24.027921 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:24.027838    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efc28864-841e-4451-bfce-5817940b27d5-kserve-provision-location\") pod \"auth-disabled-test-kserve-6448cc6bb8-x7z9x\" (UID: \"efc28864-841e-4451-bfce-5817940b27d5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x"
Sep 17 22:25:24.027921 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:24.027868    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thhxp\" (UniqueName: \"kubernetes.io/projected/efc28864-841e-4451-bfce-5817940b27d5-kube-api-access-thhxp\") pod \"auth-disabled-test-kserve-6448cc6bb8-x7z9x\" (UID: \"efc28864-841e-4451-bfce-5817940b27d5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x"
Sep 17 22:25:24.028113 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:24.027938    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efc28864-841e-4451-bfce-5817940b27d5-cabundle-cert\") pod \"auth-disabled-test-kserve-6448cc6bb8-x7z9x\" (UID: \"efc28864-841e-4451-bfce-5817940b27d5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x"
Sep 17 22:25:24.028113 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:24.028092    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efc28864-841e-4451-bfce-5817940b27d5-home\") pod \"auth-disabled-test-kserve-6448cc6bb8-x7z9x\" (UID: \"efc28864-841e-4451-bfce-5817940b27d5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x"
Sep 17 22:25:24.028253 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:24.028117    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efc28864-841e-4451-bfce-5817940b27d5-tls-certs\") pod \"auth-disabled-test-kserve-6448cc6bb8-x7z9x\" (UID: \"efc28864-841e-4451-bfce-5817940b27d5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x"
Sep 17 22:25:24.116638 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:24.116546    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:25:24.128805 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:24.128764    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efc28864-841e-4451-bfce-5817940b27d5-cabundle-cert\") pod \"auth-disabled-test-kserve-6448cc6bb8-x7z9x\" (UID: \"efc28864-841e-4451-bfce-5817940b27d5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x"
Sep 17 22:25:24.128948 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:24.128843    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efc28864-841e-4451-bfce-5817940b27d5-home\") pod \"auth-disabled-test-kserve-6448cc6bb8-x7z9x\" (UID: \"efc28864-841e-4451-bfce-5817940b27d5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x"
Sep 17 22:25:24.128948 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:24.128862    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efc28864-841e-4451-bfce-5817940b27d5-tls-certs\") pod \"auth-disabled-test-kserve-6448cc6bb8-x7z9x\" (UID: \"efc28864-841e-4451-bfce-5817940b27d5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x"
Sep 17 22:25:24.128948 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:24.128888    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efc28864-841e-4451-bfce-5817940b27d5-dshm\") pod \"auth-disabled-test-kserve-6448cc6bb8-x7z9x\" (UID: \"efc28864-841e-4451-bfce-5817940b27d5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x"
Sep 17 22:25:24.128948 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:24.128903    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efc28864-841e-4451-bfce-5817940b27d5-tmp-dir\") pod \"auth-disabled-test-kserve-6448cc6bb8-x7z9x\" (UID: \"efc28864-841e-4451-bfce-5817940b27d5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x"
Sep 17 22:25:24.129112 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:24.129025    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efc28864-841e-4451-bfce-5817940b27d5-model-cache\") pod \"auth-disabled-test-kserve-6448cc6bb8-x7z9x\" (UID: \"efc28864-841e-4451-bfce-5817940b27d5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x"
Sep 17 22:25:24.129112 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:24.129059    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efc28864-841e-4451-bfce-5817940b27d5-kserve-provision-location\") pod \"auth-disabled-test-kserve-6448cc6bb8-x7z9x\" (UID: \"efc28864-841e-4451-bfce-5817940b27d5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x"
Sep 17 22:25:24.129112 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:24.129088    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thhxp\" (UniqueName: \"kubernetes.io/projected/efc28864-841e-4451-bfce-5817940b27d5-kube-api-access-thhxp\") pod \"auth-disabled-test-kserve-6448cc6bb8-x7z9x\" (UID: \"efc28864-841e-4451-bfce-5817940b27d5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x"
Sep 17 22:25:24.129311 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:24.129290    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efc28864-841e-4451-bfce-5817940b27d5-tmp-dir\") pod \"auth-disabled-test-kserve-6448cc6bb8-x7z9x\" (UID: \"efc28864-841e-4451-bfce-5817940b27d5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x"
Sep 17 22:25:24.129406 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:24.129390    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efc28864-841e-4451-bfce-5817940b27d5-model-cache\") pod \"auth-disabled-test-kserve-6448cc6bb8-x7z9x\" (UID: \"efc28864-841e-4451-bfce-5817940b27d5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x"
Sep 17 22:25:24.129509 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:24.129475    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efc28864-841e-4451-bfce-5817940b27d5-kserve-provision-location\") pod \"auth-disabled-test-kserve-6448cc6bb8-x7z9x\" (UID: \"efc28864-841e-4451-bfce-5817940b27d5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x"
Sep 17 22:25:24.129635 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:24.129583    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efc28864-841e-4451-bfce-5817940b27d5-cabundle-cert\") pod \"auth-disabled-test-kserve-6448cc6bb8-x7z9x\" (UID: \"efc28864-841e-4451-bfce-5817940b27d5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x"
Sep 17 22:25:24.129700 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:24.129643    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efc28864-841e-4451-bfce-5817940b27d5-home\") pod \"auth-disabled-test-kserve-6448cc6bb8-x7z9x\" (UID: \"efc28864-841e-4451-bfce-5817940b27d5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x"
Sep 17 22:25:24.131226 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:24.131204    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efc28864-841e-4451-bfce-5817940b27d5-dshm\") pod \"auth-disabled-test-kserve-6448cc6bb8-x7z9x\" (UID: \"efc28864-841e-4451-bfce-5817940b27d5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x"
Sep 17 22:25:24.131415 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:24.131400    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efc28864-841e-4451-bfce-5817940b27d5-tls-certs\") pod \"auth-disabled-test-kserve-6448cc6bb8-x7z9x\" (UID: \"efc28864-841e-4451-bfce-5817940b27d5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x"
Sep 17 22:25:24.137937 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:24.137913    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thhxp\" (UniqueName: \"kubernetes.io/projected/efc28864-841e-4451-bfce-5817940b27d5-kube-api-access-thhxp\") pod \"auth-disabled-test-kserve-6448cc6bb8-x7z9x\" (UID: \"efc28864-841e-4451-bfce-5817940b27d5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x"
Sep 17 22:25:24.178090 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:24.178064    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x"
Sep 17 22:25:24.461696 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:24.461670    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x"]
Sep 17 22:25:24.463486 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:25:24.463458    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefc28864_841e_4451_bfce_5817940b27d5.slice/crio-8254cef3696eece4dc46e81bab2d9db5fa324265dcc5486280e07ff8b231b564 WatchSource:0}: Error finding container 8254cef3696eece4dc46e81bab2d9db5fa324265dcc5486280e07ff8b231b564: Status 404 returned error can't find the container with id 8254cef3696eece4dc46e81bab2d9db5fa324265dcc5486280e07ff8b231b564
Sep 17 22:25:24.819301 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:24.819211    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x" event={"ID":"efc28864-841e-4451-bfce-5817940b27d5","Type":"ContainerStarted","Data":"3b119b95635e6c323fab6ecc1b9ad0985e07a98828ea68e8499119347ac47685"}
Sep 17 22:25:24.819301 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:24.819273    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x" event={"ID":"efc28864-841e-4451-bfce-5817940b27d5","Type":"ContainerStarted","Data":"8254cef3696eece4dc46e81bab2d9db5fa324265dcc5486280e07ff8b231b564"}
Sep 17 22:25:25.049934 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:25.049888    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:25:27.837138 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:27.837104    2555 generic.go:356] "Generic (PLEG): container finished" podID="efc28864-841e-4451-bfce-5817940b27d5" containerID="3b119b95635e6c323fab6ecc1b9ad0985e07a98828ea68e8499119347ac47685" exitCode=0
Sep 17 22:25:27.837534 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:27.837158    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x" event={"ID":"efc28864-841e-4451-bfce-5817940b27d5","Type":"ContainerDied","Data":"3b119b95635e6c323fab6ecc1b9ad0985e07a98828ea68e8499119347ac47685"}
Sep 17 22:25:28.845830 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:28.845795    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x" event={"ID":"efc28864-841e-4451-bfce-5817940b27d5","Type":"ContainerStarted","Data":"af94a6e5025b96720e70b236cc3ac6c5bfe89d8027430582e3a887f1623df76b"}
Sep 17 22:25:28.868937 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:28.868878    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x" podStartSLOduration=3.553320673 podStartE2EDuration="5.868858632s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.315537959s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:25:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:25:28.865758273 +0000 UTC m=+1630.311803546" watchObservedRunningTime="2026-09-17 22:25:28.868858632 +0000 UTC m=+1630.314903904"
Sep 17 22:25:29.178289 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:29.178253    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x"
Sep 17 22:25:30.480329 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:30.480298    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:25:30.585682 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:30.585642    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:25:34.178968 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:34.178927    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x"
Sep 17 22:25:34.180868 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:34.180836    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x" podUID="efc28864-841e-4451-bfce-5817940b27d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 17 22:25:34.756698 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:34.756662    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:25:44.178849 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:44.178795    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x" podUID="efc28864-841e-4451-bfce-5817940b27d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 17 22:25:54.178622 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:25:54.178575    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x" podUID="efc28864-841e-4451-bfce-5817940b27d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 17 22:26:00.870414 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:00.870379    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:26:00.964579 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:00.964550    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:26:03.732751 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:03.732717    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8"]
Sep 17 22:26:03.736623 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:03.736597    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8"
Sep 17 22:26:03.740075 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:03.739831    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 17 22:26:03.740075 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:03.739875    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 17 22:26:03.740075 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:03.739898    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-svcqk\""
Sep 17 22:26:03.740075 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:03.740056    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 17 22:26:03.745742 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:03.745648    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8"]
Sep 17 22:26:03.767886 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:03.767855    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:26:03.888406 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:03.888361    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a50f52d-548f-4320-a0e0-48b5c5882126-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8\" (UID: \"4a50f52d-548f-4320-a0e0-48b5c5882126\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8"
Sep 17 22:26:03.888617 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:03.888420    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a50f52d-548f-4320-a0e0-48b5c5882126-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8\" (UID: \"4a50f52d-548f-4320-a0e0-48b5c5882126\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8"
Sep 17 22:26:03.888617 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:03.888495    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a50f52d-548f-4320-a0e0-48b5c5882126-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8\" (UID: \"4a50f52d-548f-4320-a0e0-48b5c5882126\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8"
Sep 17 22:26:03.888617 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:03.888557    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a50f52d-548f-4320-a0e0-48b5c5882126-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8\" (UID: \"4a50f52d-548f-4320-a0e0-48b5c5882126\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8"
Sep 17 22:26:03.888617 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:03.888584    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a50f52d-548f-4320-a0e0-48b5c5882126-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8\" (UID: \"4a50f52d-548f-4320-a0e0-48b5c5882126\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8"
Sep 17 22:26:03.888793 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:03.888644    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhwlm\" (UniqueName: \"kubernetes.io/projected/4a50f52d-548f-4320-a0e0-48b5c5882126-kube-api-access-xhwlm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8\" (UID: \"4a50f52d-548f-4320-a0e0-48b5c5882126\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8"
Sep 17 22:26:03.928763 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:03.928728    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheht2xp"]
Sep 17 22:26:03.932995 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:03.932970    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheht2xp"
Sep 17 22:26:03.935659 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:03.935632    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-nlgw6\""
Sep 17 22:26:03.937902 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:03.937872    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheht2xp"]
Sep 17 22:26:03.990329 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:03.990225    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a50f52d-548f-4320-a0e0-48b5c5882126-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8\" (UID: \"4a50f52d-548f-4320-a0e0-48b5c5882126\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8"
Sep 17 22:26:03.990329 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:03.990277    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a50f52d-548f-4320-a0e0-48b5c5882126-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8\" (UID: \"4a50f52d-548f-4320-a0e0-48b5c5882126\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8"
Sep 17 22:26:03.990574 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:03.990338    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhwlm\" (UniqueName: \"kubernetes.io/projected/4a50f52d-548f-4320-a0e0-48b5c5882126-kube-api-access-xhwlm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8\" (UID: \"4a50f52d-548f-4320-a0e0-48b5c5882126\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8"
Sep 17 22:26:03.990574 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:03.990422    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a50f52d-548f-4320-a0e0-48b5c5882126-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8\" (UID: \"4a50f52d-548f-4320-a0e0-48b5c5882126\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8"
Sep 17 22:26:03.990574 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:03.990464    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a50f52d-548f-4320-a0e0-48b5c5882126-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8\" (UID: \"4a50f52d-548f-4320-a0e0-48b5c5882126\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8"
Sep 17 22:26:03.990574 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:03.990497    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a50f52d-548f-4320-a0e0-48b5c5882126-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8\" (UID: \"4a50f52d-548f-4320-a0e0-48b5c5882126\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8"
Sep 17 22:26:03.990798 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:03.990687    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a50f52d-548f-4320-a0e0-48b5c5882126-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8\" (UID: \"4a50f52d-548f-4320-a0e0-48b5c5882126\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8"
Sep 17 22:26:03.990867 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:03.990855    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a50f52d-548f-4320-a0e0-48b5c5882126-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8\" (UID: \"4a50f52d-548f-4320-a0e0-48b5c5882126\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8"
Sep 17 22:26:03.990918 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:03.990896    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a50f52d-548f-4320-a0e0-48b5c5882126-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8\" (UID: \"4a50f52d-548f-4320-a0e0-48b5c5882126\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8"
Sep 17 22:26:03.993172 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:03.993146    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a50f52d-548f-4320-a0e0-48b5c5882126-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8\" (UID: \"4a50f52d-548f-4320-a0e0-48b5c5882126\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8"
Sep 17 22:26:03.993667 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:03.993642    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a50f52d-548f-4320-a0e0-48b5c5882126-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8\" (UID: \"4a50f52d-548f-4320-a0e0-48b5c5882126\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8"
Sep 17 22:26:03.998427 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:03.998400    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhwlm\" (UniqueName: \"kubernetes.io/projected/4a50f52d-548f-4320-a0e0-48b5c5882126-kube-api-access-xhwlm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8\" (UID: \"4a50f52d-548f-4320-a0e0-48b5c5882126\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8"
Sep 17 22:26:04.050317 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:04.050270    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8"
Sep 17 22:26:04.091634 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:04.091542    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-628dk\" (UniqueName: \"kubernetes.io/projected/3658c3da-c0fe-455d-b380-476c7efa035c-kube-api-access-628dk\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheht2xp\" (UID: \"3658c3da-c0fe-455d-b380-476c7efa035c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheht2xp"
Sep 17 22:26:04.091842 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:04.091685    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3658c3da-c0fe-455d-b380-476c7efa035c-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheht2xp\" (UID: \"3658c3da-c0fe-455d-b380-476c7efa035c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheht2xp"
Sep 17 22:26:04.179081 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:04.179037    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x" podUID="efc28864-841e-4451-bfce-5817940b27d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 17 22:26:04.192412 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:04.192380    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3658c3da-c0fe-455d-b380-476c7efa035c-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheht2xp\" (UID: \"3658c3da-c0fe-455d-b380-476c7efa035c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheht2xp"
Sep 17 22:26:04.192590 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:04.192450    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-628dk\" (UniqueName: \"kubernetes.io/projected/3658c3da-c0fe-455d-b380-476c7efa035c-kube-api-access-628dk\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheht2xp\" (UID: \"3658c3da-c0fe-455d-b380-476c7efa035c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheht2xp"
Sep 17 22:26:04.195540 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:04.195505    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3658c3da-c0fe-455d-b380-476c7efa035c-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheht2xp\" (UID: \"3658c3da-c0fe-455d-b380-476c7efa035c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheht2xp"
Sep 17 22:26:04.200301 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:04.200270    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-628dk\" (UniqueName: \"kubernetes.io/projected/3658c3da-c0fe-455d-b380-476c7efa035c-kube-api-access-628dk\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheht2xp\" (UID: \"3658c3da-c0fe-455d-b380-476c7efa035c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheht2xp"
Sep 17 22:26:04.245500 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:04.245413    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheht2xp"
Sep 17 22:26:04.332756 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:04.332725    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8"]
Sep 17 22:26:04.334457 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:26:04.334432    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a50f52d_548f_4320_a0e0_48b5c5882126.slice/crio-aadb545dd9e8fd51a717195393ae63024f01cb440ca98450589aad107bb85008 WatchSource:0}: Error finding container aadb545dd9e8fd51a717195393ae63024f01cb440ca98450589aad107bb85008: Status 404 returned error can't find the container with id aadb545dd9e8fd51a717195393ae63024f01cb440ca98450589aad107bb85008
Sep 17 22:26:04.520271 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:04.520244    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheht2xp"]
Sep 17 22:26:04.521535 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:26:04.521510    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3658c3da_c0fe_455d_b380_476c7efa035c.slice/crio-09e4e22c2f32216426bb353563f5abbb51075d6a7f811339cae316c7e5325223 WatchSource:0}: Error finding container 09e4e22c2f32216426bb353563f5abbb51075d6a7f811339cae316c7e5325223: Status 404 returned error can't find the container with id 09e4e22c2f32216426bb353563f5abbb51075d6a7f811339cae316c7e5325223
Sep 17 22:26:05.066047 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:05.066012    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8" event={"ID":"4a50f52d-548f-4320-a0e0-48b5c5882126","Type":"ContainerStarted","Data":"931c0bde184d8d3472fe764e8bdf76ac3aad10a7869e31571c1065c323eaf1a8"}
Sep 17 22:26:05.066510 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:05.066073    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8" event={"ID":"4a50f52d-548f-4320-a0e0-48b5c5882126","Type":"ContainerStarted","Data":"aadb545dd9e8fd51a717195393ae63024f01cb440ca98450589aad107bb85008"}
Sep 17 22:26:05.067827 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:05.067799    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheht2xp" event={"ID":"3658c3da-c0fe-455d-b380-476c7efa035c","Type":"ContainerStarted","Data":"d2dfb8701535b552d294b84b1c9dac5912971c84c7d6ac73ba6bd204b19d1ee7"}
Sep 17 22:26:05.067981 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:05.067838    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheht2xp" event={"ID":"3658c3da-c0fe-455d-b380-476c7efa035c","Type":"ContainerStarted","Data":"09e4e22c2f32216426bb353563f5abbb51075d6a7f811339cae316c7e5325223"}
Sep 17 22:26:05.067981 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:05.067860    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheht2xp"
Sep 17 22:26:05.083838 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:05.083761    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8" podStartSLOduration=2.083738631 podStartE2EDuration="2.083738631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:26:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:26:05.082717726 +0000 UTC m=+1666.528762994" watchObservedRunningTime="2026-09-17 22:26:05.083738631 +0000 UTC m=+1666.529783908"
Sep 17 22:26:05.100741 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:05.100692    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheht2xp" podStartSLOduration=2.10067957 podStartE2EDuration="2.10067957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:26:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:26:05.099967956 +0000 UTC m=+1666.546013229" watchObservedRunningTime="2026-09-17 22:26:05.10067957 +0000 UTC m=+1666.546724841"
Sep 17 22:26:05.278758 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:05.278714    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:26:06.051165 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:06.051127    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8"
Sep 17 22:26:07.550534 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:07.550493    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:26:12.991128 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:12.991089    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:26:13.090962 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:13.090931    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:26:14.051252 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:14.051215    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8"
Sep 17 22:26:14.064072 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:14.064046    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8"
Sep 17 22:26:14.135521 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:14.135491    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8"
Sep 17 22:26:14.179549 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:14.179508    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x" podUID="efc28864-841e-4451-bfce-5817940b27d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 17 22:26:20.796860 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:20.796823    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:26:24.179086 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:24.179046    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x" podUID="efc28864-841e-4451-bfce-5817940b27d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 17 22:26:34.179389 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:34.179290    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x" podUID="efc28864-841e-4451-bfce-5817940b27d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 17 22:26:36.075938 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:36.075905    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheht2xp"
Sep 17 22:26:37.491146 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:37.491112    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheht2xp"]
Sep 17 22:26:37.494432 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:37.494398    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8"]
Sep 17 22:26:37.506061 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:26:37.506005    2555 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 22:26:37.506061 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:26:37.506061 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:26:37.506061 ip-10-0-140-157 kubenswrapper[2555]:  > containerID="931c0bde184d8d3472fe764e8bdf76ac3aad10a7869e31571c1065c323eaf1a8" cmd=["/bin/sleep","15"]
Sep 17 22:26:37.506432 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:37.506083    2555 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8" message="" err=<
Sep 17 22:26:37.506432 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:26:37.506432 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:26:37.506432 ip-10-0-140-157 kubenswrapper[2555]:  >
Sep 17 22:26:37.506432 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:26:37.506140    2555 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 22:26:37.506432 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:26:37.506432 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:26:37.506432 ip-10-0-140-157 kubenswrapper[2555]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8" podUID="4a50f52d-548f-4320-a0e0-48b5c5882126" containerName="main" containerID="cri-o://931c0bde184d8d3472fe764e8bdf76ac3aad10a7869e31571c1065c323eaf1a8"
Sep 17 22:26:37.506432 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:37.506208    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8" podUID="4a50f52d-548f-4320-a0e0-48b5c5882126" containerName="main" containerID="cri-o://931c0bde184d8d3472fe764e8bdf76ac3aad10a7869e31571c1065c323eaf1a8" gracePeriod=60
Sep 17 22:26:37.508068 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:37.508039    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:26:37.607030 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:37.606987    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:26:37.864870 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:37.864845    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8"
Sep 17 22:26:38.017894 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:38.017857    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a50f52d-548f-4320-a0e0-48b5c5882126-dshm\") pod \"4a50f52d-548f-4320-a0e0-48b5c5882126\" (UID: \"4a50f52d-548f-4320-a0e0-48b5c5882126\") "
Sep 17 22:26:38.018078 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:38.017960    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a50f52d-548f-4320-a0e0-48b5c5882126-model-cache\") pod \"4a50f52d-548f-4320-a0e0-48b5c5882126\" (UID: \"4a50f52d-548f-4320-a0e0-48b5c5882126\") "
Sep 17 22:26:38.018078 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:38.018004    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a50f52d-548f-4320-a0e0-48b5c5882126-tmp-dir\") pod \"4a50f52d-548f-4320-a0e0-48b5c5882126\" (UID: \"4a50f52d-548f-4320-a0e0-48b5c5882126\") "
Sep 17 22:26:38.018078 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:38.018035    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a50f52d-548f-4320-a0e0-48b5c5882126-tls-certs\") pod \"4a50f52d-548f-4320-a0e0-48b5c5882126\" (UID: \"4a50f52d-548f-4320-a0e0-48b5c5882126\") "
Sep 17 22:26:38.018078 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:38.018054    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhwlm\" (UniqueName: \"kubernetes.io/projected/4a50f52d-548f-4320-a0e0-48b5c5882126-kube-api-access-xhwlm\") pod \"4a50f52d-548f-4320-a0e0-48b5c5882126\" (UID: \"4a50f52d-548f-4320-a0e0-48b5c5882126\") "
Sep 17 22:26:38.018372 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:38.018089    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a50f52d-548f-4320-a0e0-48b5c5882126-home\") pod \"4a50f52d-548f-4320-a0e0-48b5c5882126\" (UID: \"4a50f52d-548f-4320-a0e0-48b5c5882126\") "
Sep 17 22:26:38.018372 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:38.018210    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a50f52d-548f-4320-a0e0-48b5c5882126-model-cache" (OuterVolumeSpecName: "model-cache") pod "4a50f52d-548f-4320-a0e0-48b5c5882126" (UID: "4a50f52d-548f-4320-a0e0-48b5c5882126"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:26:38.018372 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:38.018334    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a50f52d-548f-4320-a0e0-48b5c5882126-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4a50f52d-548f-4320-a0e0-48b5c5882126" (UID: "4a50f52d-548f-4320-a0e0-48b5c5882126"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:26:38.018522 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:38.018376    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a50f52d-548f-4320-a0e0-48b5c5882126-home" (OuterVolumeSpecName: "home") pod "4a50f52d-548f-4320-a0e0-48b5c5882126" (UID: "4a50f52d-548f-4320-a0e0-48b5c5882126"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:26:38.018522 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:38.018410    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a50f52d-548f-4320-a0e0-48b5c5882126-model-cache\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:26:38.020479 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:38.020405    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a50f52d-548f-4320-a0e0-48b5c5882126-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4a50f52d-548f-4320-a0e0-48b5c5882126" (UID: "4a50f52d-548f-4320-a0e0-48b5c5882126"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:26:38.020479 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:38.020421    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a50f52d-548f-4320-a0e0-48b5c5882126-dshm" (OuterVolumeSpecName: "dshm") pod "4a50f52d-548f-4320-a0e0-48b5c5882126" (UID: "4a50f52d-548f-4320-a0e0-48b5c5882126"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:26:38.020479 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:38.020458    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a50f52d-548f-4320-a0e0-48b5c5882126-kube-api-access-xhwlm" (OuterVolumeSpecName: "kube-api-access-xhwlm") pod "4a50f52d-548f-4320-a0e0-48b5c5882126" (UID: "4a50f52d-548f-4320-a0e0-48b5c5882126"). InnerVolumeSpecName "kube-api-access-xhwlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:26:38.119231 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:38.119169    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a50f52d-548f-4320-a0e0-48b5c5882126-tmp-dir\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:26:38.119231 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:38.119226    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a50f52d-548f-4320-a0e0-48b5c5882126-tls-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:26:38.119231 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:38.119235    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhwlm\" (UniqueName: \"kubernetes.io/projected/4a50f52d-548f-4320-a0e0-48b5c5882126-kube-api-access-xhwlm\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:26:38.119472 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:38.119244    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a50f52d-548f-4320-a0e0-48b5c5882126-home\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:26:38.119472 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:38.119253    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a50f52d-548f-4320-a0e0-48b5c5882126-dshm\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:26:38.272402 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:38.272302    2555 generic.go:356] "Generic (PLEG): container finished" podID="4a50f52d-548f-4320-a0e0-48b5c5882126" containerID="931c0bde184d8d3472fe764e8bdf76ac3aad10a7869e31571c1065c323eaf1a8" exitCode=0
Sep 17 22:26:38.272402 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:38.272351    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8"
Sep 17 22:26:38.272402 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:38.272379    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8" event={"ID":"4a50f52d-548f-4320-a0e0-48b5c5882126","Type":"ContainerDied","Data":"931c0bde184d8d3472fe764e8bdf76ac3aad10a7869e31571c1065c323eaf1a8"}
Sep 17 22:26:38.272703 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:38.272467    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8" event={"ID":"4a50f52d-548f-4320-a0e0-48b5c5882126","Type":"ContainerDied","Data":"aadb545dd9e8fd51a717195393ae63024f01cb440ca98450589aad107bb85008"}
Sep 17 22:26:38.272703 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:38.272496    2555 scope.go:117] "RemoveContainer" containerID="931c0bde184d8d3472fe764e8bdf76ac3aad10a7869e31571c1065c323eaf1a8"
Sep 17 22:26:38.284389 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:38.284369    2555 scope.go:117] "RemoveContainer" containerID="931c0bde184d8d3472fe764e8bdf76ac3aad10a7869e31571c1065c323eaf1a8"
Sep 17 22:26:38.284761 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:26:38.284737    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"931c0bde184d8d3472fe764e8bdf76ac3aad10a7869e31571c1065c323eaf1a8\": container with ID starting with 931c0bde184d8d3472fe764e8bdf76ac3aad10a7869e31571c1065c323eaf1a8 not found: ID does not exist" containerID="931c0bde184d8d3472fe764e8bdf76ac3aad10a7869e31571c1065c323eaf1a8"
Sep 17 22:26:38.284851 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:38.284768    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"931c0bde184d8d3472fe764e8bdf76ac3aad10a7869e31571c1065c323eaf1a8"} err="failed to get container status \"931c0bde184d8d3472fe764e8bdf76ac3aad10a7869e31571c1065c323eaf1a8\": rpc error: code = NotFound desc = could not find container \"931c0bde184d8d3472fe764e8bdf76ac3aad10a7869e31571c1065c323eaf1a8\": container with ID starting with 931c0bde184d8d3472fe764e8bdf76ac3aad10a7869e31571c1065c323eaf1a8 not found: ID does not exist"
Sep 17 22:26:38.297429 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:38.297402    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8"]
Sep 17 22:26:38.302501 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:38.302478    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ht9z8"]
Sep 17 22:26:38.682124 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:38.682089    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a50f52d-548f-4320-a0e0-48b5c5882126" path="/var/lib/kubelet/pods/4a50f52d-548f-4320-a0e0-48b5c5882126/volumes"
Sep 17 22:26:44.178941 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:44.178891    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x" podUID="efc28864-841e-4451-bfce-5817940b27d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 17 22:26:46.076579 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:46.076541    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheht2xp" podUID="3658c3da-c0fe-455d-b380-476c7efa035c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:26:52.515245 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:52.515171    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheht2xp" podUID="3658c3da-c0fe-455d-b380-476c7efa035c" containerName="main" containerID="cri-o://d2dfb8701535b552d294b84b1c9dac5912971c84c7d6ac73ba6bd204b19d1ee7" gracePeriod=45
Sep 17 22:26:54.178882 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:54.178823    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x" podUID="efc28864-841e-4451-bfce-5817940b27d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 17 22:26:56.076105 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:26:56.076067    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheht2xp" podUID="3658c3da-c0fe-455d-b380-476c7efa035c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:27:04.179073 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:04.179027    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x" podUID="efc28864-841e-4451-bfce-5817940b27d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 17 22:27:06.076371 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:06.076332    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheht2xp" podUID="3658c3da-c0fe-455d-b380-476c7efa035c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:27:06.076802 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:06.076439    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheht2xp"
Sep 17 22:27:14.178609 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:14.178566    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x" podUID="efc28864-841e-4451-bfce-5817940b27d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 17 22:27:16.076121 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:16.076083    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheht2xp" podUID="3658c3da-c0fe-455d-b380-476c7efa035c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:27:22.772984 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:22.772962    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheht2xp"
Sep 17 22:27:22.923848 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:22.923819    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3658c3da-c0fe-455d-b380-476c7efa035c-tls-certs\") pod \"3658c3da-c0fe-455d-b380-476c7efa035c\" (UID: \"3658c3da-c0fe-455d-b380-476c7efa035c\") "
Sep 17 22:27:22.924009 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:22.923861    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-628dk\" (UniqueName: \"kubernetes.io/projected/3658c3da-c0fe-455d-b380-476c7efa035c-kube-api-access-628dk\") pod \"3658c3da-c0fe-455d-b380-476c7efa035c\" (UID: \"3658c3da-c0fe-455d-b380-476c7efa035c\") "
Sep 17 22:27:22.925935 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:22.925908    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3658c3da-c0fe-455d-b380-476c7efa035c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3658c3da-c0fe-455d-b380-476c7efa035c" (UID: "3658c3da-c0fe-455d-b380-476c7efa035c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:27:22.926002 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:22.925962    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3658c3da-c0fe-455d-b380-476c7efa035c-kube-api-access-628dk" (OuterVolumeSpecName: "kube-api-access-628dk") pod "3658c3da-c0fe-455d-b380-476c7efa035c" (UID: "3658c3da-c0fe-455d-b380-476c7efa035c"). InnerVolumeSpecName "kube-api-access-628dk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:27:23.025149 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:23.025116    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3658c3da-c0fe-455d-b380-476c7efa035c-tls-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:27:23.025149 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:23.025145    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-628dk\" (UniqueName: \"kubernetes.io/projected/3658c3da-c0fe-455d-b380-476c7efa035c-kube-api-access-628dk\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:27:23.541416 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:23.541383    2555 generic.go:356] "Generic (PLEG): container finished" podID="3658c3da-c0fe-455d-b380-476c7efa035c" containerID="d2dfb8701535b552d294b84b1c9dac5912971c84c7d6ac73ba6bd204b19d1ee7" exitCode=0
Sep 17 22:27:23.541589 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:23.541429    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheht2xp"
Sep 17 22:27:23.541589 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:23.541460    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheht2xp" event={"ID":"3658c3da-c0fe-455d-b380-476c7efa035c","Type":"ContainerDied","Data":"d2dfb8701535b552d294b84b1c9dac5912971c84c7d6ac73ba6bd204b19d1ee7"}
Sep 17 22:27:23.541589 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:23.541515    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheht2xp" event={"ID":"3658c3da-c0fe-455d-b380-476c7efa035c","Type":"ContainerDied","Data":"09e4e22c2f32216426bb353563f5abbb51075d6a7f811339cae316c7e5325223"}
Sep 17 22:27:23.541589 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:23.541547    2555 scope.go:117] "RemoveContainer" containerID="d2dfb8701535b552d294b84b1c9dac5912971c84c7d6ac73ba6bd204b19d1ee7"
Sep 17 22:27:23.550993 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:23.550959    2555 scope.go:117] "RemoveContainer" containerID="d2dfb8701535b552d294b84b1c9dac5912971c84c7d6ac73ba6bd204b19d1ee7"
Sep 17 22:27:23.551275 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:27:23.551252    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2dfb8701535b552d294b84b1c9dac5912971c84c7d6ac73ba6bd204b19d1ee7\": container with ID starting with d2dfb8701535b552d294b84b1c9dac5912971c84c7d6ac73ba6bd204b19d1ee7 not found: ID does not exist" containerID="d2dfb8701535b552d294b84b1c9dac5912971c84c7d6ac73ba6bd204b19d1ee7"
Sep 17 22:27:23.551368 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:23.551288    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2dfb8701535b552d294b84b1c9dac5912971c84c7d6ac73ba6bd204b19d1ee7"} err="failed to get container status \"d2dfb8701535b552d294b84b1c9dac5912971c84c7d6ac73ba6bd204b19d1ee7\": rpc error: code = NotFound desc = could not find container \"d2dfb8701535b552d294b84b1c9dac5912971c84c7d6ac73ba6bd204b19d1ee7\": container with ID starting with d2dfb8701535b552d294b84b1c9dac5912971c84c7d6ac73ba6bd204b19d1ee7 not found: ID does not exist"
Sep 17 22:27:23.562096 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:23.562073    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheht2xp"]
Sep 17 22:27:23.565508 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:23.565486    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheht2xp"]
Sep 17 22:27:24.188358 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:24.188324    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x"
Sep 17 22:27:24.196087 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:24.196061    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x"
Sep 17 22:27:24.679279 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:24.679237    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3658c3da-c0fe-455d-b380-476c7efa035c" path="/var/lib/kubelet/pods/3658c3da-c0fe-455d-b380-476c7efa035c/volumes"
Sep 17 22:27:27.220543 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:27.220509    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:27:27.320283 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:27.320253    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:27:28.547171 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:28.547133    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:27:28.647608 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:28.647571    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:27:33.000476 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:33.000437    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:27:37.620776 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:37.620741    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f"]
Sep 17 22:27:37.623243 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:37.621118    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a50f52d-548f-4320-a0e0-48b5c5882126" containerName="main"
Sep 17 22:27:37.623243 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:37.621130    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a50f52d-548f-4320-a0e0-48b5c5882126" containerName="main"
Sep 17 22:27:37.623243 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:37.621155    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3658c3da-c0fe-455d-b380-476c7efa035c" containerName="main"
Sep 17 22:27:37.623243 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:37.621161    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3658c3da-c0fe-455d-b380-476c7efa035c" containerName="main"
Sep 17 22:27:37.623243 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:37.621236    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a50f52d-548f-4320-a0e0-48b5c5882126" containerName="main"
Sep 17 22:27:37.623243 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:37.621247    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3658c3da-c0fe-455d-b380-476c7efa035c" containerName="main"
Sep 17 22:27:37.624164 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:37.624145    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f"
Sep 17 22:27:37.627238 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:37.627212    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 17 22:27:37.627366 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:37.627258    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 17 22:27:37.627366 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:37.627278    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-j7qlc\""
Sep 17 22:27:37.627366 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:37.627278    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 17 22:27:37.633154 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:37.633134    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f"]
Sep 17 22:27:37.637276 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:37.637249    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:27:37.756495 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:37.756467    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f\" (UID: \"e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f"
Sep 17 22:27:37.756495 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:37.756497    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f\" (UID: \"e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f"
Sep 17 22:27:37.756722 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:37.756521    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f\" (UID: \"e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f"
Sep 17 22:27:37.756722 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:37.756625    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f\" (UID: \"e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f"
Sep 17 22:27:37.756722 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:37.756664    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxcnl\" (UniqueName: \"kubernetes.io/projected/e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8-kube-api-access-qxcnl\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f\" (UID: \"e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f"
Sep 17 22:27:37.756722 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:37.756711    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f\" (UID: \"e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f"
Sep 17 22:27:37.857592 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:37.857557    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f\" (UID: \"e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f"
Sep 17 22:27:37.857592 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:37.857596    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f\" (UID: \"e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f"
Sep 17 22:27:37.857836 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:37.857627    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f\" (UID: \"e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f"
Sep 17 22:27:37.857836 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:37.857685    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f\" (UID: \"e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f"
Sep 17 22:27:37.857836 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:37.857720    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxcnl\" (UniqueName: \"kubernetes.io/projected/e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8-kube-api-access-qxcnl\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f\" (UID: \"e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f"
Sep 17 22:27:37.857988 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:37.857876    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f\" (UID: \"e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f"
Sep 17 22:27:37.858133 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:37.858110    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f\" (UID: \"e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f"
Sep 17 22:27:37.858260 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:37.858148    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f\" (UID: \"e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f"
Sep 17 22:27:37.858325 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:37.858277    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f\" (UID: \"e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f"
Sep 17 22:27:37.860051 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:37.860029    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f\" (UID: \"e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f"
Sep 17 22:27:37.860129 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:37.860115    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f\" (UID: \"e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f"
Sep 17 22:27:37.866739 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:37.866721    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxcnl\" (UniqueName: \"kubernetes.io/projected/e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8-kube-api-access-qxcnl\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f\" (UID: \"e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f"
Sep 17 22:27:37.936432 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:37.936400    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f"
Sep 17 22:27:37.941123 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:37.941098    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fqdr"]
Sep 17 22:27:37.944892 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:37.944872    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fqdr"
Sep 17 22:27:37.947174 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:37.947156    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-nhrvh\""
Sep 17 22:27:37.963109 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:37.963076    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fqdr"]
Sep 17 22:27:37.971765 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:37.970754    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:27:38.061152 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:38.061112    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwgtc\" (UniqueName: \"kubernetes.io/projected/f8e728a1-e0a5-40ea-b0cc-fd2c5a2d1369-kube-api-access-qwgtc\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fqdr\" (UID: \"f8e728a1-e0a5-40ea-b0cc-fd2c5a2d1369\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fqdr"
Sep 17 22:27:38.061355 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:38.061206    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8e728a1-e0a5-40ea-b0cc-fd2c5a2d1369-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fqdr\" (UID: \"f8e728a1-e0a5-40ea-b0cc-fd2c5a2d1369\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fqdr"
Sep 17 22:27:38.162651 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:38.162618    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8e728a1-e0a5-40ea-b0cc-fd2c5a2d1369-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fqdr\" (UID: \"f8e728a1-e0a5-40ea-b0cc-fd2c5a2d1369\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fqdr"
Sep 17 22:27:38.162827 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:38.162679    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwgtc\" (UniqueName: \"kubernetes.io/projected/f8e728a1-e0a5-40ea-b0cc-fd2c5a2d1369-kube-api-access-qwgtc\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fqdr\" (UID: \"f8e728a1-e0a5-40ea-b0cc-fd2c5a2d1369\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fqdr"
Sep 17 22:27:38.165139 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:38.165118    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8e728a1-e0a5-40ea-b0cc-fd2c5a2d1369-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fqdr\" (UID: \"f8e728a1-e0a5-40ea-b0cc-fd2c5a2d1369\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fqdr"
Sep 17 22:27:38.170202 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:38.170169    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwgtc\" (UniqueName: \"kubernetes.io/projected/f8e728a1-e0a5-40ea-b0cc-fd2c5a2d1369-kube-api-access-qwgtc\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fqdr\" (UID: \"f8e728a1-e0a5-40ea-b0cc-fd2c5a2d1369\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fqdr"
Sep 17 22:27:38.218676 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:38.218653    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f"]
Sep 17 22:27:38.220500 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:27:38.220473    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode821d4ae_7c6e_4b32_a1b7_867e0ca6bea8.slice/crio-6e16cf019a6764a470372c406bb211642fba18a0fc8e6841e10be4d9c8a14fcc WatchSource:0}: Error finding container 6e16cf019a6764a470372c406bb211642fba18a0fc8e6841e10be4d9c8a14fcc: Status 404 returned error can't find the container with id 6e16cf019a6764a470372c406bb211642fba18a0fc8e6841e10be4d9c8a14fcc
Sep 17 22:27:38.283397 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:38.283370    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fqdr"
Sep 17 22:27:38.559107 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:38.559080    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fqdr"]
Sep 17 22:27:38.560627 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:27:38.560602    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8e728a1_e0a5_40ea_b0cc_fd2c5a2d1369.slice/crio-4f7391ee78876cb12710435ff07953cc05dc3119d032e8119e66c1776d770dca WatchSource:0}: Error finding container 4f7391ee78876cb12710435ff07953cc05dc3119d032e8119e66c1776d770dca: Status 404 returned error can't find the container with id 4f7391ee78876cb12710435ff07953cc05dc3119d032e8119e66c1776d770dca
Sep 17 22:27:38.637457 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:38.637425    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fqdr" event={"ID":"f8e728a1-e0a5-40ea-b0cc-fd2c5a2d1369","Type":"ContainerStarted","Data":"ed0a3c2480c292ab740196b1a456b5bfc42a4dc8f743f8b956c2440cf5dce410"}
Sep 17 22:27:38.637901 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:38.637486    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fqdr" event={"ID":"f8e728a1-e0a5-40ea-b0cc-fd2c5a2d1369","Type":"ContainerStarted","Data":"4f7391ee78876cb12710435ff07953cc05dc3119d032e8119e66c1776d770dca"}
Sep 17 22:27:38.637901 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:38.637606    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fqdr"
Sep 17 22:27:38.639161 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:38.639137    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f" event={"ID":"e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8","Type":"ContainerStarted","Data":"222728302c6d7ff7f193f65ea669eff803cdc9d983bd56504200c23722713a9b"}
Sep 17 22:27:38.639303 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:38.639196    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f" event={"ID":"e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8","Type":"ContainerStarted","Data":"6e16cf019a6764a470372c406bb211642fba18a0fc8e6841e10be4d9c8a14fcc"}
Sep 17 22:27:38.654721 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:38.654667    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fqdr" podStartSLOduration=1.654650615 podStartE2EDuration="1.654650615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:27:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:27:38.651963519 +0000 UTC m=+1760.098008794" watchObservedRunningTime="2026-09-17 22:27:38.654650615 +0000 UTC m=+1760.100695886"
Sep 17 22:27:38.667070 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:38.667021    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f" podStartSLOduration=1.667005494 podStartE2EDuration="1.667005494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:27:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:27:38.666446543 +0000 UTC m=+1760.112491827" watchObservedRunningTime="2026-09-17 22:27:38.667005494 +0000 UTC m=+1760.113050765"
Sep 17 22:27:38.860264 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:38.860152    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:27:38.936861 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:38.936823    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f"
Sep 17 22:27:38.960793 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:38.960763    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:27:47.439225 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:47.439175    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:27:47.937201 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:47.937155    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f"
Sep 17 22:27:47.950043 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:47.950020    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f"
Sep 17 22:27:48.705403 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:48.705369    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f"
Sep 17 22:27:48.940684 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:48.940651    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x"]
Sep 17 22:27:48.948899 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:48.948870    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:27:49.048225 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:27:49.048116    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:28:03.964729 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:03.964617    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x" podUID="efc28864-841e-4451-bfce-5817940b27d5" containerName="main" containerID="cri-o://af94a6e5025b96720e70b236cc3ac6c5bfe89d8027430582e3a887f1623df76b" gracePeriod=45
Sep 17 22:28:04.189910 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:04.189852    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x" podUID="efc28864-841e-4451-bfce-5817940b27d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 17 22:28:04.716257 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:04.716234    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x"
Sep 17 22:28:04.793893 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:04.793816    2555 generic.go:356] "Generic (PLEG): container finished" podID="efc28864-841e-4451-bfce-5817940b27d5" containerID="af94a6e5025b96720e70b236cc3ac6c5bfe89d8027430582e3a887f1623df76b" exitCode=0
Sep 17 22:28:04.793893 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:04.793875    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x"
Sep 17 22:28:04.794067 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:04.793890    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x" event={"ID":"efc28864-841e-4451-bfce-5817940b27d5","Type":"ContainerDied","Data":"af94a6e5025b96720e70b236cc3ac6c5bfe89d8027430582e3a887f1623df76b"}
Sep 17 22:28:04.794067 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:04.793939    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x" event={"ID":"efc28864-841e-4451-bfce-5817940b27d5","Type":"ContainerDied","Data":"8254cef3696eece4dc46e81bab2d9db5fa324265dcc5486280e07ff8b231b564"}
Sep 17 22:28:04.794067 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:04.793961    2555 scope.go:117] "RemoveContainer" containerID="af94a6e5025b96720e70b236cc3ac6c5bfe89d8027430582e3a887f1623df76b"
Sep 17 22:28:04.794232 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:04.794128    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efc28864-841e-4451-bfce-5817940b27d5-tls-certs\") pod \"efc28864-841e-4451-bfce-5817940b27d5\" (UID: \"efc28864-841e-4451-bfce-5817940b27d5\") "
Sep 17 22:28:04.794232 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:04.794197    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efc28864-841e-4451-bfce-5817940b27d5-tmp-dir\") pod \"efc28864-841e-4451-bfce-5817940b27d5\" (UID: \"efc28864-841e-4451-bfce-5817940b27d5\") "
Sep 17 22:28:04.794354 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:04.794233    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thhxp\" (UniqueName: \"kubernetes.io/projected/efc28864-841e-4451-bfce-5817940b27d5-kube-api-access-thhxp\") pod \"efc28864-841e-4451-bfce-5817940b27d5\" (UID: \"efc28864-841e-4451-bfce-5817940b27d5\") "
Sep 17 22:28:04.794354 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:04.794283    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efc28864-841e-4451-bfce-5817940b27d5-dshm\") pod \"efc28864-841e-4451-bfce-5817940b27d5\" (UID: \"efc28864-841e-4451-bfce-5817940b27d5\") "
Sep 17 22:28:04.794354 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:04.794322    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efc28864-841e-4451-bfce-5817940b27d5-model-cache\") pod \"efc28864-841e-4451-bfce-5817940b27d5\" (UID: \"efc28864-841e-4451-bfce-5817940b27d5\") "
Sep 17 22:28:04.794354 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:04.794344    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efc28864-841e-4451-bfce-5817940b27d5-home\") pod \"efc28864-841e-4451-bfce-5817940b27d5\" (UID: \"efc28864-841e-4451-bfce-5817940b27d5\") "
Sep 17 22:28:04.794554 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:04.794378    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efc28864-841e-4451-bfce-5817940b27d5-kserve-provision-location\") pod \"efc28864-841e-4451-bfce-5817940b27d5\" (UID: \"efc28864-841e-4451-bfce-5817940b27d5\") "
Sep 17 22:28:04.794554 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:04.794409    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efc28864-841e-4451-bfce-5817940b27d5-cabundle-cert\") pod \"efc28864-841e-4451-bfce-5817940b27d5\" (UID: \"efc28864-841e-4451-bfce-5817940b27d5\") "
Sep 17 22:28:04.794802 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:04.794746    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc28864-841e-4451-bfce-5817940b27d5-model-cache" (OuterVolumeSpecName: "model-cache") pod "efc28864-841e-4451-bfce-5817940b27d5" (UID: "efc28864-841e-4451-bfce-5817940b27d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:28:04.795051 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:04.795027    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efc28864-841e-4451-bfce-5817940b27d5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "efc28864-841e-4451-bfce-5817940b27d5" (UID: "efc28864-841e-4451-bfce-5817940b27d5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:28:04.795201 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:04.795046    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc28864-841e-4451-bfce-5817940b27d5-home" (OuterVolumeSpecName: "home") pod "efc28864-841e-4451-bfce-5817940b27d5" (UID: "efc28864-841e-4451-bfce-5817940b27d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:28:04.797036 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:04.797013    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efc28864-841e-4451-bfce-5817940b27d5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "efc28864-841e-4451-bfce-5817940b27d5" (UID: "efc28864-841e-4451-bfce-5817940b27d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:28:04.797154 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:04.797014    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efc28864-841e-4451-bfce-5817940b27d5-kube-api-access-thhxp" (OuterVolumeSpecName: "kube-api-access-thhxp") pod "efc28864-841e-4451-bfce-5817940b27d5" (UID: "efc28864-841e-4451-bfce-5817940b27d5"). InnerVolumeSpecName "kube-api-access-thhxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:28:04.797154 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:04.797081    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc28864-841e-4451-bfce-5817940b27d5-dshm" (OuterVolumeSpecName: "dshm") pod "efc28864-841e-4451-bfce-5817940b27d5" (UID: "efc28864-841e-4451-bfce-5817940b27d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:28:04.807163 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:04.807140    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc28864-841e-4451-bfce-5817940b27d5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "efc28864-841e-4451-bfce-5817940b27d5" (UID: "efc28864-841e-4451-bfce-5817940b27d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:28:04.815474 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:04.815454    2555 scope.go:117] "RemoveContainer" containerID="3b119b95635e6c323fab6ecc1b9ad0985e07a98828ea68e8499119347ac47685"
Sep 17 22:28:04.819962 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:04.819935    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc28864-841e-4451-bfce-5817940b27d5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "efc28864-841e-4451-bfce-5817940b27d5" (UID: "efc28864-841e-4451-bfce-5817940b27d5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:28:04.823586 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:04.823564    2555 scope.go:117] "RemoveContainer" containerID="af94a6e5025b96720e70b236cc3ac6c5bfe89d8027430582e3a887f1623df76b"
Sep 17 22:28:04.823859 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:28:04.823841    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af94a6e5025b96720e70b236cc3ac6c5bfe89d8027430582e3a887f1623df76b\": container with ID starting with af94a6e5025b96720e70b236cc3ac6c5bfe89d8027430582e3a887f1623df76b not found: ID does not exist" containerID="af94a6e5025b96720e70b236cc3ac6c5bfe89d8027430582e3a887f1623df76b"
Sep 17 22:28:04.823908 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:04.823868    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af94a6e5025b96720e70b236cc3ac6c5bfe89d8027430582e3a887f1623df76b"} err="failed to get container status \"af94a6e5025b96720e70b236cc3ac6c5bfe89d8027430582e3a887f1623df76b\": rpc error: code = NotFound desc = could not find container \"af94a6e5025b96720e70b236cc3ac6c5bfe89d8027430582e3a887f1623df76b\": container with ID starting with af94a6e5025b96720e70b236cc3ac6c5bfe89d8027430582e3a887f1623df76b not found: ID does not exist"
Sep 17 22:28:04.823908 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:04.823887    2555 scope.go:117] "RemoveContainer" containerID="3b119b95635e6c323fab6ecc1b9ad0985e07a98828ea68e8499119347ac47685"
Sep 17 22:28:04.824165 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:28:04.824146    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b119b95635e6c323fab6ecc1b9ad0985e07a98828ea68e8499119347ac47685\": container with ID starting with 3b119b95635e6c323fab6ecc1b9ad0985e07a98828ea68e8499119347ac47685 not found: ID does not exist" containerID="3b119b95635e6c323fab6ecc1b9ad0985e07a98828ea68e8499119347ac47685"
Sep 17 22:28:04.824236 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:04.824171    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b119b95635e6c323fab6ecc1b9ad0985e07a98828ea68e8499119347ac47685"} err="failed to get container status \"3b119b95635e6c323fab6ecc1b9ad0985e07a98828ea68e8499119347ac47685\": rpc error: code = NotFound desc = could not find container \"3b119b95635e6c323fab6ecc1b9ad0985e07a98828ea68e8499119347ac47685\": container with ID starting with 3b119b95635e6c323fab6ecc1b9ad0985e07a98828ea68e8499119347ac47685 not found: ID does not exist"
Sep 17 22:28:04.895865 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:04.895841    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efc28864-841e-4451-bfce-5817940b27d5-dshm\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:28:04.895865 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:04.895864    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efc28864-841e-4451-bfce-5817940b27d5-model-cache\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:28:04.896006 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:04.895873    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efc28864-841e-4451-bfce-5817940b27d5-home\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:28:04.896006 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:04.895881    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efc28864-841e-4451-bfce-5817940b27d5-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:28:04.896006 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:04.895890    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efc28864-841e-4451-bfce-5817940b27d5-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:28:04.896006 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:04.895898    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efc28864-841e-4451-bfce-5817940b27d5-tls-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:28:04.896006 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:04.895924    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efc28864-841e-4451-bfce-5817940b27d5-tmp-dir\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:28:04.896006 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:04.895932    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thhxp\" (UniqueName: \"kubernetes.io/projected/efc28864-841e-4451-bfce-5817940b27d5-kube-api-access-thhxp\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:28:05.117083 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:05.117052    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x"]
Sep 17 22:28:05.121933 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:05.121912    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6448cc6bb8-x7z9x"]
Sep 17 22:28:06.685020 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:06.684982    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efc28864-841e-4451-bfce-5817940b27d5" path="/var/lib/kubelet/pods/efc28864-841e-4451-bfce-5817940b27d5/volumes"
Sep 17 22:28:07.842367 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:07.842334    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:28:09.647783 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:09.647754    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fqdr"
Sep 17 22:28:19.734026 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:19.733974    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf"]
Sep 17 22:28:19.734592 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:19.734572    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efc28864-841e-4451-bfce-5817940b27d5" containerName="main"
Sep 17 22:28:19.734592 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:19.734593    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="efc28864-841e-4451-bfce-5817940b27d5" containerName="main"
Sep 17 22:28:19.734723 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:19.734607    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efc28864-841e-4451-bfce-5817940b27d5" containerName="storage-initializer"
Sep 17 22:28:19.734723 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:19.734612    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="efc28864-841e-4451-bfce-5817940b27d5" containerName="storage-initializer"
Sep 17 22:28:19.734723 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:19.734681    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="efc28864-841e-4451-bfce-5817940b27d5" containerName="main"
Sep 17 22:28:19.737600 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:19.737585    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf"
Sep 17 22:28:19.739824 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:19.739802    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 17 22:28:19.746673 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:19.746647    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf"]
Sep 17 22:28:19.759836 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:19.759809    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:28:19.826511 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:19.826484    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kgsl\" (UniqueName: \"kubernetes.io/projected/9afde973-9b07-4237-abf5-2b69e09ddefe-kube-api-access-9kgsl\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf\" (UID: \"9afde973-9b07-4237-abf5-2b69e09ddefe\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf"
Sep 17 22:28:19.826657 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:19.826537    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9afde973-9b07-4237-abf5-2b69e09ddefe-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf\" (UID: \"9afde973-9b07-4237-abf5-2b69e09ddefe\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf"
Sep 17 22:28:19.826657 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:19.826601    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9afde973-9b07-4237-abf5-2b69e09ddefe-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf\" (UID: \"9afde973-9b07-4237-abf5-2b69e09ddefe\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf"
Sep 17 22:28:19.826784 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:19.826665    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9afde973-9b07-4237-abf5-2b69e09ddefe-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf\" (UID: \"9afde973-9b07-4237-abf5-2b69e09ddefe\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf"
Sep 17 22:28:19.826784 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:19.826713    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9afde973-9b07-4237-abf5-2b69e09ddefe-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf\" (UID: \"9afde973-9b07-4237-abf5-2b69e09ddefe\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf"
Sep 17 22:28:19.826784 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:19.826748    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9afde973-9b07-4237-abf5-2b69e09ddefe-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf\" (UID: \"9afde973-9b07-4237-abf5-2b69e09ddefe\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf"
Sep 17 22:28:19.927542 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:19.927512    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9afde973-9b07-4237-abf5-2b69e09ddefe-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf\" (UID: \"9afde973-9b07-4237-abf5-2b69e09ddefe\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf"
Sep 17 22:28:19.927744 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:19.927551    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9afde973-9b07-4237-abf5-2b69e09ddefe-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf\" (UID: \"9afde973-9b07-4237-abf5-2b69e09ddefe\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf"
Sep 17 22:28:19.927744 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:19.927578    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9afde973-9b07-4237-abf5-2b69e09ddefe-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf\" (UID: \"9afde973-9b07-4237-abf5-2b69e09ddefe\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf"
Sep 17 22:28:19.927744 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:19.927604    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9afde973-9b07-4237-abf5-2b69e09ddefe-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf\" (UID: \"9afde973-9b07-4237-abf5-2b69e09ddefe\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf"
Sep 17 22:28:19.927744 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:19.927621    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9afde973-9b07-4237-abf5-2b69e09ddefe-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf\" (UID: \"9afde973-9b07-4237-abf5-2b69e09ddefe\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf"
Sep 17 22:28:19.927744 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:19.927671    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kgsl\" (UniqueName: \"kubernetes.io/projected/9afde973-9b07-4237-abf5-2b69e09ddefe-kube-api-access-9kgsl\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf\" (UID: \"9afde973-9b07-4237-abf5-2b69e09ddefe\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf"
Sep 17 22:28:19.928051 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:19.928012    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9afde973-9b07-4237-abf5-2b69e09ddefe-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf\" (UID: \"9afde973-9b07-4237-abf5-2b69e09ddefe\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf"
Sep 17 22:28:19.928051 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:19.928025    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9afde973-9b07-4237-abf5-2b69e09ddefe-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf\" (UID: \"9afde973-9b07-4237-abf5-2b69e09ddefe\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf"
Sep 17 22:28:19.928125 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:19.928073    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9afde973-9b07-4237-abf5-2b69e09ddefe-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf\" (UID: \"9afde973-9b07-4237-abf5-2b69e09ddefe\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf"
Sep 17 22:28:19.929903 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:19.929884    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9afde973-9b07-4237-abf5-2b69e09ddefe-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf\" (UID: \"9afde973-9b07-4237-abf5-2b69e09ddefe\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf"
Sep 17 22:28:19.930131 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:19.930115    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9afde973-9b07-4237-abf5-2b69e09ddefe-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf\" (UID: \"9afde973-9b07-4237-abf5-2b69e09ddefe\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf"
Sep 17 22:28:19.935097 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:19.935077    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kgsl\" (UniqueName: \"kubernetes.io/projected/9afde973-9b07-4237-abf5-2b69e09ddefe-kube-api-access-9kgsl\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf\" (UID: \"9afde973-9b07-4237-abf5-2b69e09ddefe\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf"
Sep 17 22:28:20.018796 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:20.018707    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:28:20.048752 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:20.048720    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf"
Sep 17 22:28:20.323720 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:20.323695    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf"]
Sep 17 22:28:20.325435 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:28:20.325404    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9afde973_9b07_4237_abf5_2b69e09ddefe.slice/crio-b9c020f2783b22e9f74874cf1346e7abb9518e8ced07cf7b353cdbac33fce551 WatchSource:0}: Error finding container b9c020f2783b22e9f74874cf1346e7abb9518e8ced07cf7b353cdbac33fce551: Status 404 returned error can't find the container with id b9c020f2783b22e9f74874cf1346e7abb9518e8ced07cf7b353cdbac33fce551
Sep 17 22:28:20.894321 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:20.894284    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf" event={"ID":"9afde973-9b07-4237-abf5-2b69e09ddefe","Type":"ContainerStarted","Data":"1edcb06fef5a495b4aec22e1bba745ecb2cc2f2a5bd83737efb9a8bb4f230f34"}
Sep 17 22:28:20.894705 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:20.894341    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf" event={"ID":"9afde973-9b07-4237-abf5-2b69e09ddefe","Type":"ContainerStarted","Data":"b9c020f2783b22e9f74874cf1346e7abb9518e8ced07cf7b353cdbac33fce551"}
Sep 17 22:28:20.911833 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:20.911782    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf" podStartSLOduration=1.9117667360000001 podStartE2EDuration="1.911766736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:28:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:28:20.909982417 +0000 UTC m=+1802.356027698" watchObservedRunningTime="2026-09-17 22:28:20.911766736 +0000 UTC m=+1802.357812006"
Sep 17 22:28:21.048939 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:21.048904    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf"
Sep 17 22:28:22.744302 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:22.744275    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-7njvj_a324c9ab-5d42-4d31-8ff7-449f619be774/console-operator/1.log"
Sep 17 22:28:22.748401 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:22.748380    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s7f5_4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf/ovn-acl-logging/0.log"
Sep 17 22:28:22.749575 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:22.749543    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-7njvj_a324c9ab-5d42-4d31-8ff7-449f619be774/console-operator/1.log"
Sep 17 22:28:22.753967 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:22.753948    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s7f5_4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf/ovn-acl-logging/0.log"
Sep 17 22:28:30.049045 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:30.049009    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf"
Sep 17 22:28:30.061198 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:30.061158    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf"
Sep 17 22:28:30.962177 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:30.962146    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf"
Sep 17 22:28:32.871716 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:32.871687    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:28:34.505673 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:34.505642    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:28:34.602278 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:34.602247    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:28:34.869144 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:34.869115    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:28:34.945802 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:34.945774    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:28:38.292360 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:38.292328    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:28:40.660062 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:40.660021    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:28:42.880913 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:42.880864    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:28:42.981689 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:42.981655    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:28:45.159698 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:45.159663    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:28:46.640612 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:46.640572    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:28:46.703853 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:28:46.703821    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:29:03.840424 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:03.840387    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:29:07.397020 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:07.396660    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:29:07.398286 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:07.398246    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf"]
Sep 17 22:29:07.414897 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:29:07.414857    2555 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 22:29:07.414897 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:29:07.414897 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:29:07.414897 ip-10-0-140-157 kubenswrapper[2555]:  > containerID="1edcb06fef5a495b4aec22e1bba745ecb2cc2f2a5bd83737efb9a8bb4f230f34" cmd=["/bin/sleep","15"]
Sep 17 22:29:07.415280 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:07.414918    2555 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf" message="" err=<
Sep 17 22:29:07.415280 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:29:07.415280 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:29:07.415280 ip-10-0-140-157 kubenswrapper[2555]:  >
Sep 17 22:29:07.415280 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:29:07.414935    2555 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 22:29:07.415280 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:29:07.415280 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:29:07.415280 ip-10-0-140-157 kubenswrapper[2555]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf" podUID="9afde973-9b07-4237-abf5-2b69e09ddefe" containerName="main" containerID="cri-o://1edcb06fef5a495b4aec22e1bba745ecb2cc2f2a5bd83737efb9a8bb4f230f34"
Sep 17 22:29:07.415280 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:07.414993    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf" podUID="9afde973-9b07-4237-abf5-2b69e09ddefe" containerName="main" containerID="cri-o://1edcb06fef5a495b4aec22e1bba745ecb2cc2f2a5bd83737efb9a8bb4f230f34" gracePeriod=60
Sep 17 22:29:07.496685 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:07.496656    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:29:07.952111 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:07.952066    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf" podUID="9afde973-9b07-4237-abf5-2b69e09ddefe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Sep 17 22:29:08.160713 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:08.160675    2555 generic.go:356] "Generic (PLEG): container finished" podID="9afde973-9b07-4237-abf5-2b69e09ddefe" containerID="1edcb06fef5a495b4aec22e1bba745ecb2cc2f2a5bd83737efb9a8bb4f230f34" exitCode=0
Sep 17 22:29:08.160713 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:08.160702    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf" event={"ID":"9afde973-9b07-4237-abf5-2b69e09ddefe","Type":"ContainerDied","Data":"1edcb06fef5a495b4aec22e1bba745ecb2cc2f2a5bd83737efb9a8bb4f230f34"}
Sep 17 22:29:08.175874 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:08.175848    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf"
Sep 17 22:29:08.266524 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:08.266439    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9afde973-9b07-4237-abf5-2b69e09ddefe-dshm\") pod \"9afde973-9b07-4237-abf5-2b69e09ddefe\" (UID: \"9afde973-9b07-4237-abf5-2b69e09ddefe\") "
Sep 17 22:29:08.266524 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:08.266485    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9afde973-9b07-4237-abf5-2b69e09ddefe-home\") pod \"9afde973-9b07-4237-abf5-2b69e09ddefe\" (UID: \"9afde973-9b07-4237-abf5-2b69e09ddefe\") "
Sep 17 22:29:08.266703 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:08.266552    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9afde973-9b07-4237-abf5-2b69e09ddefe-tmp-dir\") pod \"9afde973-9b07-4237-abf5-2b69e09ddefe\" (UID: \"9afde973-9b07-4237-abf5-2b69e09ddefe\") "
Sep 17 22:29:08.266703 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:08.266588    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9afde973-9b07-4237-abf5-2b69e09ddefe-model-cache\") pod \"9afde973-9b07-4237-abf5-2b69e09ddefe\" (UID: \"9afde973-9b07-4237-abf5-2b69e09ddefe\") "
Sep 17 22:29:08.266703 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:08.266612    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9afde973-9b07-4237-abf5-2b69e09ddefe-tls-certs\") pod \"9afde973-9b07-4237-abf5-2b69e09ddefe\" (UID: \"9afde973-9b07-4237-abf5-2b69e09ddefe\") "
Sep 17 22:29:08.266703 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:08.266663    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kgsl\" (UniqueName: \"kubernetes.io/projected/9afde973-9b07-4237-abf5-2b69e09ddefe-kube-api-access-9kgsl\") pod \"9afde973-9b07-4237-abf5-2b69e09ddefe\" (UID: \"9afde973-9b07-4237-abf5-2b69e09ddefe\") "
Sep 17 22:29:08.266898 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:08.266804    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9afde973-9b07-4237-abf5-2b69e09ddefe-home" (OuterVolumeSpecName: "home") pod "9afde973-9b07-4237-abf5-2b69e09ddefe" (UID: "9afde973-9b07-4237-abf5-2b69e09ddefe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:29:08.266898 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:08.266822    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9afde973-9b07-4237-abf5-2b69e09ddefe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9afde973-9b07-4237-abf5-2b69e09ddefe" (UID: "9afde973-9b07-4237-abf5-2b69e09ddefe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:29:08.266898 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:08.266879    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9afde973-9b07-4237-abf5-2b69e09ddefe-model-cache" (OuterVolumeSpecName: "model-cache") pod "9afde973-9b07-4237-abf5-2b69e09ddefe" (UID: "9afde973-9b07-4237-abf5-2b69e09ddefe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:29:08.267059 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:08.266991    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9afde973-9b07-4237-abf5-2b69e09ddefe-tmp-dir\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:29:08.267059 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:08.267010    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9afde973-9b07-4237-abf5-2b69e09ddefe-model-cache\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:29:08.267059 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:08.267024    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9afde973-9b07-4237-abf5-2b69e09ddefe-home\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:29:08.268790 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:08.268760    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9afde973-9b07-4237-abf5-2b69e09ddefe-dshm" (OuterVolumeSpecName: "dshm") pod "9afde973-9b07-4237-abf5-2b69e09ddefe" (UID: "9afde973-9b07-4237-abf5-2b69e09ddefe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:29:08.268940 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:08.268809    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9afde973-9b07-4237-abf5-2b69e09ddefe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9afde973-9b07-4237-abf5-2b69e09ddefe" (UID: "9afde973-9b07-4237-abf5-2b69e09ddefe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:29:08.268940 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:08.268906    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9afde973-9b07-4237-abf5-2b69e09ddefe-kube-api-access-9kgsl" (OuterVolumeSpecName: "kube-api-access-9kgsl") pod "9afde973-9b07-4237-abf5-2b69e09ddefe" (UID: "9afde973-9b07-4237-abf5-2b69e09ddefe"). InnerVolumeSpecName "kube-api-access-9kgsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:29:08.368216 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:08.368167    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9afde973-9b07-4237-abf5-2b69e09ddefe-tls-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:29:08.368216 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:08.368213    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kgsl\" (UniqueName: \"kubernetes.io/projected/9afde973-9b07-4237-abf5-2b69e09ddefe-kube-api-access-9kgsl\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:29:08.368216 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:08.368224    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9afde973-9b07-4237-abf5-2b69e09ddefe-dshm\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:29:09.169137 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:09.169104    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf"
Sep 17 22:29:09.169610 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:09.169111    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf" event={"ID":"9afde973-9b07-4237-abf5-2b69e09ddefe","Type":"ContainerDied","Data":"b9c020f2783b22e9f74874cf1346e7abb9518e8ced07cf7b353cdbac33fce551"}
Sep 17 22:29:09.169610 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:09.169249    2555 scope.go:117] "RemoveContainer" containerID="1edcb06fef5a495b4aec22e1bba745ecb2cc2f2a5bd83737efb9a8bb4f230f34"
Sep 17 22:29:09.185475 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:09.185451    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf"]
Sep 17 22:29:09.190708 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:09.190685    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995s5xtf"]
Sep 17 22:29:10.679703 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:10.679670    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9afde973-9b07-4237-abf5-2b69e09ddefe" path="/var/lib/kubelet/pods/9afde973-9b07-4237-abf5-2b69e09ddefe/volumes"
Sep 17 22:29:57.085636 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:57.085529    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f"]
Sep 17 22:29:57.088557 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:57.088526    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fqdr"]
Sep 17 22:29:57.098573 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:29:57.098480    2555 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 22:29:57.098573 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:29:57.098573 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:29:57.098573 ip-10-0-140-157 kubenswrapper[2555]:  > containerID="222728302c6d7ff7f193f65ea669eff803cdc9d983bd56504200c23722713a9b" cmd=["/bin/sleep","15"]
Sep 17 22:29:57.098573 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:57.098542    2555 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f" message="" err=<
Sep 17 22:29:57.098573 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:29:57.098573 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:29:57.098573 ip-10-0-140-157 kubenswrapper[2555]:  >
Sep 17 22:29:57.098573 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:29:57.098558    2555 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 22:29:57.098573 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:29:57.098573 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:29:57.098573 ip-10-0-140-157 kubenswrapper[2555]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f" podUID="e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8" containerName="main" containerID="cri-o://222728302c6d7ff7f193f65ea669eff803cdc9d983bd56504200c23722713a9b"
Sep 17 22:29:57.099283 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:57.098612    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f" podUID="e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8" containerName="main" containerID="cri-o://222728302c6d7ff7f193f65ea669eff803cdc9d983bd56504200c23722713a9b" gracePeriod=60
Sep 17 22:29:57.112113 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:57.112089    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:29:57.173967 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:57.173929    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:29:57.440136 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:57.440106    2555 generic.go:356] "Generic (PLEG): container finished" podID="e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8" containerID="222728302c6d7ff7f193f65ea669eff803cdc9d983bd56504200c23722713a9b" exitCode=0
Sep 17 22:29:57.440287 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:57.440197    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f" event={"ID":"e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8","Type":"ContainerDied","Data":"222728302c6d7ff7f193f65ea669eff803cdc9d983bd56504200c23722713a9b"}
Sep 17 22:29:57.460726 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:57.460703    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f"
Sep 17 22:29:57.607631 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:57.607590    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8-tmp-dir\") pod \"e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8\" (UID: \"e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8\") "
Sep 17 22:29:57.607857 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:57.607653    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8-dshm\") pod \"e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8\" (UID: \"e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8\") "
Sep 17 22:29:57.607857 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:57.607729    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxcnl\" (UniqueName: \"kubernetes.io/projected/e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8-kube-api-access-qxcnl\") pod \"e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8\" (UID: \"e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8\") "
Sep 17 22:29:57.607857 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:57.607798    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8-tls-certs\") pod \"e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8\" (UID: \"e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8\") "
Sep 17 22:29:57.608035 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:57.607883    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8-home\") pod \"e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8\" (UID: \"e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8\") "
Sep 17 22:29:57.608035 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:57.607906    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8-model-cache\") pod \"e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8\" (UID: \"e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8\") "
Sep 17 22:29:57.608345 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:57.608317    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8" (UID: "e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:29:57.608467 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:57.608421    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8-home" (OuterVolumeSpecName: "home") pod "e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8" (UID: "e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:29:57.608539 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:57.608500    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8-model-cache" (OuterVolumeSpecName: "model-cache") pod "e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8" (UID: "e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:29:57.610171 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:57.610141    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8-dshm" (OuterVolumeSpecName: "dshm") pod "e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8" (UID: "e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:29:57.610402 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:57.610354    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8-kube-api-access-qxcnl" (OuterVolumeSpecName: "kube-api-access-qxcnl") pod "e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8" (UID: "e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8"). InnerVolumeSpecName "kube-api-access-qxcnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:29:57.610402 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:57.610368    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8" (UID: "e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:29:57.709010 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:57.708974    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8-home\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:29:57.709010 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:57.709002    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8-model-cache\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:29:57.709010 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:57.709011    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8-tmp-dir\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:29:57.709010 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:57.709019    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8-dshm\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:29:57.709331 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:57.709027    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxcnl\" (UniqueName: \"kubernetes.io/projected/e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8-kube-api-access-qxcnl\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:29:57.709331 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:57.709035    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8-tls-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:29:58.447421 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:58.447392    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f"
Sep 17 22:29:58.447421 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:58.447408    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f" event={"ID":"e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8","Type":"ContainerDied","Data":"6e16cf019a6764a470372c406bb211642fba18a0fc8e6841e10be4d9c8a14fcc"}
Sep 17 22:29:58.447976 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:58.447464    2555 scope.go:117] "RemoveContainer" containerID="222728302c6d7ff7f193f65ea669eff803cdc9d983bd56504200c23722713a9b"
Sep 17 22:29:58.469413 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:58.469379    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f"]
Sep 17 22:29:58.477308 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:58.477279    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fgj4f"]
Sep 17 22:29:58.685738 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:58.685700    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8" path="/var/lib/kubelet/pods/e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8/volumes"
Sep 17 22:29:59.647284 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:29:59.647245    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fqdr" podUID="f8e728a1-e0a5-40ea-b0cc-fd2c5a2d1369" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:30:09.647826 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:09.647784    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fqdr" podUID="f8e728a1-e0a5-40ea-b0cc-fd2c5a2d1369" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:30:12.109948 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:12.109896    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fqdr" podUID="f8e728a1-e0a5-40ea-b0cc-fd2c5a2d1369" containerName="main" containerID="cri-o://ed0a3c2480c292ab740196b1a456b5bfc42a4dc8f743f8b956c2440cf5dce410" gracePeriod=45
Sep 17 22:30:19.647052 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:19.647013    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fqdr" podUID="f8e728a1-e0a5-40ea-b0cc-fd2c5a2d1369" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:30:19.647475 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:19.647125    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fqdr"
Sep 17 22:30:29.647744 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:29.647704    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fqdr" podUID="f8e728a1-e0a5-40ea-b0cc-fd2c5a2d1369" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:30:39.646911 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:39.646863    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fqdr" podUID="f8e728a1-e0a5-40ea-b0cc-fd2c5a2d1369" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:30:42.359917 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:42.359891    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fqdr"
Sep 17 22:30:42.483106 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:42.483062    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwgtc\" (UniqueName: \"kubernetes.io/projected/f8e728a1-e0a5-40ea-b0cc-fd2c5a2d1369-kube-api-access-qwgtc\") pod \"f8e728a1-e0a5-40ea-b0cc-fd2c5a2d1369\" (UID: \"f8e728a1-e0a5-40ea-b0cc-fd2c5a2d1369\") "
Sep 17 22:30:42.483333 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:42.483128    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8e728a1-e0a5-40ea-b0cc-fd2c5a2d1369-tls-certs\") pod \"f8e728a1-e0a5-40ea-b0cc-fd2c5a2d1369\" (UID: \"f8e728a1-e0a5-40ea-b0cc-fd2c5a2d1369\") "
Sep 17 22:30:42.485261 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:42.485234    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8e728a1-e0a5-40ea-b0cc-fd2c5a2d1369-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f8e728a1-e0a5-40ea-b0cc-fd2c5a2d1369" (UID: "f8e728a1-e0a5-40ea-b0cc-fd2c5a2d1369"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:30:42.485374 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:42.485340    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8e728a1-e0a5-40ea-b0cc-fd2c5a2d1369-kube-api-access-qwgtc" (OuterVolumeSpecName: "kube-api-access-qwgtc") pod "f8e728a1-e0a5-40ea-b0cc-fd2c5a2d1369" (UID: "f8e728a1-e0a5-40ea-b0cc-fd2c5a2d1369"). InnerVolumeSpecName "kube-api-access-qwgtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:30:42.583978 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:42.583940    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwgtc\" (UniqueName: \"kubernetes.io/projected/f8e728a1-e0a5-40ea-b0cc-fd2c5a2d1369-kube-api-access-qwgtc\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:30:42.583978 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:42.583972    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8e728a1-e0a5-40ea-b0cc-fd2c5a2d1369-tls-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:30:42.694851 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:42.694820    2555 generic.go:356] "Generic (PLEG): container finished" podID="f8e728a1-e0a5-40ea-b0cc-fd2c5a2d1369" containerID="ed0a3c2480c292ab740196b1a456b5bfc42a4dc8f743f8b956c2440cf5dce410" exitCode=0
Sep 17 22:30:42.694984 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:42.694858    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fqdr"
Sep 17 22:30:42.694984 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:42.694898    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fqdr" event={"ID":"f8e728a1-e0a5-40ea-b0cc-fd2c5a2d1369","Type":"ContainerDied","Data":"ed0a3c2480c292ab740196b1a456b5bfc42a4dc8f743f8b956c2440cf5dce410"}
Sep 17 22:30:42.694984 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:42.694945    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fqdr" event={"ID":"f8e728a1-e0a5-40ea-b0cc-fd2c5a2d1369","Type":"ContainerDied","Data":"4f7391ee78876cb12710435ff07953cc05dc3119d032e8119e66c1776d770dca"}
Sep 17 22:30:42.694984 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:42.694968    2555 scope.go:117] "RemoveContainer" containerID="ed0a3c2480c292ab740196b1a456b5bfc42a4dc8f743f8b956c2440cf5dce410"
Sep 17 22:30:42.704189 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:42.704159    2555 scope.go:117] "RemoveContainer" containerID="ed0a3c2480c292ab740196b1a456b5bfc42a4dc8f743f8b956c2440cf5dce410"
Sep 17 22:30:42.704448 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:30:42.704430    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed0a3c2480c292ab740196b1a456b5bfc42a4dc8f743f8b956c2440cf5dce410\": container with ID starting with ed0a3c2480c292ab740196b1a456b5bfc42a4dc8f743f8b956c2440cf5dce410 not found: ID does not exist" containerID="ed0a3c2480c292ab740196b1a456b5bfc42a4dc8f743f8b956c2440cf5dce410"
Sep 17 22:30:42.704496 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:42.704455    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed0a3c2480c292ab740196b1a456b5bfc42a4dc8f743f8b956c2440cf5dce410"} err="failed to get container status \"ed0a3c2480c292ab740196b1a456b5bfc42a4dc8f743f8b956c2440cf5dce410\": rpc error: code = NotFound desc = could not find container \"ed0a3c2480c292ab740196b1a456b5bfc42a4dc8f743f8b956c2440cf5dce410\": container with ID starting with ed0a3c2480c292ab740196b1a456b5bfc42a4dc8f743f8b956c2440cf5dce410 not found: ID does not exist"
Sep 17 22:30:42.710711 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:42.710688    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fqdr"]
Sep 17 22:30:42.714238 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:42.714216    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fqdr"]
Sep 17 22:30:44.679240 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:44.679208    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8e728a1-e0a5-40ea-b0cc-fd2c5a2d1369" path="/var/lib/kubelet/pods/f8e728a1-e0a5-40ea-b0cc-fd2c5a2d1369/volumes"
Sep 17 22:30:47.158067 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:47.158035    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:30:47.256325 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:47.256295    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:30:53.602077 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:53.602036    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:30:57.020628 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:57.020588    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv"]
Sep 17 22:30:57.021518 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:57.021491    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9afde973-9b07-4237-abf5-2b69e09ddefe" containerName="main"
Sep 17 22:30:57.021518 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:57.021519    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9afde973-9b07-4237-abf5-2b69e09ddefe" containerName="main"
Sep 17 22:30:57.021706 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:57.021584    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8" containerName="main"
Sep 17 22:30:57.021706 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:57.021593    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8" containerName="main"
Sep 17 22:30:57.021706 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:57.021609    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8e728a1-e0a5-40ea-b0cc-fd2c5a2d1369" containerName="main"
Sep 17 22:30:57.021706 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:57.021619    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8e728a1-e0a5-40ea-b0cc-fd2c5a2d1369" containerName="main"
Sep 17 22:30:57.021914 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:57.021806    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8e728a1-e0a5-40ea-b0cc-fd2c5a2d1369" containerName="main"
Sep 17 22:30:57.021914 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:57.021822    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e821d4ae-7c6e-4b32-a1b7-867e0ca6bea8" containerName="main"
Sep 17 22:30:57.021914 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:57.021840    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9afde973-9b07-4237-abf5-2b69e09ddefe" containerName="main"
Sep 17 22:30:57.028016 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:57.027462    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv"
Sep 17 22:30:57.031864 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:57.031840    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 17 22:30:57.032078 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:57.031894    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 17 22:30:57.032609 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:57.032589    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-b7d4m\""
Sep 17 22:30:57.032734 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:57.032601    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 17 22:30:57.033706 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:57.033673    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv"]
Sep 17 22:30:57.056492 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:57.056468    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:30:57.105556 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:57.105527    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zxrc\" (UniqueName: \"kubernetes.io/projected/1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a-kube-api-access-9zxrc\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv\" (UID: \"1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv"
Sep 17 22:30:57.105689 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:57.105561    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv\" (UID: \"1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv"
Sep 17 22:30:57.105689 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:57.105586    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv\" (UID: \"1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv"
Sep 17 22:30:57.105689 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:57.105601    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv\" (UID: \"1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv"
Sep 17 22:30:57.105814 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:57.105704    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv\" (UID: \"1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv"
Sep 17 22:30:57.105814 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:57.105730    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv\" (UID: \"1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv"
Sep 17 22:30:57.206947 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:57.206915    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zxrc\" (UniqueName: \"kubernetes.io/projected/1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a-kube-api-access-9zxrc\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv\" (UID: \"1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv"
Sep 17 22:30:57.207130 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:57.206961    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv\" (UID: \"1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv"
Sep 17 22:30:57.207130 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:57.207000    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv\" (UID: \"1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv"
Sep 17 22:30:57.207130 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:57.207022    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv\" (UID: \"1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv"
Sep 17 22:30:57.207130 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:57.207115    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv\" (UID: \"1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv"
Sep 17 22:30:57.207386 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:57.207136    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv\" (UID: \"1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv"
Sep 17 22:30:57.207602 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:57.207573    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv\" (UID: \"1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv"
Sep 17 22:30:57.207864 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:57.207843    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv\" (UID: \"1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv"
Sep 17 22:30:57.208092 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:57.208067    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv\" (UID: \"1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv"
Sep 17 22:30:57.209915 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:57.209889    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv\" (UID: \"1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv"
Sep 17 22:30:57.210210 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:57.210176    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv\" (UID: \"1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv"
Sep 17 22:30:57.220020 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:57.219998    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zxrc\" (UniqueName: \"kubernetes.io/projected/1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a-kube-api-access-9zxrc\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv\" (UID: \"1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv"
Sep 17 22:30:57.240044 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:57.240022    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:30:57.340816 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:57.340738    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv"
Sep 17 22:30:57.614709 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:57.614684    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv"]
Sep 17 22:30:57.616448 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:30:57.616416    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c6b5a02_ad7e_4f4f_99bc_972fe3ae2f2a.slice/crio-4d340f4bb99566cc6cc027359f0a559427652c5b8e65b744a94ca5d9e7fc42f0 WatchSource:0}: Error finding container 4d340f4bb99566cc6cc027359f0a559427652c5b8e65b744a94ca5d9e7fc42f0: Status 404 returned error can't find the container with id 4d340f4bb99566cc6cc027359f0a559427652c5b8e65b744a94ca5d9e7fc42f0
Sep 17 22:30:57.783602 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:57.783553    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv" event={"ID":"1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a","Type":"ContainerStarted","Data":"c27673f17c3e8946f85272b45445c23e507dfd524b52d42d79d9f903ada4eb6d"}
Sep 17 22:30:57.783771 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:57.783617    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv" event={"ID":"1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a","Type":"ContainerStarted","Data":"4d340f4bb99566cc6cc027359f0a559427652c5b8e65b744a94ca5d9e7fc42f0"}
Sep 17 22:30:57.801416 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:57.801362    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv" podStartSLOduration=0.801347915 podStartE2EDuration="801.347915ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:30:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:30:57.799371641 +0000 UTC m=+1959.245416913" watchObservedRunningTime="2026-09-17 22:30:57.801347915 +0000 UTC m=+1959.247393186"
Sep 17 22:30:58.341322 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:58.341289    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv"
Sep 17 22:30:59.270836 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:59.270796    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:30:59.372872 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:30:59.372831    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:31:07.341309 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:31:07.341218    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv"
Sep 17 22:31:07.354074 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:31:07.354052    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv"
Sep 17 22:31:07.863590 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:31:07.863556    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv"
Sep 17 22:31:25.975465 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:31:25.975418    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:31:27.472130 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:31:27.472086    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:31:30.477702 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:31:30.477670    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:31:30.616063 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:31:30.616029    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:31:35.461597 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:31:35.461560    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:31:40.742480 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:31:40.742451    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94"]
Sep 17 22:31:40.747383 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:31:40.747362    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94"
Sep 17 22:31:40.749600 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:31:40.749578    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 17 22:31:40.755332 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:31:40.755310    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94"]
Sep 17 22:31:40.874067 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:31:40.874028    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6c95c24-6280-4080-9d11-510014886369-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94\" (UID: \"b6c95c24-6280-4080-9d11-510014886369\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94"
Sep 17 22:31:40.874067 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:31:40.874068    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6c95c24-6280-4080-9d11-510014886369-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94\" (UID: \"b6c95c24-6280-4080-9d11-510014886369\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94"
Sep 17 22:31:40.874316 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:31:40.874089    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6c95c24-6280-4080-9d11-510014886369-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94\" (UID: \"b6c95c24-6280-4080-9d11-510014886369\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94"
Sep 17 22:31:40.874316 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:31:40.874109    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq2m4\" (UniqueName: \"kubernetes.io/projected/b6c95c24-6280-4080-9d11-510014886369-kube-api-access-fq2m4\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94\" (UID: \"b6c95c24-6280-4080-9d11-510014886369\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94"
Sep 17 22:31:40.874316 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:31:40.874208    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6c95c24-6280-4080-9d11-510014886369-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94\" (UID: \"b6c95c24-6280-4080-9d11-510014886369\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94"
Sep 17 22:31:40.874316 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:31:40.874237    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6c95c24-6280-4080-9d11-510014886369-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94\" (UID: \"b6c95c24-6280-4080-9d11-510014886369\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94"
Sep 17 22:31:40.904482 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:31:40.904457    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:31:40.975084 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:31:40.975012    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6c95c24-6280-4080-9d11-510014886369-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94\" (UID: \"b6c95c24-6280-4080-9d11-510014886369\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94"
Sep 17 22:31:40.975084 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:31:40.975045    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6c95c24-6280-4080-9d11-510014886369-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94\" (UID: \"b6c95c24-6280-4080-9d11-510014886369\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94"
Sep 17 22:31:40.975084 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:31:40.975065    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6c95c24-6280-4080-9d11-510014886369-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94\" (UID: \"b6c95c24-6280-4080-9d11-510014886369\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94"
Sep 17 22:31:40.975365 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:31:40.975085    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq2m4\" (UniqueName: \"kubernetes.io/projected/b6c95c24-6280-4080-9d11-510014886369-kube-api-access-fq2m4\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94\" (UID: \"b6c95c24-6280-4080-9d11-510014886369\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94"
Sep 17 22:31:40.975365 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:31:40.975130    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6c95c24-6280-4080-9d11-510014886369-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94\" (UID: \"b6c95c24-6280-4080-9d11-510014886369\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94"
Sep 17 22:31:40.975365 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:31:40.975155    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6c95c24-6280-4080-9d11-510014886369-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94\" (UID: \"b6c95c24-6280-4080-9d11-510014886369\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94"
Sep 17 22:31:40.975596 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:31:40.975566    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6c95c24-6280-4080-9d11-510014886369-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94\" (UID: \"b6c95c24-6280-4080-9d11-510014886369\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94"
Sep 17 22:31:40.975596 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:31:40.975588    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6c95c24-6280-4080-9d11-510014886369-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94\" (UID: \"b6c95c24-6280-4080-9d11-510014886369\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94"
Sep 17 22:31:40.975794 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:31:40.975649    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6c95c24-6280-4080-9d11-510014886369-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94\" (UID: \"b6c95c24-6280-4080-9d11-510014886369\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94"
Sep 17 22:31:40.977568 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:31:40.977543    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6c95c24-6280-4080-9d11-510014886369-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94\" (UID: \"b6c95c24-6280-4080-9d11-510014886369\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94"
Sep 17 22:31:40.977784 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:31:40.977765    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6c95c24-6280-4080-9d11-510014886369-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94\" (UID: \"b6c95c24-6280-4080-9d11-510014886369\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94"
Sep 17 22:31:40.986571 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:31:40.986547    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq2m4\" (UniqueName: \"kubernetes.io/projected/b6c95c24-6280-4080-9d11-510014886369-kube-api-access-fq2m4\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94\" (UID: \"b6c95c24-6280-4080-9d11-510014886369\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94"
Sep 17 22:31:41.057769 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:31:41.057740    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94"
Sep 17 22:31:41.220318 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:31:41.220283    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:31:41.332936 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:31:41.332912    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94"]
Sep 17 22:31:41.335163 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:31:41.335138    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6c95c24_6280_4080_9d11_510014886369.slice/crio-47541182c803ee4577fcfa3dad79bad755dd5b795cd393965243ffd09dbfee86 WatchSource:0}: Error finding container 47541182c803ee4577fcfa3dad79bad755dd5b795cd393965243ffd09dbfee86: Status 404 returned error can't find the container with id 47541182c803ee4577fcfa3dad79bad755dd5b795cd393965243ffd09dbfee86
Sep 17 22:31:42.041493 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:31:42.041458    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94" event={"ID":"b6c95c24-6280-4080-9d11-510014886369","Type":"ContainerStarted","Data":"138ebdd09f55167e985e424d21cc5e4a58eb28d874d911313963763bda2cdad1"}
Sep 17 22:31:42.041971 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:31:42.041523    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94" event={"ID":"b6c95c24-6280-4080-9d11-510014886369","Type":"ContainerStarted","Data":"47541182c803ee4577fcfa3dad79bad755dd5b795cd393965243ffd09dbfee86"}
Sep 17 22:31:42.058326 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:31:42.058267    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94" podStartSLOduration=2.058250431 podStartE2EDuration="2.058250431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:31:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:31:42.057254608 +0000 UTC m=+2003.503299880" watchObservedRunningTime="2026-09-17 22:31:42.058250431 +0000 UTC m=+2003.504295704"
Sep 17 22:31:42.058513 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:31:42.058492    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94"
Sep 17 22:31:51.058123 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:31:51.058090    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94"
Sep 17 22:31:51.070942 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:31:51.070918    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94"
Sep 17 22:31:51.103684 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:31:51.103662    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94"
Sep 17 22:31:56.548414 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:31:56.548378    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:31:58.024471 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:31:58.024431    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:32:03.442359 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:32:03.442320    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:32:07.931099 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:32:07.931071    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:32:09.075611 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:32:09.075577    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:32:09.145968 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:32:09.145935    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:32:10.511739 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:32:10.511710    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:32:10.697557 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:32:10.697527    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:32:19.810279 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:32:19.810248    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:32:23.877340 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:32:23.877302    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:32:32.858878 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:32:32.858839    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:32:37.546403 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:32:37.546362    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:32:37.552897 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:32:37.552869    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94"]
Sep 17 22:32:37.562566 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:32:37.562535    2555 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 22:32:37.562566 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:32:37.562566 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:32:37.562566 ip-10-0-140-157 kubenswrapper[2555]:  > containerID="138ebdd09f55167e985e424d21cc5e4a58eb28d874d911313963763bda2cdad1" cmd=["/bin/sleep","15"]
Sep 17 22:32:37.562879 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:32:37.562582    2555 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94" message="" err=<
Sep 17 22:32:37.562879 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:32:37.562879 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:32:37.562879 ip-10-0-140-157 kubenswrapper[2555]:  >
Sep 17 22:32:37.562879 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:32:37.562603    2555 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 22:32:37.562879 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:32:37.562879 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:32:37.562879 ip-10-0-140-157 kubenswrapper[2555]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94" podUID="b6c95c24-6280-4080-9d11-510014886369" containerName="main" containerID="cri-o://138ebdd09f55167e985e424d21cc5e4a58eb28d874d911313963763bda2cdad1"
Sep 17 22:32:37.562879 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:32:37.562650    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94" podUID="b6c95c24-6280-4080-9d11-510014886369" containerName="main" containerID="cri-o://138ebdd09f55167e985e424d21cc5e4a58eb28d874d911313963763bda2cdad1" gracePeriod=60
Sep 17 22:32:37.642652 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:32:37.642622    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:32:37.917814 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:32:37.917788    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94"
Sep 17 22:32:38.068046 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:32:38.068012    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6c95c24-6280-4080-9d11-510014886369-tmp-dir\") pod \"b6c95c24-6280-4080-9d11-510014886369\" (UID: \"b6c95c24-6280-4080-9d11-510014886369\") "
Sep 17 22:32:38.068046 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:32:38.068050    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6c95c24-6280-4080-9d11-510014886369-model-cache\") pod \"b6c95c24-6280-4080-9d11-510014886369\" (UID: \"b6c95c24-6280-4080-9d11-510014886369\") "
Sep 17 22:32:38.068332 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:32:38.068079    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6c95c24-6280-4080-9d11-510014886369-home\") pod \"b6c95c24-6280-4080-9d11-510014886369\" (UID: \"b6c95c24-6280-4080-9d11-510014886369\") "
Sep 17 22:32:38.068332 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:32:38.068128    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6c95c24-6280-4080-9d11-510014886369-dshm\") pod \"b6c95c24-6280-4080-9d11-510014886369\" (UID: \"b6c95c24-6280-4080-9d11-510014886369\") "
Sep 17 22:32:38.068332 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:32:38.068145    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fq2m4\" (UniqueName: \"kubernetes.io/projected/b6c95c24-6280-4080-9d11-510014886369-kube-api-access-fq2m4\") pod \"b6c95c24-6280-4080-9d11-510014886369\" (UID: \"b6c95c24-6280-4080-9d11-510014886369\") "
Sep 17 22:32:38.068332 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:32:38.068178    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6c95c24-6280-4080-9d11-510014886369-tls-certs\") pod \"b6c95c24-6280-4080-9d11-510014886369\" (UID: \"b6c95c24-6280-4080-9d11-510014886369\") "
Sep 17 22:32:38.068518 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:32:38.068399    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6c95c24-6280-4080-9d11-510014886369-model-cache" (OuterVolumeSpecName: "model-cache") pod "b6c95c24-6280-4080-9d11-510014886369" (UID: "b6c95c24-6280-4080-9d11-510014886369"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:32:38.068518 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:32:38.068412    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6c95c24-6280-4080-9d11-510014886369-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b6c95c24-6280-4080-9d11-510014886369" (UID: "b6c95c24-6280-4080-9d11-510014886369"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:32:38.068518 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:32:38.068436    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6c95c24-6280-4080-9d11-510014886369-home" (OuterVolumeSpecName: "home") pod "b6c95c24-6280-4080-9d11-510014886369" (UID: "b6c95c24-6280-4080-9d11-510014886369"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:32:38.070605 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:32:38.070576    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6c95c24-6280-4080-9d11-510014886369-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b6c95c24-6280-4080-9d11-510014886369" (UID: "b6c95c24-6280-4080-9d11-510014886369"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:32:38.070605 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:32:38.070577    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6c95c24-6280-4080-9d11-510014886369-kube-api-access-fq2m4" (OuterVolumeSpecName: "kube-api-access-fq2m4") pod "b6c95c24-6280-4080-9d11-510014886369" (UID: "b6c95c24-6280-4080-9d11-510014886369"). InnerVolumeSpecName "kube-api-access-fq2m4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:32:38.070802 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:32:38.070630    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6c95c24-6280-4080-9d11-510014886369-dshm" (OuterVolumeSpecName: "dshm") pod "b6c95c24-6280-4080-9d11-510014886369" (UID: "b6c95c24-6280-4080-9d11-510014886369"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:32:38.169363 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:32:38.169330    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6c95c24-6280-4080-9d11-510014886369-dshm\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:32:38.169506 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:32:38.169364    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq2m4\" (UniqueName: \"kubernetes.io/projected/b6c95c24-6280-4080-9d11-510014886369-kube-api-access-fq2m4\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:32:38.169506 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:32:38.169380    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6c95c24-6280-4080-9d11-510014886369-tls-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:32:38.169506 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:32:38.169393    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6c95c24-6280-4080-9d11-510014886369-tmp-dir\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:32:38.169506 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:32:38.169405    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6c95c24-6280-4080-9d11-510014886369-model-cache\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:32:38.169506 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:32:38.169416    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6c95c24-6280-4080-9d11-510014886369-home\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:32:38.366804 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:32:38.366771    2555 generic.go:356] "Generic (PLEG): container finished" podID="b6c95c24-6280-4080-9d11-510014886369" containerID="138ebdd09f55167e985e424d21cc5e4a58eb28d874d911313963763bda2cdad1" exitCode=0
Sep 17 22:32:38.366969 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:32:38.366818    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94"
Sep 17 22:32:38.366969 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:32:38.366850    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94" event={"ID":"b6c95c24-6280-4080-9d11-510014886369","Type":"ContainerDied","Data":"138ebdd09f55167e985e424d21cc5e4a58eb28d874d911313963763bda2cdad1"}
Sep 17 22:32:38.366969 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:32:38.366905    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94" event={"ID":"b6c95c24-6280-4080-9d11-510014886369","Type":"ContainerDied","Data":"47541182c803ee4577fcfa3dad79bad755dd5b795cd393965243ffd09dbfee86"}
Sep 17 22:32:38.366969 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:32:38.366932    2555 scope.go:117] "RemoveContainer" containerID="138ebdd09f55167e985e424d21cc5e4a58eb28d874d911313963763bda2cdad1"
Sep 17 22:32:38.377274 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:32:38.377250    2555 scope.go:117] "RemoveContainer" containerID="138ebdd09f55167e985e424d21cc5e4a58eb28d874d911313963763bda2cdad1"
Sep 17 22:32:38.377580 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:32:38.377560    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"138ebdd09f55167e985e424d21cc5e4a58eb28d874d911313963763bda2cdad1\": container with ID starting with 138ebdd09f55167e985e424d21cc5e4a58eb28d874d911313963763bda2cdad1 not found: ID does not exist" containerID="138ebdd09f55167e985e424d21cc5e4a58eb28d874d911313963763bda2cdad1"
Sep 17 22:32:38.377662 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:32:38.377589    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"138ebdd09f55167e985e424d21cc5e4a58eb28d874d911313963763bda2cdad1"} err="failed to get container status \"138ebdd09f55167e985e424d21cc5e4a58eb28d874d911313963763bda2cdad1\": rpc error: code = NotFound desc = could not find container \"138ebdd09f55167e985e424d21cc5e4a58eb28d874d911313963763bda2cdad1\": container with ID starting with 138ebdd09f55167e985e424d21cc5e4a58eb28d874d911313963763bda2cdad1 not found: ID does not exist"
Sep 17 22:32:38.390141 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:32:38.390120    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94"]
Sep 17 22:32:38.394327 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:32:38.394302    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qtq94"]
Sep 17 22:32:38.680200 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:32:38.680161    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6c95c24-6280-4080-9d11-510014886369" path="/var/lib/kubelet/pods/b6c95c24-6280-4080-9d11-510014886369/volumes"
Sep 17 22:33:11.997548 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:33:11.997504    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:33:22.781922 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:33:22.781893    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-7njvj_a324c9ab-5d42-4d31-8ff7-449f619be774/console-operator/1.log"
Sep 17 22:33:22.786120 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:33:22.786100    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s7f5_4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf/ovn-acl-logging/0.log"
Sep 17 22:33:22.791637 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:33:22.791618    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-7njvj_a324c9ab-5d42-4d31-8ff7-449f619be774/console-operator/1.log"
Sep 17 22:33:22.795272 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:33:22.795255    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s7f5_4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf/ovn-acl-logging/0.log"
Sep 17 22:33:26.902272 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:33:26.902236    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:33:26.903859 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:33:26.903824    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv"]
Sep 17 22:33:26.913868 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:33:26.913830    2555 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 22:33:26.913868 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:33:26.913868 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:33:26.913868 ip-10-0-140-157 kubenswrapper[2555]:  > containerID="c27673f17c3e8946f85272b45445c23e507dfd524b52d42d79d9f903ada4eb6d" cmd=["/bin/sleep","15"]
Sep 17 22:33:26.914121 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:33:26.913893    2555 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv" message="" err=<
Sep 17 22:33:26.914121 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:33:26.914121 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:33:26.914121 ip-10-0-140-157 kubenswrapper[2555]:  >
Sep 17 22:33:26.914121 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:33:26.913911    2555 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 22:33:26.914121 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:33:26.914121 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 22:33:26.914121 ip-10-0-140-157 kubenswrapper[2555]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv" podUID="1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a" containerName="main" containerID="cri-o://c27673f17c3e8946f85272b45445c23e507dfd524b52d42d79d9f903ada4eb6d"
Sep 17 22:33:26.914121 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:33:26.913966    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv" podUID="1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a" containerName="main" containerID="cri-o://c27673f17c3e8946f85272b45445c23e507dfd524b52d42d79d9f903ada4eb6d" gracePeriod=60
Sep 17 22:33:26.968668 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:33:26.968642    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:33:27.278698 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:33:27.278671    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv"
Sep 17 22:33:27.382774 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:33:27.382741    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a-model-cache\") pod \"1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a\" (UID: \"1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a\") "
Sep 17 22:33:27.382774 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:33:27.382779    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a-dshm\") pod \"1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a\" (UID: \"1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a\") "
Sep 17 22:33:27.383039 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:33:27.382809    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a-tmp-dir\") pod \"1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a\" (UID: \"1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a\") "
Sep 17 22:33:27.383039 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:33:27.382854    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a-home\") pod \"1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a\" (UID: \"1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a\") "
Sep 17 22:33:27.383039 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:33:27.382883    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zxrc\" (UniqueName: \"kubernetes.io/projected/1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a-kube-api-access-9zxrc\") pod \"1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a\" (UID: \"1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a\") "
Sep 17 22:33:27.383039 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:33:27.382930    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a-tls-certs\") pod \"1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a\" (UID: \"1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a\") "
Sep 17 22:33:27.383266 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:33:27.383088    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a-model-cache" (OuterVolumeSpecName: "model-cache") pod "1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a" (UID: "1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:33:27.383266 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:33:27.383100    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a" (UID: "1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:33:27.383266 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:33:27.383118    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a-home" (OuterVolumeSpecName: "home") pod "1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a" (UID: "1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:33:27.383266 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:33:27.383247    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a-model-cache\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:33:27.383266 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:33:27.383263    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a-tmp-dir\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:33:27.383518 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:33:27.383272    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a-home\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:33:27.385062 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:33:27.385036    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a-dshm" (OuterVolumeSpecName: "dshm") pod "1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a" (UID: "1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:33:27.385312 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:33:27.385296    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a-kube-api-access-9zxrc" (OuterVolumeSpecName: "kube-api-access-9zxrc") pod "1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a" (UID: "1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a"). InnerVolumeSpecName "kube-api-access-9zxrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:33:27.385392 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:33:27.385295    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a" (UID: "1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:33:27.483732 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:33:27.483649    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zxrc\" (UniqueName: \"kubernetes.io/projected/1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a-kube-api-access-9zxrc\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:33:27.483732 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:33:27.483675    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a-tls-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:33:27.483732 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:33:27.483684    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a-dshm\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:33:27.639549 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:33:27.639514    2555 generic.go:356] "Generic (PLEG): container finished" podID="1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a" containerID="c27673f17c3e8946f85272b45445c23e507dfd524b52d42d79d9f903ada4eb6d" exitCode=0
Sep 17 22:33:27.639741 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:33:27.639567    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv"
Sep 17 22:33:27.639741 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:33:27.639598    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv" event={"ID":"1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a","Type":"ContainerDied","Data":"c27673f17c3e8946f85272b45445c23e507dfd524b52d42d79d9f903ada4eb6d"}
Sep 17 22:33:27.639741 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:33:27.639689    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv" event={"ID":"1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a","Type":"ContainerDied","Data":"4d340f4bb99566cc6cc027359f0a559427652c5b8e65b744a94ca5d9e7fc42f0"}
Sep 17 22:33:27.639741 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:33:27.639713    2555 scope.go:117] "RemoveContainer" containerID="c27673f17c3e8946f85272b45445c23e507dfd524b52d42d79d9f903ada4eb6d"
Sep 17 22:33:27.650366 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:33:27.650347    2555 scope.go:117] "RemoveContainer" containerID="c27673f17c3e8946f85272b45445c23e507dfd524b52d42d79d9f903ada4eb6d"
Sep 17 22:33:27.650634 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:33:27.650612    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c27673f17c3e8946f85272b45445c23e507dfd524b52d42d79d9f903ada4eb6d\": container with ID starting with c27673f17c3e8946f85272b45445c23e507dfd524b52d42d79d9f903ada4eb6d not found: ID does not exist" containerID="c27673f17c3e8946f85272b45445c23e507dfd524b52d42d79d9f903ada4eb6d"
Sep 17 22:33:27.650697 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:33:27.650648    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c27673f17c3e8946f85272b45445c23e507dfd524b52d42d79d9f903ada4eb6d"} err="failed to get container status \"c27673f17c3e8946f85272b45445c23e507dfd524b52d42d79d9f903ada4eb6d\": rpc error: code = NotFound desc = could not find container \"c27673f17c3e8946f85272b45445c23e507dfd524b52d42d79d9f903ada4eb6d\": container with ID starting with c27673f17c3e8946f85272b45445c23e507dfd524b52d42d79d9f903ada4eb6d not found: ID does not exist"
Sep 17 22:33:27.661800 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:33:27.661775    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv"]
Sep 17 22:33:27.665233 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:33:27.665212    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969kj7nv"]
Sep 17 22:33:28.678777 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:33:28.678746    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a" path="/var/lib/kubelet/pods/1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a/volumes"
Sep 17 22:34:17.007336 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:17.007260    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:34:17.107055 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:17.107018    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:34:23.295132 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:23.295098    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:34:28.873866 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:28.873831    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:34:28.977643 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:28.977612    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:34:31.054311 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:31.054280    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd"]
Sep 17 22:34:31.054753 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:31.054650    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a" containerName="main"
Sep 17 22:34:31.054753 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:31.054661    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a" containerName="main"
Sep 17 22:34:31.054753 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:31.054671    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6c95c24-6280-4080-9d11-510014886369" containerName="main"
Sep 17 22:34:31.054753 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:31.054676    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6c95c24-6280-4080-9d11-510014886369" containerName="main"
Sep 17 22:34:31.054753 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:31.054737    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6c95c24-6280-4080-9d11-510014886369" containerName="main"
Sep 17 22:34:31.054753 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:31.054746    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c6b5a02-ad7e-4f4f-99bc-972fe3ae2f2a" containerName="main"
Sep 17 22:34:31.061128 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:31.061096    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd"
Sep 17 22:34:31.064000 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:31.063971    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Sep 17 22:34:31.064195 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:31.064154    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 17 22:34:31.064339 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:31.064117    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 17 22:34:31.064339 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:31.064261    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 17 22:34:31.064486 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:31.064469    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-bt8z8\""
Sep 17 22:34:31.064710 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:31.064694    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:34:31.069718 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:31.069480    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd"]
Sep 17 22:34:31.071378 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:31.071357    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:34:31.130563 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:31.130528    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa330235-8e77-4123-87a8-6093f4fe1c44-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd\" (UID: \"fa330235-8e77-4123-87a8-6093f4fe1c44\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd"
Sep 17 22:34:31.130721 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:31.130577    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa330235-8e77-4123-87a8-6093f4fe1c44-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd\" (UID: \"fa330235-8e77-4123-87a8-6093f4fe1c44\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd"
Sep 17 22:34:31.130721 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:31.130657    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa330235-8e77-4123-87a8-6093f4fe1c44-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd\" (UID: \"fa330235-8e77-4123-87a8-6093f4fe1c44\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd"
Sep 17 22:34:31.130721 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:31.130692    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa330235-8e77-4123-87a8-6093f4fe1c44-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd\" (UID: \"fa330235-8e77-4123-87a8-6093f4fe1c44\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd"
Sep 17 22:34:31.130834 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:31.130733    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa330235-8e77-4123-87a8-6093f4fe1c44-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd\" (UID: \"fa330235-8e77-4123-87a8-6093f4fe1c44\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd"
Sep 17 22:34:31.130834 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:31.130809    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa330235-8e77-4123-87a8-6093f4fe1c44-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd\" (UID: \"fa330235-8e77-4123-87a8-6093f4fe1c44\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd"
Sep 17 22:34:31.130910 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:31.130834    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa330235-8e77-4123-87a8-6093f4fe1c44-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd\" (UID: \"fa330235-8e77-4123-87a8-6093f4fe1c44\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd"
Sep 17 22:34:31.130910 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:31.130853    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzcpd\" (UniqueName: \"kubernetes.io/projected/fa330235-8e77-4123-87a8-6093f4fe1c44-kube-api-access-tzcpd\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd\" (UID: \"fa330235-8e77-4123-87a8-6093f4fe1c44\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd"
Sep 17 22:34:31.231903 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:31.231865    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa330235-8e77-4123-87a8-6093f4fe1c44-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd\" (UID: \"fa330235-8e77-4123-87a8-6093f4fe1c44\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd"
Sep 17 22:34:31.231903 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:31.231905    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa330235-8e77-4123-87a8-6093f4fe1c44-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd\" (UID: \"fa330235-8e77-4123-87a8-6093f4fe1c44\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd"
Sep 17 22:34:31.232111 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:31.232025    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa330235-8e77-4123-87a8-6093f4fe1c44-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd\" (UID: \"fa330235-8e77-4123-87a8-6093f4fe1c44\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd"
Sep 17 22:34:31.232111 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:31.232073    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa330235-8e77-4123-87a8-6093f4fe1c44-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd\" (UID: \"fa330235-8e77-4123-87a8-6093f4fe1c44\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd"
Sep 17 22:34:31.232111 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:31.232090    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa330235-8e77-4123-87a8-6093f4fe1c44-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd\" (UID: \"fa330235-8e77-4123-87a8-6093f4fe1c44\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd"
Sep 17 22:34:31.232288 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:31.232109    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzcpd\" (UniqueName: \"kubernetes.io/projected/fa330235-8e77-4123-87a8-6093f4fe1c44-kube-api-access-tzcpd\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd\" (UID: \"fa330235-8e77-4123-87a8-6093f4fe1c44\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd"
Sep 17 22:34:31.232288 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:31.232156    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa330235-8e77-4123-87a8-6093f4fe1c44-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd\" (UID: \"fa330235-8e77-4123-87a8-6093f4fe1c44\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd"
Sep 17 22:34:31.232288 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:31.232212    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa330235-8e77-4123-87a8-6093f4fe1c44-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd\" (UID: \"fa330235-8e77-4123-87a8-6093f4fe1c44\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd"
Sep 17 22:34:31.232455 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:31.232432    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa330235-8e77-4123-87a8-6093f4fe1c44-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd\" (UID: \"fa330235-8e77-4123-87a8-6093f4fe1c44\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd"
Sep 17 22:34:31.232517 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:31.232452    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa330235-8e77-4123-87a8-6093f4fe1c44-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd\" (UID: \"fa330235-8e77-4123-87a8-6093f4fe1c44\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd"
Sep 17 22:34:31.232602 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:31.232581    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa330235-8e77-4123-87a8-6093f4fe1c44-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd\" (UID: \"fa330235-8e77-4123-87a8-6093f4fe1c44\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd"
Sep 17 22:34:31.232655 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:31.232627    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa330235-8e77-4123-87a8-6093f4fe1c44-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd\" (UID: \"fa330235-8e77-4123-87a8-6093f4fe1c44\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd"
Sep 17 22:34:31.233036 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:31.233020    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa330235-8e77-4123-87a8-6093f4fe1c44-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd\" (UID: \"fa330235-8e77-4123-87a8-6093f4fe1c44\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd"
Sep 17 22:34:31.234262 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:31.234244    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa330235-8e77-4123-87a8-6093f4fe1c44-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd\" (UID: \"fa330235-8e77-4123-87a8-6093f4fe1c44\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd"
Sep 17 22:34:31.234582 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:31.234563    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa330235-8e77-4123-87a8-6093f4fe1c44-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd\" (UID: \"fa330235-8e77-4123-87a8-6093f4fe1c44\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd"
Sep 17 22:34:31.239928 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:31.239902    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzcpd\" (UniqueName: \"kubernetes.io/projected/fa330235-8e77-4123-87a8-6093f4fe1c44-kube-api-access-tzcpd\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd\" (UID: \"fa330235-8e77-4123-87a8-6093f4fe1c44\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd"
Sep 17 22:34:31.374367 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:31.374148    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd"
Sep 17 22:34:31.431505 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:31.431379    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:34:31.664876 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:31.664846    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd"]
Sep 17 22:34:31.666370 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:34:31.666342    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa330235_8e77_4123_87a8_6093f4fe1c44.slice/crio-d98cbe9ffa8efdefd9f1c92bb1da45e64c81edb606b98200d59d374e4f6e3da9 WatchSource:0}: Error finding container d98cbe9ffa8efdefd9f1c92bb1da45e64c81edb606b98200d59d374e4f6e3da9: Status 404 returned error can't find the container with id d98cbe9ffa8efdefd9f1c92bb1da45e64c81edb606b98200d59d374e4f6e3da9
Sep 17 22:34:32.004032 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:32.003948    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd" event={"ID":"fa330235-8e77-4123-87a8-6093f4fe1c44","Type":"ContainerStarted","Data":"6dfc62aa5aabb8a891520aa325b57331841df64578aa21f566e4d3f3fc2056d2"}
Sep 17 22:34:32.004032 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:32.004000    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd" event={"ID":"fa330235-8e77-4123-87a8-6093f4fe1c44","Type":"ContainerStarted","Data":"d98cbe9ffa8efdefd9f1c92bb1da45e64c81edb606b98200d59d374e4f6e3da9"}
Sep 17 22:34:35.022656 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:35.022621    2555 generic.go:356] "Generic (PLEG): container finished" podID="fa330235-8e77-4123-87a8-6093f4fe1c44" containerID="6dfc62aa5aabb8a891520aa325b57331841df64578aa21f566e4d3f3fc2056d2" exitCode=0
Sep 17 22:34:35.023041 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:35.022673    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd" event={"ID":"fa330235-8e77-4123-87a8-6093f4fe1c44","Type":"ContainerDied","Data":"6dfc62aa5aabb8a891520aa325b57331841df64578aa21f566e4d3f3fc2056d2"}
Sep 17 22:34:36.032497 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:36.032456    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd" event={"ID":"fa330235-8e77-4123-87a8-6093f4fe1c44","Type":"ContainerStarted","Data":"b2e967e7b8aa8964c4755b2616025f13a04fa776472cd44b8716759d453417a3"}
Sep 17 22:34:36.053783 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:36.053679    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd" podStartSLOduration=2.263774351 podStartE2EDuration="5.05366288s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.789888529s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:34:36.052588557 +0000 UTC m=+2177.498633853" watchObservedRunningTime="2026-09-17 22:34:36.05366288 +0000 UTC m=+2177.499708152"
Sep 17 22:34:36.374607 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:36.374516    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd"
Sep 17 22:34:41.374829 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:41.374785    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd"
Sep 17 22:34:41.376823 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:41.376790    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd" podUID="fa330235-8e77-4123-87a8-6093f4fe1c44" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 17 22:34:49.774832 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:49.774779    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:34:51.375030 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:51.374977    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd" podUID="fa330235-8e77-4123-87a8-6093f4fe1c44" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 17 22:34:56.893902 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:56.893864    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:34:56.991095 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:34:56.991062    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:35:01.375119 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:35:01.375066    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd" podUID="fa330235-8e77-4123-87a8-6093f4fe1c44" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 17 22:35:11.374945 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:35:11.374901    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd" podUID="fa330235-8e77-4123-87a8-6093f4fe1c44" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 17 22:35:21.375500 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:35:21.375454    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd" podUID="fa330235-8e77-4123-87a8-6093f4fe1c44" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 17 22:35:31.374789 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:35:31.374747    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd" podUID="fa330235-8e77-4123-87a8-6093f4fe1c44" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 17 22:35:41.375674 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:35:41.375577    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd" podUID="fa330235-8e77-4123-87a8-6093f4fe1c44" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 17 22:35:46.855948 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:35:46.855912    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:35:46.950604 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:35:46.950565    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:35:51.385325 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:35:51.385285    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd"
Sep 17 22:35:51.393048 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:35:51.393020    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd"
Sep 17 22:35:52.887216 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:35:52.887166    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:35:58.316519 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:35:58.316485    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:35:58.387835 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:35:58.387805    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:36:03.449652 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:03.449613    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s"]
Sep 17 22:36:03.453279 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:03.453256    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s"
Sep 17 22:36:03.455691 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:03.455665    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-mwztz\""
Sep 17 22:36:03.455799 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:03.455675    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 17 22:36:03.455799 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:03.455721    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 17 22:36:03.455799 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:03.455772    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 17 22:36:03.463544 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:03.463520    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s"]
Sep 17 22:36:03.465234 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:03.465209    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:36:03.559911 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:03.559877    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78be1225-53ec-4c70-8dfc-eb628f6f979e-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s\" (UID: \"78be1225-53ec-4c70-8dfc-eb628f6f979e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s"
Sep 17 22:36:03.559911 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:03.559910    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78be1225-53ec-4c70-8dfc-eb628f6f979e-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s\" (UID: \"78be1225-53ec-4c70-8dfc-eb628f6f979e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s"
Sep 17 22:36:03.560227 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:03.559940    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78be1225-53ec-4c70-8dfc-eb628f6f979e-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s\" (UID: \"78be1225-53ec-4c70-8dfc-eb628f6f979e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s"
Sep 17 22:36:03.560227 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:03.560016    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78be1225-53ec-4c70-8dfc-eb628f6f979e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s\" (UID: \"78be1225-53ec-4c70-8dfc-eb628f6f979e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s"
Sep 17 22:36:03.560227 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:03.560062    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78be1225-53ec-4c70-8dfc-eb628f6f979e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s\" (UID: \"78be1225-53ec-4c70-8dfc-eb628f6f979e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s"
Sep 17 22:36:03.560227 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:03.560132    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78be1225-53ec-4c70-8dfc-eb628f6f979e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s\" (UID: \"78be1225-53ec-4c70-8dfc-eb628f6f979e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s"
Sep 17 22:36:03.560227 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:03.560161    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqd7c\" (UniqueName: \"kubernetes.io/projected/78be1225-53ec-4c70-8dfc-eb628f6f979e-kube-api-access-gqd7c\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s\" (UID: \"78be1225-53ec-4c70-8dfc-eb628f6f979e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s"
Sep 17 22:36:03.661193 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:03.661154    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78be1225-53ec-4c70-8dfc-eb628f6f979e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s\" (UID: \"78be1225-53ec-4c70-8dfc-eb628f6f979e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s"
Sep 17 22:36:03.661384 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:03.661222    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78be1225-53ec-4c70-8dfc-eb628f6f979e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s\" (UID: \"78be1225-53ec-4c70-8dfc-eb628f6f979e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s"
Sep 17 22:36:03.661384 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:03.661264    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqd7c\" (UniqueName: \"kubernetes.io/projected/78be1225-53ec-4c70-8dfc-eb628f6f979e-kube-api-access-gqd7c\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s\" (UID: \"78be1225-53ec-4c70-8dfc-eb628f6f979e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s"
Sep 17 22:36:03.661384 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:03.661301    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78be1225-53ec-4c70-8dfc-eb628f6f979e-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s\" (UID: \"78be1225-53ec-4c70-8dfc-eb628f6f979e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s"
Sep 17 22:36:03.661384 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:03.661322    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78be1225-53ec-4c70-8dfc-eb628f6f979e-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s\" (UID: \"78be1225-53ec-4c70-8dfc-eb628f6f979e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s"
Sep 17 22:36:03.661572 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:03.661390    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78be1225-53ec-4c70-8dfc-eb628f6f979e-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s\" (UID: \"78be1225-53ec-4c70-8dfc-eb628f6f979e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s"
Sep 17 22:36:03.661572 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:03.661441    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78be1225-53ec-4c70-8dfc-eb628f6f979e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s\" (UID: \"78be1225-53ec-4c70-8dfc-eb628f6f979e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s"
Sep 17 22:36:03.661709 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:03.661633    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78be1225-53ec-4c70-8dfc-eb628f6f979e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s\" (UID: \"78be1225-53ec-4c70-8dfc-eb628f6f979e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s"
Sep 17 22:36:03.661849 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:03.661828    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78be1225-53ec-4c70-8dfc-eb628f6f979e-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s\" (UID: \"78be1225-53ec-4c70-8dfc-eb628f6f979e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s"
Sep 17 22:36:03.661922 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:03.661840    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78be1225-53ec-4c70-8dfc-eb628f6f979e-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s\" (UID: \"78be1225-53ec-4c70-8dfc-eb628f6f979e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s"
Sep 17 22:36:03.661962 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:03.661916    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78be1225-53ec-4c70-8dfc-eb628f6f979e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s\" (UID: \"78be1225-53ec-4c70-8dfc-eb628f6f979e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s"
Sep 17 22:36:03.663654 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:03.663631    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78be1225-53ec-4c70-8dfc-eb628f6f979e-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s\" (UID: \"78be1225-53ec-4c70-8dfc-eb628f6f979e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s"
Sep 17 22:36:03.663790 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:03.663770    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78be1225-53ec-4c70-8dfc-eb628f6f979e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s\" (UID: \"78be1225-53ec-4c70-8dfc-eb628f6f979e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s"
Sep 17 22:36:03.671836 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:03.671808    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqd7c\" (UniqueName: \"kubernetes.io/projected/78be1225-53ec-4c70-8dfc-eb628f6f979e-kube-api-access-gqd7c\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s\" (UID: \"78be1225-53ec-4c70-8dfc-eb628f6f979e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s"
Sep 17 22:36:03.727640 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:03.727557    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:36:03.765647 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:03.765611    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s"
Sep 17 22:36:04.045037 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:04.045008    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s"]
Sep 17 22:36:04.104142 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:04.104117    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 22:36:04.561679 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:04.561634    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s" event={"ID":"78be1225-53ec-4c70-8dfc-eb628f6f979e","Type":"ContainerStarted","Data":"484a8100593c6538c3a8def315a663d419e164e2cf699df5fd08ac2d8ea0218b"}
Sep 17 22:36:10.602080 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:10.602049    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s" event={"ID":"78be1225-53ec-4c70-8dfc-eb628f6f979e","Type":"ContainerStarted","Data":"9a608ffe0a3a38160a7e3fcf0dcb745eebdf69341aa39314501c5d29512dd8a8"}
Sep 17 22:36:12.616400 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:12.616363    2555 generic.go:356] "Generic (PLEG): container finished" podID="78be1225-53ec-4c70-8dfc-eb628f6f979e" containerID="9a608ffe0a3a38160a7e3fcf0dcb745eebdf69341aa39314501c5d29512dd8a8" exitCode=0
Sep 17 22:36:12.616783 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:12.616433    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s" event={"ID":"78be1225-53ec-4c70-8dfc-eb628f6f979e","Type":"ContainerDied","Data":"9a608ffe0a3a38160a7e3fcf0dcb745eebdf69341aa39314501c5d29512dd8a8"}
Sep 17 22:36:13.624468 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:13.624429    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s" event={"ID":"78be1225-53ec-4c70-8dfc-eb628f6f979e","Type":"ContainerStarted","Data":"79e36fbffb441021cd15a8459b802930e7a552cc143ed3e049766d570268db71"}
Sep 17 22:36:13.624868 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:13.624509    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s" event={"ID":"78be1225-53ec-4c70-8dfc-eb628f6f979e","Type":"ContainerStarted","Data":"e41c4d0775062a968e2be6e9e3fa9f295df9d95c98119a103d5b7ae176f81698"}
Sep 17 22:36:13.647151 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:13.647041    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s" podStartSLOduration=2.8331903389999997 podStartE2EDuration="10.647025565s" totalImagesPullingTime="6.528948713s" totalInitContainerRuntime="1.284886513s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:36:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:36:13.64442663 +0000 UTC m=+2275.090471903" watchObservedRunningTime="2026-09-17 22:36:13.647025565 +0000 UTC m=+2275.093070837"
Sep 17 22:36:13.766632 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:13.766595    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s"
Sep 17 22:36:13.766813 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:13.766664    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s"
Sep 17 22:36:13.768302 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:13.768274    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s" podUID="78be1225-53ec-4c70-8dfc-eb628f6f979e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Sep 17 22:36:23.766859 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:23.766821    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s" podUID="78be1225-53ec-4c70-8dfc-eb628f6f979e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Sep 17 22:36:33.766567 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:33.766517    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s" podUID="78be1225-53ec-4c70-8dfc-eb628f6f979e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Sep 17 22:36:41.740024 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:41.739988    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:36:43.766464 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:43.766419    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s" podUID="78be1225-53ec-4c70-8dfc-eb628f6f979e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Sep 17 22:36:53.766338 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:36:53.766287    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s" podUID="78be1225-53ec-4c70-8dfc-eb628f6f979e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Sep 17 22:37:03.766741 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:03.766635    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s" podUID="78be1225-53ec-4c70-8dfc-eb628f6f979e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Sep 17 22:37:10.481448 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:10.481414    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:37:10.482524 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:10.482493    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd"]
Sep 17 22:37:10.538725 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:10.538692    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:37:13.766539 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:13.766497    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s" podUID="78be1225-53ec-4c70-8dfc-eb628f6f979e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Sep 17 22:37:23.766900 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:23.766837    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s" podUID="78be1225-53ec-4c70-8dfc-eb628f6f979e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Sep 17 22:37:24.671074 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:24.670973    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:37:24.772521 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:24.772483    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:37:25.508251 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:25.508204    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd" podUID="fa330235-8e77-4123-87a8-6093f4fe1c44" containerName="main" containerID="cri-o://b2e967e7b8aa8964c4755b2616025f13a04fa776472cd44b8716759d453417a3" gracePeriod=45
Sep 17 22:37:26.476984 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:26.476958    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd"
Sep 17 22:37:26.550402 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:26.550317    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa330235-8e77-4123-87a8-6093f4fe1c44-model-cache\") pod \"fa330235-8e77-4123-87a8-6093f4fe1c44\" (UID: \"fa330235-8e77-4123-87a8-6093f4fe1c44\") "
Sep 17 22:37:26.550402 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:26.550356    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa330235-8e77-4123-87a8-6093f4fe1c44-home\") pod \"fa330235-8e77-4123-87a8-6093f4fe1c44\" (UID: \"fa330235-8e77-4123-87a8-6093f4fe1c44\") "
Sep 17 22:37:26.550622 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:26.550411    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa330235-8e77-4123-87a8-6093f4fe1c44-cabundle-cert\") pod \"fa330235-8e77-4123-87a8-6093f4fe1c44\" (UID: \"fa330235-8e77-4123-87a8-6093f4fe1c44\") "
Sep 17 22:37:26.550622 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:26.550428    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa330235-8e77-4123-87a8-6093f4fe1c44-tmp-dir\") pod \"fa330235-8e77-4123-87a8-6093f4fe1c44\" (UID: \"fa330235-8e77-4123-87a8-6093f4fe1c44\") "
Sep 17 22:37:26.550622 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:26.550481    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa330235-8e77-4123-87a8-6093f4fe1c44-tls-certs\") pod \"fa330235-8e77-4123-87a8-6093f4fe1c44\" (UID: \"fa330235-8e77-4123-87a8-6093f4fe1c44\") "
Sep 17 22:37:26.550622 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:26.550509    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzcpd\" (UniqueName: \"kubernetes.io/projected/fa330235-8e77-4123-87a8-6093f4fe1c44-kube-api-access-tzcpd\") pod \"fa330235-8e77-4123-87a8-6093f4fe1c44\" (UID: \"fa330235-8e77-4123-87a8-6093f4fe1c44\") "
Sep 17 22:37:26.550622 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:26.550552    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa330235-8e77-4123-87a8-6093f4fe1c44-dshm\") pod \"fa330235-8e77-4123-87a8-6093f4fe1c44\" (UID: \"fa330235-8e77-4123-87a8-6093f4fe1c44\") "
Sep 17 22:37:26.550622 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:26.550570    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa330235-8e77-4123-87a8-6093f4fe1c44-model-cache" (OuterVolumeSpecName: "model-cache") pod "fa330235-8e77-4123-87a8-6093f4fe1c44" (UID: "fa330235-8e77-4123-87a8-6093f4fe1c44"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:37:26.550622 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:26.550598    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa330235-8e77-4123-87a8-6093f4fe1c44-kserve-provision-location\") pod \"fa330235-8e77-4123-87a8-6093f4fe1c44\" (UID: \"fa330235-8e77-4123-87a8-6093f4fe1c44\") "
Sep 17 22:37:26.550973 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:26.550853    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa330235-8e77-4123-87a8-6093f4fe1c44-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fa330235-8e77-4123-87a8-6093f4fe1c44" (UID: "fa330235-8e77-4123-87a8-6093f4fe1c44"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:37:26.550973 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:26.550877    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa330235-8e77-4123-87a8-6093f4fe1c44-model-cache\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:37:26.550973 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:26.550875    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa330235-8e77-4123-87a8-6093f4fe1c44-home" (OuterVolumeSpecName: "home") pod "fa330235-8e77-4123-87a8-6093f4fe1c44" (UID: "fa330235-8e77-4123-87a8-6093f4fe1c44"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:37:26.553029 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:26.552979    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa330235-8e77-4123-87a8-6093f4fe1c44-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fa330235-8e77-4123-87a8-6093f4fe1c44" (UID: "fa330235-8e77-4123-87a8-6093f4fe1c44"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:37:26.553029 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:26.553011    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa330235-8e77-4123-87a8-6093f4fe1c44-dshm" (OuterVolumeSpecName: "dshm") pod "fa330235-8e77-4123-87a8-6093f4fe1c44" (UID: "fa330235-8e77-4123-87a8-6093f4fe1c44"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:37:26.553213 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:26.553116    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa330235-8e77-4123-87a8-6093f4fe1c44-kube-api-access-tzcpd" (OuterVolumeSpecName: "kube-api-access-tzcpd") pod "fa330235-8e77-4123-87a8-6093f4fe1c44" (UID: "fa330235-8e77-4123-87a8-6093f4fe1c44"). InnerVolumeSpecName "kube-api-access-tzcpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:37:26.560941 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:26.560916    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa330235-8e77-4123-87a8-6093f4fe1c44-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fa330235-8e77-4123-87a8-6093f4fe1c44" (UID: "fa330235-8e77-4123-87a8-6093f4fe1c44"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:37:26.563473 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:26.563443    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa330235-8e77-4123-87a8-6093f4fe1c44-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fa330235-8e77-4123-87a8-6093f4fe1c44" (UID: "fa330235-8e77-4123-87a8-6093f4fe1c44"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:37:26.652248 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:26.652220    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa330235-8e77-4123-87a8-6093f4fe1c44-dshm\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:37:26.652248 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:26.652244    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa330235-8e77-4123-87a8-6093f4fe1c44-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:37:26.652424 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:26.652255    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa330235-8e77-4123-87a8-6093f4fe1c44-home\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:37:26.652424 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:26.652264    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa330235-8e77-4123-87a8-6093f4fe1c44-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:37:26.652424 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:26.652296    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa330235-8e77-4123-87a8-6093f4fe1c44-tmp-dir\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:37:26.652424 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:26.652305    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa330235-8e77-4123-87a8-6093f4fe1c44-tls-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:37:26.652424 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:26.652313    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzcpd\" (UniqueName: \"kubernetes.io/projected/fa330235-8e77-4123-87a8-6093f4fe1c44-kube-api-access-tzcpd\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:37:27.115346 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:27.115308    2555 generic.go:356] "Generic (PLEG): container finished" podID="fa330235-8e77-4123-87a8-6093f4fe1c44" containerID="b2e967e7b8aa8964c4755b2616025f13a04fa776472cd44b8716759d453417a3" exitCode=0
Sep 17 22:37:27.115547 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:27.115364    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd"
Sep 17 22:37:27.115547 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:27.115395    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd" event={"ID":"fa330235-8e77-4123-87a8-6093f4fe1c44","Type":"ContainerDied","Data":"b2e967e7b8aa8964c4755b2616025f13a04fa776472cd44b8716759d453417a3"}
Sep 17 22:37:27.115547 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:27.115451    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd" event={"ID":"fa330235-8e77-4123-87a8-6093f4fe1c44","Type":"ContainerDied","Data":"d98cbe9ffa8efdefd9f1c92bb1da45e64c81edb606b98200d59d374e4f6e3da9"}
Sep 17 22:37:27.115547 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:27.115475    2555 scope.go:117] "RemoveContainer" containerID="b2e967e7b8aa8964c4755b2616025f13a04fa776472cd44b8716759d453417a3"
Sep 17 22:37:27.124847 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:27.124826    2555 scope.go:117] "RemoveContainer" containerID="6dfc62aa5aabb8a891520aa325b57331841df64578aa21f566e4d3f3fc2056d2"
Sep 17 22:37:27.133362 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:27.133331    2555 scope.go:117] "RemoveContainer" containerID="b2e967e7b8aa8964c4755b2616025f13a04fa776472cd44b8716759d453417a3"
Sep 17 22:37:27.133643 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:37:27.133622    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2e967e7b8aa8964c4755b2616025f13a04fa776472cd44b8716759d453417a3\": container with ID starting with b2e967e7b8aa8964c4755b2616025f13a04fa776472cd44b8716759d453417a3 not found: ID does not exist" containerID="b2e967e7b8aa8964c4755b2616025f13a04fa776472cd44b8716759d453417a3"
Sep 17 22:37:27.133717 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:27.133657    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2e967e7b8aa8964c4755b2616025f13a04fa776472cd44b8716759d453417a3"} err="failed to get container status \"b2e967e7b8aa8964c4755b2616025f13a04fa776472cd44b8716759d453417a3\": rpc error: code = NotFound desc = could not find container \"b2e967e7b8aa8964c4755b2616025f13a04fa776472cd44b8716759d453417a3\": container with ID starting with b2e967e7b8aa8964c4755b2616025f13a04fa776472cd44b8716759d453417a3 not found: ID does not exist"
Sep 17 22:37:27.133717 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:27.133682    2555 scope.go:117] "RemoveContainer" containerID="6dfc62aa5aabb8a891520aa325b57331841df64578aa21f566e4d3f3fc2056d2"
Sep 17 22:37:27.134031 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:37:27.134000    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dfc62aa5aabb8a891520aa325b57331841df64578aa21f566e4d3f3fc2056d2\": container with ID starting with 6dfc62aa5aabb8a891520aa325b57331841df64578aa21f566e4d3f3fc2056d2 not found: ID does not exist" containerID="6dfc62aa5aabb8a891520aa325b57331841df64578aa21f566e4d3f3fc2056d2"
Sep 17 22:37:27.134128 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:27.134039    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dfc62aa5aabb8a891520aa325b57331841df64578aa21f566e4d3f3fc2056d2"} err="failed to get container status \"6dfc62aa5aabb8a891520aa325b57331841df64578aa21f566e4d3f3fc2056d2\": rpc error: code = NotFound desc = could not find container \"6dfc62aa5aabb8a891520aa325b57331841df64578aa21f566e4d3f3fc2056d2\": container with ID starting with 6dfc62aa5aabb8a891520aa325b57331841df64578aa21f566e4d3f3fc2056d2 not found: ID does not exist"
Sep 17 22:37:27.136010 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:27.135985    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd"]
Sep 17 22:37:27.139303 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:27.139279    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd"]
Sep 17 22:37:27.387120 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:27.387083    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-575fcb895f-nmcpd" podUID="fa330235-8e77-4123-87a8-6093f4fe1c44" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": context deadline exceeded"
Sep 17 22:37:28.679460 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:28.679415    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa330235-8e77-4123-87a8-6093f4fe1c44" path="/var/lib/kubelet/pods/fa330235-8e77-4123-87a8-6093f4fe1c44/volumes"
Sep 17 22:37:28.716049 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:28.715999    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:37:33.766278 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:33.766233    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s" podUID="78be1225-53ec-4c70-8dfc-eb628f6f979e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Sep 17 22:37:43.766546 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:43.766501    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s" podUID="78be1225-53ec-4c70-8dfc-eb628f6f979e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Sep 17 22:37:53.766729 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:37:53.766684    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s" podUID="78be1225-53ec-4c70-8dfc-eb628f6f979e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Sep 17 22:38:00.438545 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:00.438503    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:38:00.530944 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:00.530903    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:38:03.766405 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:03.766354    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s" podUID="78be1225-53ec-4c70-8dfc-eb628f6f979e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Sep 17 22:38:06.329166 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:06.329131    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:38:07.876440 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:07.876407    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:38:08.128399 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:08.128308    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:38:12.016814 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:12.016778    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:38:12.116541 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:12.116505    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:38:13.777036 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:13.777007    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s"
Sep 17 22:38:13.786736 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:13.786703    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s"
Sep 17 22:38:17.935129 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:17.935076    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:38:22.825810 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:22.825778    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-7njvj_a324c9ab-5d42-4d31-8ff7-449f619be774/console-operator/1.log"
Sep 17 22:38:22.830591 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:22.830565    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s7f5_4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf/ovn-acl-logging/0.log"
Sep 17 22:38:22.836001 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:22.835971    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-7njvj_a324c9ab-5d42-4d31-8ff7-449f619be774/console-operator/1.log"
Sep 17 22:38:22.840219 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:22.840173    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s7f5_4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf/ovn-acl-logging/0.log"
Sep 17 22:38:23.803152 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:23.803119    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:38:23.815596 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:23.815565    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s"]
Sep 17 22:38:23.815903 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:23.815843    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s" podUID="78be1225-53ec-4c70-8dfc-eb628f6f979e" containerName="modelcar" containerID="cri-o://79e36fbffb441021cd15a8459b802930e7a552cc143ed3e049766d570268db71" gracePeriod=60
Sep 17 22:38:23.871564 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:23.871541    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:38:24.457795 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:24.457761    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s_78be1225-53ec-4c70-8dfc-eb628f6f979e/modelcar/0.log"
Sep 17 22:38:24.458452 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:24.458427    2555 generic.go:356] "Generic (PLEG): container finished" podID="78be1225-53ec-4c70-8dfc-eb628f6f979e" containerID="79e36fbffb441021cd15a8459b802930e7a552cc143ed3e049766d570268db71" exitCode=143
Sep 17 22:38:24.458586 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:24.458506    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s" event={"ID":"78be1225-53ec-4c70-8dfc-eb628f6f979e","Type":"ContainerDied","Data":"79e36fbffb441021cd15a8459b802930e7a552cc143ed3e049766d570268db71"}
Sep 17 22:38:38.840049 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:38.839968    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s" podUID="78be1225-53ec-4c70-8dfc-eb628f6f979e" containerName="main" containerID="cri-o://e41c4d0775062a968e2be6e9e3fa9f295df9d95c98119a103d5b7ae176f81698" gracePeriod=45
Sep 17 22:38:39.547549 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:39.547518    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s_78be1225-53ec-4c70-8dfc-eb628f6f979e/modelcar/0.log"
Sep 17 22:38:39.548474 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:39.548444    2555 generic.go:356] "Generic (PLEG): container finished" podID="78be1225-53ec-4c70-8dfc-eb628f6f979e" containerID="e41c4d0775062a968e2be6e9e3fa9f295df9d95c98119a103d5b7ae176f81698" exitCode=0
Sep 17 22:38:39.548575 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:39.548508    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s" event={"ID":"78be1225-53ec-4c70-8dfc-eb628f6f979e","Type":"ContainerDied","Data":"e41c4d0775062a968e2be6e9e3fa9f295df9d95c98119a103d5b7ae176f81698"}
Sep 17 22:38:39.778363 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:39.778327    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s" podUID="78be1225-53ec-4c70-8dfc-eb628f6f979e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Sep 17 22:38:40.097846 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:40.097821    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s_78be1225-53ec-4c70-8dfc-eb628f6f979e/modelcar/0.log"
Sep 17 22:38:40.100804 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:40.100786    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s"
Sep 17 22:38:40.194010 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:40.193979    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78be1225-53ec-4c70-8dfc-eb628f6f979e-kserve-provision-location\") pod \"78be1225-53ec-4c70-8dfc-eb628f6f979e\" (UID: \"78be1225-53ec-4c70-8dfc-eb628f6f979e\") "
Sep 17 22:38:40.194010 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:40.194010    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78be1225-53ec-4c70-8dfc-eb628f6f979e-model-cache\") pod \"78be1225-53ec-4c70-8dfc-eb628f6f979e\" (UID: \"78be1225-53ec-4c70-8dfc-eb628f6f979e\") "
Sep 17 22:38:40.194268 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:40.194060    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78be1225-53ec-4c70-8dfc-eb628f6f979e-tls-certs\") pod \"78be1225-53ec-4c70-8dfc-eb628f6f979e\" (UID: \"78be1225-53ec-4c70-8dfc-eb628f6f979e\") "
Sep 17 22:38:40.194268 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:40.194092    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78be1225-53ec-4c70-8dfc-eb628f6f979e-tmp-dir\") pod \"78be1225-53ec-4c70-8dfc-eb628f6f979e\" (UID: \"78be1225-53ec-4c70-8dfc-eb628f6f979e\") "
Sep 17 22:38:40.194268 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:40.194110    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78be1225-53ec-4c70-8dfc-eb628f6f979e-dshm\") pod \"78be1225-53ec-4c70-8dfc-eb628f6f979e\" (UID: \"78be1225-53ec-4c70-8dfc-eb628f6f979e\") "
Sep 17 22:38:40.194268 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:40.194126    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78be1225-53ec-4c70-8dfc-eb628f6f979e-home\") pod \"78be1225-53ec-4c70-8dfc-eb628f6f979e\" (UID: \"78be1225-53ec-4c70-8dfc-eb628f6f979e\") "
Sep 17 22:38:40.194268 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:40.194150    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqd7c\" (UniqueName: \"kubernetes.io/projected/78be1225-53ec-4c70-8dfc-eb628f6f979e-kube-api-access-gqd7c\") pod \"78be1225-53ec-4c70-8dfc-eb628f6f979e\" (UID: \"78be1225-53ec-4c70-8dfc-eb628f6f979e\") "
Sep 17 22:38:40.194543 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:40.194371    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78be1225-53ec-4c70-8dfc-eb628f6f979e-model-cache" (OuterVolumeSpecName: "model-cache") pod "78be1225-53ec-4c70-8dfc-eb628f6f979e" (UID: "78be1225-53ec-4c70-8dfc-eb628f6f979e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:38:40.194543 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:40.194377    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78be1225-53ec-4c70-8dfc-eb628f6f979e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "78be1225-53ec-4c70-8dfc-eb628f6f979e" (UID: "78be1225-53ec-4c70-8dfc-eb628f6f979e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:38:40.194784 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:40.194760    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78be1225-53ec-4c70-8dfc-eb628f6f979e-home" (OuterVolumeSpecName: "home") pod "78be1225-53ec-4c70-8dfc-eb628f6f979e" (UID: "78be1225-53ec-4c70-8dfc-eb628f6f979e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:38:40.196497 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:40.196468    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78be1225-53ec-4c70-8dfc-eb628f6f979e-dshm" (OuterVolumeSpecName: "dshm") pod "78be1225-53ec-4c70-8dfc-eb628f6f979e" (UID: "78be1225-53ec-4c70-8dfc-eb628f6f979e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:38:40.196808 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:40.196773    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78be1225-53ec-4c70-8dfc-eb628f6f979e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "78be1225-53ec-4c70-8dfc-eb628f6f979e" (UID: "78be1225-53ec-4c70-8dfc-eb628f6f979e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:38:40.196808 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:40.196795    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78be1225-53ec-4c70-8dfc-eb628f6f979e-kube-api-access-gqd7c" (OuterVolumeSpecName: "kube-api-access-gqd7c") pod "78be1225-53ec-4c70-8dfc-eb628f6f979e" (UID: "78be1225-53ec-4c70-8dfc-eb628f6f979e"). InnerVolumeSpecName "kube-api-access-gqd7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:38:40.210961 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:40.210941    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78be1225-53ec-4c70-8dfc-eb628f6f979e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "78be1225-53ec-4c70-8dfc-eb628f6f979e" (UID: "78be1225-53ec-4c70-8dfc-eb628f6f979e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:38:40.295386 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:40.295357    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78be1225-53ec-4c70-8dfc-eb628f6f979e-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:38:40.295386 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:40.295384    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78be1225-53ec-4c70-8dfc-eb628f6f979e-model-cache\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:38:40.295386 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:40.295393    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78be1225-53ec-4c70-8dfc-eb628f6f979e-tls-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:38:40.295588 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:40.295401    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78be1225-53ec-4c70-8dfc-eb628f6f979e-tmp-dir\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:38:40.295588 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:40.295409    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78be1225-53ec-4c70-8dfc-eb628f6f979e-dshm\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:38:40.295588 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:40.295416    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78be1225-53ec-4c70-8dfc-eb628f6f979e-home\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:38:40.295588 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:40.295424    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqd7c\" (UniqueName: \"kubernetes.io/projected/78be1225-53ec-4c70-8dfc-eb628f6f979e-kube-api-access-gqd7c\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:38:40.555340 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:40.555308    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s_78be1225-53ec-4c70-8dfc-eb628f6f979e/modelcar/0.log"
Sep 17 22:38:40.556086 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:40.556069    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s"
Sep 17 22:38:40.556145 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:40.556080    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s" event={"ID":"78be1225-53ec-4c70-8dfc-eb628f6f979e","Type":"ContainerDied","Data":"484a8100593c6538c3a8def315a663d419e164e2cf699df5fd08ac2d8ea0218b"}
Sep 17 22:38:40.556206 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:40.556170    2555 scope.go:117] "RemoveContainer" containerID="79e36fbffb441021cd15a8459b802930e7a552cc143ed3e049766d570268db71"
Sep 17 22:38:40.566746 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:40.566724    2555 scope.go:117] "RemoveContainer" containerID="e41c4d0775062a968e2be6e9e3fa9f295df9d95c98119a103d5b7ae176f81698"
Sep 17 22:38:40.575276 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:40.575173    2555 scope.go:117] "RemoveContainer" containerID="9a608ffe0a3a38160a7e3fcf0dcb745eebdf69341aa39314501c5d29512dd8a8"
Sep 17 22:38:40.577924 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:40.577900    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s"]
Sep 17 22:38:40.582773 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:40.582750    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dfmr4s"]
Sep 17 22:38:40.680883 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:38:40.680850    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78be1225-53ec-4c70-8dfc-eb628f6f979e" path="/var/lib/kubelet/pods/78be1225-53ec-4c70-8dfc-eb628f6f979e/volumes"
Sep 17 22:39:13.138392 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:13.138352    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:39:13.933414 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:13.933375    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:39:14.030104 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:14.030064    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:39:19.523888 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.523853    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6"]
Sep 17 22:39:19.524431 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.524410    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78be1225-53ec-4c70-8dfc-eb628f6f979e" containerName="modelcar"
Sep 17 22:39:19.524431 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.524431    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="78be1225-53ec-4c70-8dfc-eb628f6f979e" containerName="modelcar"
Sep 17 22:39:19.524579 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.524445    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa330235-8e77-4123-87a8-6093f4fe1c44" containerName="storage-initializer"
Sep 17 22:39:19.524579 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.524451    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa330235-8e77-4123-87a8-6093f4fe1c44" containerName="storage-initializer"
Sep 17 22:39:19.524579 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.524467    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa330235-8e77-4123-87a8-6093f4fe1c44" containerName="main"
Sep 17 22:39:19.524579 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.524472    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa330235-8e77-4123-87a8-6093f4fe1c44" containerName="main"
Sep 17 22:39:19.524579 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.524479    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78be1225-53ec-4c70-8dfc-eb628f6f979e" containerName="main"
Sep 17 22:39:19.524579 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.524484    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="78be1225-53ec-4c70-8dfc-eb628f6f979e" containerName="main"
Sep 17 22:39:19.524579 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.524497    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78be1225-53ec-4c70-8dfc-eb628f6f979e" containerName="modelcar-init"
Sep 17 22:39:19.524579 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.524505    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="78be1225-53ec-4c70-8dfc-eb628f6f979e" containerName="modelcar-init"
Sep 17 22:39:19.525028 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.524586    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa330235-8e77-4123-87a8-6093f4fe1c44" containerName="main"
Sep 17 22:39:19.525028 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.524598    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="78be1225-53ec-4c70-8dfc-eb628f6f979e" containerName="main"
Sep 17 22:39:19.525028 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.524605    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="78be1225-53ec-4c70-8dfc-eb628f6f979e" containerName="modelcar"
Sep 17 22:39:19.527916 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.527893    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6"
Sep 17 22:39:19.530583 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.530555    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:39:19.530709 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.530600    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-2c7f8\""
Sep 17 22:39:19.530709 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.530556    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 17 22:39:19.530709 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.530611    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 17 22:39:19.530709 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.530702    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"seaweedfs-s3-creds\""
Sep 17 22:39:19.530918 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.530751    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 17 22:39:19.539332 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.539306    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6"]
Sep 17 22:39:19.559890 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.559859    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:39:19.630046 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.630015    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-tmp-dir\") pod \"custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6\" (UID: \"25ac006a-2265-4c27-95bb-c9d6cbb7fd43\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6"
Sep 17 22:39:19.630232 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.630065    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-home\") pod \"custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6\" (UID: \"25ac006a-2265-4c27-95bb-c9d6cbb7fd43\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6"
Sep 17 22:39:19.630232 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.630138    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-model-cache\") pod \"custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6\" (UID: \"25ac006a-2265-4c27-95bb-c9d6cbb7fd43\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6"
Sep 17 22:39:19.630232 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.630168    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-cabundle-cert\") pod \"custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6\" (UID: \"25ac006a-2265-4c27-95bb-c9d6cbb7fd43\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6"
Sep 17 22:39:19.630377 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.630236    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6\" (UID: \"25ac006a-2265-4c27-95bb-c9d6cbb7fd43\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6"
Sep 17 22:39:19.630377 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.630274    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ttp5\" (UniqueName: \"kubernetes.io/projected/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-kube-api-access-5ttp5\") pod \"custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6\" (UID: \"25ac006a-2265-4c27-95bb-c9d6cbb7fd43\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6"
Sep 17 22:39:19.630377 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.630292    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-dshm\") pod \"custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6\" (UID: \"25ac006a-2265-4c27-95bb-c9d6cbb7fd43\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6"
Sep 17 22:39:19.630377 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.630326    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-tls-certs\") pod \"custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6\" (UID: \"25ac006a-2265-4c27-95bb-c9d6cbb7fd43\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6"
Sep 17 22:39:19.731007 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.730963    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-home\") pod \"custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6\" (UID: \"25ac006a-2265-4c27-95bb-c9d6cbb7fd43\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6"
Sep 17 22:39:19.731249 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.731022    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-model-cache\") pod \"custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6\" (UID: \"25ac006a-2265-4c27-95bb-c9d6cbb7fd43\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6"
Sep 17 22:39:19.731453 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.731265    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-cabundle-cert\") pod \"custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6\" (UID: \"25ac006a-2265-4c27-95bb-c9d6cbb7fd43\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6"
Sep 17 22:39:19.731453 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.731326    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6\" (UID: \"25ac006a-2265-4c27-95bb-c9d6cbb7fd43\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6"
Sep 17 22:39:19.731453 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.731371    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ttp5\" (UniqueName: \"kubernetes.io/projected/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-kube-api-access-5ttp5\") pod \"custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6\" (UID: \"25ac006a-2265-4c27-95bb-c9d6cbb7fd43\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6"
Sep 17 22:39:19.731453 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.731394    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-dshm\") pod \"custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6\" (UID: \"25ac006a-2265-4c27-95bb-c9d6cbb7fd43\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6"
Sep 17 22:39:19.731453 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.731417    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-tls-certs\") pod \"custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6\" (UID: \"25ac006a-2265-4c27-95bb-c9d6cbb7fd43\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6"
Sep 17 22:39:19.731743 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.731463    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-tmp-dir\") pod \"custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6\" (UID: \"25ac006a-2265-4c27-95bb-c9d6cbb7fd43\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6"
Sep 17 22:39:19.731743 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.731584    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-home\") pod \"custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6\" (UID: \"25ac006a-2265-4c27-95bb-c9d6cbb7fd43\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6"
Sep 17 22:39:19.731875 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.731790    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-tmp-dir\") pod \"custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6\" (UID: \"25ac006a-2265-4c27-95bb-c9d6cbb7fd43\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6"
Sep 17 22:39:19.732107 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.732081    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-model-cache\") pod \"custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6\" (UID: \"25ac006a-2265-4c27-95bb-c9d6cbb7fd43\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6"
Sep 17 22:39:19.732281 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.732154    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6\" (UID: \"25ac006a-2265-4c27-95bb-c9d6cbb7fd43\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6"
Sep 17 22:39:19.732428 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.732273    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-cabundle-cert\") pod \"custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6\" (UID: \"25ac006a-2265-4c27-95bb-c9d6cbb7fd43\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6"
Sep 17 22:39:19.734730 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.734706    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-dshm\") pod \"custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6\" (UID: \"25ac006a-2265-4c27-95bb-c9d6cbb7fd43\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6"
Sep 17 22:39:19.734931 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.734912    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-tls-certs\") pod \"custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6\" (UID: \"25ac006a-2265-4c27-95bb-c9d6cbb7fd43\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6"
Sep 17 22:39:19.742201 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.742154    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ttp5\" (UniqueName: \"kubernetes.io/projected/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-kube-api-access-5ttp5\") pod \"custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6\" (UID: \"25ac006a-2265-4c27-95bb-c9d6cbb7fd43\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6"
Sep 17 22:39:19.780578 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.780503    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:39:19.840682 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:19.840649    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6"
Sep 17 22:39:20.120535 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:20.120504    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6"]
Sep 17 22:39:20.121849 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:39:20.121819    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25ac006a_2265_4c27_95bb_c9d6cbb7fd43.slice/crio-4fa80a74e1b566aaa48c8427975de604177ff9348612eae75ae5cfd6e6bdffeb WatchSource:0}: Error finding container 4fa80a74e1b566aaa48c8427975de604177ff9348612eae75ae5cfd6e6bdffeb: Status 404 returned error can't find the container with id 4fa80a74e1b566aaa48c8427975de604177ff9348612eae75ae5cfd6e6bdffeb
Sep 17 22:39:20.390364 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:20.390303    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:39:20.787682 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:20.787582    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6" event={"ID":"25ac006a-2265-4c27-95bb-c9d6cbb7fd43","Type":"ContainerStarted","Data":"0b8f8f281b2bdd424c5a56e3b9be4ca8ad61e25be07ca65821125f413fd91976"}
Sep 17 22:39:20.787682 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:20.787651    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6" event={"ID":"25ac006a-2265-4c27-95bb-c9d6cbb7fd43","Type":"ContainerStarted","Data":"4fa80a74e1b566aaa48c8427975de604177ff9348612eae75ae5cfd6e6bdffeb"}
Sep 17 22:39:22.808503 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:22.808417    2555 generic.go:356] "Generic (PLEG): container finished" podID="25ac006a-2265-4c27-95bb-c9d6cbb7fd43" containerID="0b8f8f281b2bdd424c5a56e3b9be4ca8ad61e25be07ca65821125f413fd91976" exitCode=0
Sep 17 22:39:22.808503 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:22.808490    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6" event={"ID":"25ac006a-2265-4c27-95bb-c9d6cbb7fd43","Type":"ContainerDied","Data":"0b8f8f281b2bdd424c5a56e3b9be4ca8ad61e25be07ca65821125f413fd91976"}
Sep 17 22:39:23.815890 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:23.815849    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6" event={"ID":"25ac006a-2265-4c27-95bb-c9d6cbb7fd43","Type":"ContainerStarted","Data":"099192196fcd4be1deba306a72563dbe4cdc9e92fa912056aa2ed4d927ab2cac"}
Sep 17 22:39:23.838971 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:23.838922    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6" podStartSLOduration=2.598637412 podStartE2EDuration="4.838908113s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.240270701s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:39:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:39:23.835817732 +0000 UTC m=+2465.281863004" watchObservedRunningTime="2026-09-17 22:39:23.838908113 +0000 UTC m=+2465.284953384"
Sep 17 22:39:23.840769 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:23.840749    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6"
Sep 17 22:39:25.975669 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:25.975637    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:39:26.076707 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:26.076668    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:39:29.841878 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:29.841841    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6"
Sep 17 22:39:29.843630 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:29.843599    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6" podUID="25ac006a-2265-4c27-95bb-c9d6cbb7fd43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 17 22:39:30.713607 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:30.713572    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:39:34.026027 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:34.025952    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:39:34.124849 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:34.124818    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:39:39.841211 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:39.841127    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6" podUID="25ac006a-2265-4c27-95bb-c9d6cbb7fd43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 17 22:39:49.841395 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:49.841348    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6" podUID="25ac006a-2265-4c27-95bb-c9d6cbb7fd43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 17 22:39:59.841736 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:39:59.841689    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6" podUID="25ac006a-2265-4c27-95bb-c9d6cbb7fd43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 17 22:40:09.841263 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:09.841101    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6" podUID="25ac006a-2265-4c27-95bb-c9d6cbb7fd43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 17 22:40:19.841614 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:19.841566    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6" podUID="25ac006a-2265-4c27-95bb-c9d6cbb7fd43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 17 22:40:23.666496 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:23.666447    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:40:23.763454 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:23.763420    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:40:29.592550 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:29.592497    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:40:29.841547 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:29.841502    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6" podUID="25ac006a-2265-4c27-95bb-c9d6cbb7fd43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 17 22:40:35.236085 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:35.236049    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:40:35.304391 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:35.304356    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:40:38.031244 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:38.031210    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp"]
Sep 17 22:40:38.035036 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:38.035014    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp"
Sep 17 22:40:38.037305 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:38.037281    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 17 22:40:38.037551 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:38.037527    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 17 22:40:38.037709 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:38.037365    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-qv8fc\""
Sep 17 22:40:38.037709 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:38.037379    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 17 22:40:38.037709 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:38.037486    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:40:38.037939 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:38.037515    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 17 22:40:38.046255 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:38.046234    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp"]
Sep 17 22:40:38.062007 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:38.061976    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:40:38.130282 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:38.130250    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/576beb41-890e-41b2-b4c7-fba1c29a8385-model-cache\") pod \"stop-feature-test-kserve-85486f684b-mtlrp\" (UID: \"576beb41-890e-41b2-b4c7-fba1c29a8385\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp"
Sep 17 22:40:38.130424 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:38.130289    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/576beb41-890e-41b2-b4c7-fba1c29a8385-kserve-provision-location\") pod \"stop-feature-test-kserve-85486f684b-mtlrp\" (UID: \"576beb41-890e-41b2-b4c7-fba1c29a8385\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp"
Sep 17 22:40:38.130424 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:38.130306    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/576beb41-890e-41b2-b4c7-fba1c29a8385-cabundle-cert\") pod \"stop-feature-test-kserve-85486f684b-mtlrp\" (UID: \"576beb41-890e-41b2-b4c7-fba1c29a8385\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp"
Sep 17 22:40:38.130424 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:38.130352    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/576beb41-890e-41b2-b4c7-fba1c29a8385-tmp-dir\") pod \"stop-feature-test-kserve-85486f684b-mtlrp\" (UID: \"576beb41-890e-41b2-b4c7-fba1c29a8385\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp"
Sep 17 22:40:38.130424 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:38.130374    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9wph\" (UniqueName: \"kubernetes.io/projected/576beb41-890e-41b2-b4c7-fba1c29a8385-kube-api-access-j9wph\") pod \"stop-feature-test-kserve-85486f684b-mtlrp\" (UID: \"576beb41-890e-41b2-b4c7-fba1c29a8385\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp"
Sep 17 22:40:38.130424 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:38.130401    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/576beb41-890e-41b2-b4c7-fba1c29a8385-home\") pod \"stop-feature-test-kserve-85486f684b-mtlrp\" (UID: \"576beb41-890e-41b2-b4c7-fba1c29a8385\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp"
Sep 17 22:40:38.130605 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:38.130487    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/576beb41-890e-41b2-b4c7-fba1c29a8385-tls-certs\") pod \"stop-feature-test-kserve-85486f684b-mtlrp\" (UID: \"576beb41-890e-41b2-b4c7-fba1c29a8385\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp"
Sep 17 22:40:38.130605 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:38.130559    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/576beb41-890e-41b2-b4c7-fba1c29a8385-dshm\") pod \"stop-feature-test-kserve-85486f684b-mtlrp\" (UID: \"576beb41-890e-41b2-b4c7-fba1c29a8385\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp"
Sep 17 22:40:38.231892 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:38.231857    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/576beb41-890e-41b2-b4c7-fba1c29a8385-kserve-provision-location\") pod \"stop-feature-test-kserve-85486f684b-mtlrp\" (UID: \"576beb41-890e-41b2-b4c7-fba1c29a8385\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp"
Sep 17 22:40:38.231892 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:38.231891    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/576beb41-890e-41b2-b4c7-fba1c29a8385-cabundle-cert\") pod \"stop-feature-test-kserve-85486f684b-mtlrp\" (UID: \"576beb41-890e-41b2-b4c7-fba1c29a8385\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp"
Sep 17 22:40:38.232146 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:38.231912    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/576beb41-890e-41b2-b4c7-fba1c29a8385-tmp-dir\") pod \"stop-feature-test-kserve-85486f684b-mtlrp\" (UID: \"576beb41-890e-41b2-b4c7-fba1c29a8385\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp"
Sep 17 22:40:38.232146 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:38.232031    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9wph\" (UniqueName: \"kubernetes.io/projected/576beb41-890e-41b2-b4c7-fba1c29a8385-kube-api-access-j9wph\") pod \"stop-feature-test-kserve-85486f684b-mtlrp\" (UID: \"576beb41-890e-41b2-b4c7-fba1c29a8385\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp"
Sep 17 22:40:38.232146 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:38.232089    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/576beb41-890e-41b2-b4c7-fba1c29a8385-home\") pod \"stop-feature-test-kserve-85486f684b-mtlrp\" (UID: \"576beb41-890e-41b2-b4c7-fba1c29a8385\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp"
Sep 17 22:40:38.232146 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:38.232135    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/576beb41-890e-41b2-b4c7-fba1c29a8385-tls-certs\") pod \"stop-feature-test-kserve-85486f684b-mtlrp\" (UID: \"576beb41-890e-41b2-b4c7-fba1c29a8385\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp"
Sep 17 22:40:38.232408 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:38.232217    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/576beb41-890e-41b2-b4c7-fba1c29a8385-dshm\") pod \"stop-feature-test-kserve-85486f684b-mtlrp\" (UID: \"576beb41-890e-41b2-b4c7-fba1c29a8385\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp"
Sep 17 22:40:38.232408 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:38.232292    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/576beb41-890e-41b2-b4c7-fba1c29a8385-kserve-provision-location\") pod \"stop-feature-test-kserve-85486f684b-mtlrp\" (UID: \"576beb41-890e-41b2-b4c7-fba1c29a8385\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp"
Sep 17 22:40:38.232408 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:38.232309    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/576beb41-890e-41b2-b4c7-fba1c29a8385-model-cache\") pod \"stop-feature-test-kserve-85486f684b-mtlrp\" (UID: \"576beb41-890e-41b2-b4c7-fba1c29a8385\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp"
Sep 17 22:40:38.232408 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:38.232335    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/576beb41-890e-41b2-b4c7-fba1c29a8385-tmp-dir\") pod \"stop-feature-test-kserve-85486f684b-mtlrp\" (UID: \"576beb41-890e-41b2-b4c7-fba1c29a8385\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp"
Sep 17 22:40:38.232676 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:38.232613    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/576beb41-890e-41b2-b4c7-fba1c29a8385-model-cache\") pod \"stop-feature-test-kserve-85486f684b-mtlrp\" (UID: \"576beb41-890e-41b2-b4c7-fba1c29a8385\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp"
Sep 17 22:40:38.232676 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:38.232637    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/576beb41-890e-41b2-b4c7-fba1c29a8385-home\") pod \"stop-feature-test-kserve-85486f684b-mtlrp\" (UID: \"576beb41-890e-41b2-b4c7-fba1c29a8385\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp"
Sep 17 22:40:38.232790 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:38.232712    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/576beb41-890e-41b2-b4c7-fba1c29a8385-cabundle-cert\") pod \"stop-feature-test-kserve-85486f684b-mtlrp\" (UID: \"576beb41-890e-41b2-b4c7-fba1c29a8385\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp"
Sep 17 22:40:38.234716 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:38.234692    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/576beb41-890e-41b2-b4c7-fba1c29a8385-dshm\") pod \"stop-feature-test-kserve-85486f684b-mtlrp\" (UID: \"576beb41-890e-41b2-b4c7-fba1c29a8385\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp"
Sep 17 22:40:38.235070 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:38.235047    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/576beb41-890e-41b2-b4c7-fba1c29a8385-tls-certs\") pod \"stop-feature-test-kserve-85486f684b-mtlrp\" (UID: \"576beb41-890e-41b2-b4c7-fba1c29a8385\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp"
Sep 17 22:40:38.239953 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:38.239929    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9wph\" (UniqueName: \"kubernetes.io/projected/576beb41-890e-41b2-b4c7-fba1c29a8385-kube-api-access-j9wph\") pod \"stop-feature-test-kserve-85486f684b-mtlrp\" (UID: \"576beb41-890e-41b2-b4c7-fba1c29a8385\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp"
Sep 17 22:40:38.316156 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:38.316062    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:40:38.349522 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:38.349484    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp"
Sep 17 22:40:38.631382 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:38.631354    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp"]
Sep 17 22:40:38.632929 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:40:38.632900    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod576beb41_890e_41b2_b4c7_fba1c29a8385.slice/crio-c0cbaa705ac4d0f3314a51d398efc0a056f6add5fd139b6fbf63b83831ebab5a WatchSource:0}: Error finding container c0cbaa705ac4d0f3314a51d398efc0a056f6add5fd139b6fbf63b83831ebab5a: Status 404 returned error can't find the container with id c0cbaa705ac4d0f3314a51d398efc0a056f6add5fd139b6fbf63b83831ebab5a
Sep 17 22:40:39.298500 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:39.298059    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp" event={"ID":"576beb41-890e-41b2-b4c7-fba1c29a8385","Type":"ContainerStarted","Data":"d8afd82283751404f0664ddcc192c32e4f6774704e598217612cb8d8b8971aeb"}
Sep 17 22:40:39.298500 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:39.298174    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp" event={"ID":"576beb41-890e-41b2-b4c7-fba1c29a8385","Type":"ContainerStarted","Data":"c0cbaa705ac4d0f3314a51d398efc0a056f6add5fd139b6fbf63b83831ebab5a"}
Sep 17 22:40:39.841237 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:39.841164    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6" podUID="25ac006a-2265-4c27-95bb-c9d6cbb7fd43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 17 22:40:41.312742 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:41.312703    2555 generic.go:356] "Generic (PLEG): container finished" podID="576beb41-890e-41b2-b4c7-fba1c29a8385" containerID="d8afd82283751404f0664ddcc192c32e4f6774704e598217612cb8d8b8971aeb" exitCode=0
Sep 17 22:40:41.313255 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:41.312776    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp" event={"ID":"576beb41-890e-41b2-b4c7-fba1c29a8385","Type":"ContainerDied","Data":"d8afd82283751404f0664ddcc192c32e4f6774704e598217612cb8d8b8971aeb"}
Sep 17 22:40:42.322236 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:42.322174    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp" event={"ID":"576beb41-890e-41b2-b4c7-fba1c29a8385","Type":"ContainerStarted","Data":"e52136aedcff314586f57620561ed1cbe6bc6a07a10f546f96ad511ee6f0d8a5"}
Sep 17 22:40:42.346595 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:42.346543    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp" podStartSLOduration=1.968623852 podStartE2EDuration="4.346528842s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.37790499s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:40:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:40:42.343862812 +0000 UTC m=+2543.789908084" watchObservedRunningTime="2026-09-17 22:40:42.346528842 +0000 UTC m=+2543.792574113"
Sep 17 22:40:42.349893 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:42.349868    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp"
Sep 17 22:40:48.349910 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:48.349876    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp"
Sep 17 22:40:48.351674 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:48.351644    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp" podUID="576beb41-890e-41b2-b4c7-fba1c29a8385" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Sep 17 22:40:49.841504 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:49.841451    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6" podUID="25ac006a-2265-4c27-95bb-c9d6cbb7fd43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 17 22:40:53.436309 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:53.436272    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:40:58.350503 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:58.350449    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp" podUID="576beb41-890e-41b2-b4c7-fba1c29a8385" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Sep 17 22:40:59.336255 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:59.336224    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:40:59.389437 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:59.389398    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:40:59.842159 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:40:59.842117    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6" podUID="25ac006a-2265-4c27-95bb-c9d6cbb7fd43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 17 22:41:08.350832 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:08.350783    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp" podUID="576beb41-890e-41b2-b4c7-fba1c29a8385" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Sep 17 22:41:09.841282 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:09.841231    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6" podUID="25ac006a-2265-4c27-95bb-c9d6cbb7fd43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 17 22:41:18.350132 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:18.350088    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp" podUID="576beb41-890e-41b2-b4c7-fba1c29a8385" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Sep 17 22:41:19.851232 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:19.851176    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6"
Sep 17 22:41:19.859195 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:19.859148    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6"
Sep 17 22:41:25.085925 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:25.085838    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:41:25.097268 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:25.097236    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6"]
Sep 17 22:41:25.181813 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:25.181781    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:41:28.349931 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:28.349878    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp" podUID="576beb41-890e-41b2-b4c7-fba1c29a8385" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Sep 17 22:41:38.349884 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:38.349792    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp" podUID="576beb41-890e-41b2-b4c7-fba1c29a8385" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Sep 17 22:41:40.125427 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:40.125356    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6" podUID="25ac006a-2265-4c27-95bb-c9d6cbb7fd43" containerName="main" containerID="cri-o://099192196fcd4be1deba306a72563dbe4cdc9e92fa912056aa2ed4d927ab2cac" gracePeriod=45
Sep 17 22:41:40.869602 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:40.869578    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6"
Sep 17 22:41:41.025641 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:41.025539    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-home\") pod \"25ac006a-2265-4c27-95bb-c9d6cbb7fd43\" (UID: \"25ac006a-2265-4c27-95bb-c9d6cbb7fd43\") "
Sep 17 22:41:41.025641 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:41.025586    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-model-cache\") pod \"25ac006a-2265-4c27-95bb-c9d6cbb7fd43\" (UID: \"25ac006a-2265-4c27-95bb-c9d6cbb7fd43\") "
Sep 17 22:41:41.025641 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:41.025622    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ttp5\" (UniqueName: \"kubernetes.io/projected/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-kube-api-access-5ttp5\") pod \"25ac006a-2265-4c27-95bb-c9d6cbb7fd43\" (UID: \"25ac006a-2265-4c27-95bb-c9d6cbb7fd43\") "
Sep 17 22:41:41.025641 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:41.025643    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-dshm\") pod \"25ac006a-2265-4c27-95bb-c9d6cbb7fd43\" (UID: \"25ac006a-2265-4c27-95bb-c9d6cbb7fd43\") "
Sep 17 22:41:41.025995 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:41.025698    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-tmp-dir\") pod \"25ac006a-2265-4c27-95bb-c9d6cbb7fd43\" (UID: \"25ac006a-2265-4c27-95bb-c9d6cbb7fd43\") "
Sep 17 22:41:41.025995 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:41.025741    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-tls-certs\") pod \"25ac006a-2265-4c27-95bb-c9d6cbb7fd43\" (UID: \"25ac006a-2265-4c27-95bb-c9d6cbb7fd43\") "
Sep 17 22:41:41.025995 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:41.025800    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-cabundle-cert\") pod \"25ac006a-2265-4c27-95bb-c9d6cbb7fd43\" (UID: \"25ac006a-2265-4c27-95bb-c9d6cbb7fd43\") "
Sep 17 22:41:41.025995 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:41.025849    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-kserve-provision-location\") pod \"25ac006a-2265-4c27-95bb-c9d6cbb7fd43\" (UID: \"25ac006a-2265-4c27-95bb-c9d6cbb7fd43\") "
Sep 17 22:41:41.025995 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:41.025919    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-model-cache" (OuterVolumeSpecName: "model-cache") pod "25ac006a-2265-4c27-95bb-c9d6cbb7fd43" (UID: "25ac006a-2265-4c27-95bb-c9d6cbb7fd43"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:41:41.026281 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:41.026203    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "25ac006a-2265-4c27-95bb-c9d6cbb7fd43" (UID: "25ac006a-2265-4c27-95bb-c9d6cbb7fd43"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:41:41.026281 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:41.026224    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-model-cache\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:41:41.026395 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:41.026378    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-home" (OuterVolumeSpecName: "home") pod "25ac006a-2265-4c27-95bb-c9d6cbb7fd43" (UID: "25ac006a-2265-4c27-95bb-c9d6cbb7fd43"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:41:41.028234 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:41.028174    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-kube-api-access-5ttp5" (OuterVolumeSpecName: "kube-api-access-5ttp5") pod "25ac006a-2265-4c27-95bb-c9d6cbb7fd43" (UID: "25ac006a-2265-4c27-95bb-c9d6cbb7fd43"). InnerVolumeSpecName "kube-api-access-5ttp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:41:41.028460 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:41.028431    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "25ac006a-2265-4c27-95bb-c9d6cbb7fd43" (UID: "25ac006a-2265-4c27-95bb-c9d6cbb7fd43"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:41:41.029358 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:41.029333    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-dshm" (OuterVolumeSpecName: "dshm") pod "25ac006a-2265-4c27-95bb-c9d6cbb7fd43" (UID: "25ac006a-2265-4c27-95bb-c9d6cbb7fd43"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:41:41.039795 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:41.039769    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "25ac006a-2265-4c27-95bb-c9d6cbb7fd43" (UID: "25ac006a-2265-4c27-95bb-c9d6cbb7fd43"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:41:41.055563 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:41.055541    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "25ac006a-2265-4c27-95bb-c9d6cbb7fd43" (UID: "25ac006a-2265-4c27-95bb-c9d6cbb7fd43"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:41:41.127094 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:41.127066    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:41:41.127094 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:41.127094    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-home\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:41:41.127470 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:41.127104    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ttp5\" (UniqueName: \"kubernetes.io/projected/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-kube-api-access-5ttp5\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:41:41.127470 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:41.127113    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-dshm\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:41:41.127470 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:41.127121    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-tmp-dir\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:41:41.127470 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:41.127128    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-tls-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:41:41.127470 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:41.127136    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25ac006a-2265-4c27-95bb-c9d6cbb7fd43-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:41:41.690711 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:41.690675    2555 generic.go:356] "Generic (PLEG): container finished" podID="25ac006a-2265-4c27-95bb-c9d6cbb7fd43" containerID="099192196fcd4be1deba306a72563dbe4cdc9e92fa912056aa2ed4d927ab2cac" exitCode=0
Sep 17 22:41:41.690918 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:41.690754    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6"
Sep 17 22:41:41.690918 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:41.690755    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6" event={"ID":"25ac006a-2265-4c27-95bb-c9d6cbb7fd43","Type":"ContainerDied","Data":"099192196fcd4be1deba306a72563dbe4cdc9e92fa912056aa2ed4d927ab2cac"}
Sep 17 22:41:41.690918 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:41.690865    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6" event={"ID":"25ac006a-2265-4c27-95bb-c9d6cbb7fd43","Type":"ContainerDied","Data":"4fa80a74e1b566aaa48c8427975de604177ff9348612eae75ae5cfd6e6bdffeb"}
Sep 17 22:41:41.690918 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:41.690887    2555 scope.go:117] "RemoveContainer" containerID="099192196fcd4be1deba306a72563dbe4cdc9e92fa912056aa2ed4d927ab2cac"
Sep 17 22:41:41.701392 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:41.701374    2555 scope.go:117] "RemoveContainer" containerID="0b8f8f281b2bdd424c5a56e3b9be4ca8ad61e25be07ca65821125f413fd91976"
Sep 17 22:41:41.710669 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:41.710650    2555 scope.go:117] "RemoveContainer" containerID="099192196fcd4be1deba306a72563dbe4cdc9e92fa912056aa2ed4d927ab2cac"
Sep 17 22:41:41.710927 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:41:41.710909    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"099192196fcd4be1deba306a72563dbe4cdc9e92fa912056aa2ed4d927ab2cac\": container with ID starting with 099192196fcd4be1deba306a72563dbe4cdc9e92fa912056aa2ed4d927ab2cac not found: ID does not exist" containerID="099192196fcd4be1deba306a72563dbe4cdc9e92fa912056aa2ed4d927ab2cac"
Sep 17 22:41:41.710990 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:41.710936    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"099192196fcd4be1deba306a72563dbe4cdc9e92fa912056aa2ed4d927ab2cac"} err="failed to get container status \"099192196fcd4be1deba306a72563dbe4cdc9e92fa912056aa2ed4d927ab2cac\": rpc error: code = NotFound desc = could not find container \"099192196fcd4be1deba306a72563dbe4cdc9e92fa912056aa2ed4d927ab2cac\": container with ID starting with 099192196fcd4be1deba306a72563dbe4cdc9e92fa912056aa2ed4d927ab2cac not found: ID does not exist"
Sep 17 22:41:41.710990 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:41.710952    2555 scope.go:117] "RemoveContainer" containerID="0b8f8f281b2bdd424c5a56e3b9be4ca8ad61e25be07ca65821125f413fd91976"
Sep 17 22:41:41.711280 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:41:41.711251    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b8f8f281b2bdd424c5a56e3b9be4ca8ad61e25be07ca65821125f413fd91976\": container with ID starting with 0b8f8f281b2bdd424c5a56e3b9be4ca8ad61e25be07ca65821125f413fd91976 not found: ID does not exist" containerID="0b8f8f281b2bdd424c5a56e3b9be4ca8ad61e25be07ca65821125f413fd91976"
Sep 17 22:41:41.711280 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:41.711288    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b8f8f281b2bdd424c5a56e3b9be4ca8ad61e25be07ca65821125f413fd91976"} err="failed to get container status \"0b8f8f281b2bdd424c5a56e3b9be4ca8ad61e25be07ca65821125f413fd91976\": rpc error: code = NotFound desc = could not find container \"0b8f8f281b2bdd424c5a56e3b9be4ca8ad61e25be07ca65821125f413fd91976\": container with ID starting with 0b8f8f281b2bdd424c5a56e3b9be4ca8ad61e25be07ca65821125f413fd91976 not found: ID does not exist"
Sep 17 22:41:41.714045 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:41.714023    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6"]
Sep 17 22:41:41.717491 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:41.717469    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6"]
Sep 17 22:41:41.852861 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:41.852799    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d9c7fcf8d-zsrn6" podUID="25ac006a-2265-4c27-95bb-c9d6cbb7fd43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": context deadline exceeded"
Sep 17 22:41:42.681423 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:42.681387    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25ac006a-2265-4c27-95bb-c9d6cbb7fd43" path="/var/lib/kubelet/pods/25ac006a-2265-4c27-95bb-c9d6cbb7fd43/volumes"
Sep 17 22:41:47.843320 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:47.843282    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:41:48.350300 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:48.350254    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp" podUID="576beb41-890e-41b2-b4c7-fba1c29a8385" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Sep 17 22:41:58.350592 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:41:58.350540    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp" podUID="576beb41-890e-41b2-b4c7-fba1c29a8385" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Sep 17 22:42:08.350126 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:08.350080    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp" podUID="576beb41-890e-41b2-b4c7-fba1c29a8385" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Sep 17 22:42:15.022970 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:15.022937    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:42:15.121474 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:15.121439    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:42:15.227637 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:15.227603    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:42:15.323531 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:15.323447    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:42:18.280108 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:18.280029    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:42:18.350687 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:18.350649    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp" podUID="576beb41-890e-41b2-b4c7-fba1c29a8385" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Sep 17 22:42:21.200937 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:21.200904    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:42:26.794937 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:26.794907    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:42:26.895298 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:26.895267    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:42:28.350274 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:28.350227    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp" podUID="576beb41-890e-41b2-b4c7-fba1c29a8385" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Sep 17 22:42:29.252908 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.252867    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng"]
Sep 17 22:42:29.253272 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.253258    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25ac006a-2265-4c27-95bb-c9d6cbb7fd43" containerName="storage-initializer"
Sep 17 22:42:29.253349 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.253274    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="25ac006a-2265-4c27-95bb-c9d6cbb7fd43" containerName="storage-initializer"
Sep 17 22:42:29.253349 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.253290    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25ac006a-2265-4c27-95bb-c9d6cbb7fd43" containerName="main"
Sep 17 22:42:29.253349 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.253295    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="25ac006a-2265-4c27-95bb-c9d6cbb7fd43" containerName="main"
Sep 17 22:42:29.253523 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.253406    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="25ac006a-2265-4c27-95bb-c9d6cbb7fd43" containerName="main"
Sep 17 22:42:29.257969 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.257946    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng"
Sep 17 22:42:29.260760 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.260733    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 17 22:42:29.260895 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.260771    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:42:29.260895 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.260818    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\""
Sep 17 22:42:29.260895 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.260886    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-572zm\""
Sep 17 22:42:29.261082 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.261071    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 17 22:42:29.261166 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.261143    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 17 22:42:29.269784 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.269762    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng"]
Sep 17 22:42:29.285766 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.285700    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:42:29.340953 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.340927    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc5b869b-bcbd-4c76-af22-ef150e14981b-kserve-provision-location\") pod \"router-with-refs-test-kserve-9c56b4c96-svrng\" (UID: \"bc5b869b-bcbd-4c76-af22-ef150e14981b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng"
Sep 17 22:42:29.341102 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.340971    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc5b869b-bcbd-4c76-af22-ef150e14981b-tls-certs\") pod \"router-with-refs-test-kserve-9c56b4c96-svrng\" (UID: \"bc5b869b-bcbd-4c76-af22-ef150e14981b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng"
Sep 17 22:42:29.341102 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.341009    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc5b869b-bcbd-4c76-af22-ef150e14981b-model-cache\") pod \"router-with-refs-test-kserve-9c56b4c96-svrng\" (UID: \"bc5b869b-bcbd-4c76-af22-ef150e14981b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng"
Sep 17 22:42:29.341102 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.341039    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc5b869b-bcbd-4c76-af22-ef150e14981b-home\") pod \"router-with-refs-test-kserve-9c56b4c96-svrng\" (UID: \"bc5b869b-bcbd-4c76-af22-ef150e14981b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng"
Sep 17 22:42:29.341271 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.341107    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc5b869b-bcbd-4c76-af22-ef150e14981b-dshm\") pod \"router-with-refs-test-kserve-9c56b4c96-svrng\" (UID: \"bc5b869b-bcbd-4c76-af22-ef150e14981b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng"
Sep 17 22:42:29.341271 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.341127    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bc5b869b-bcbd-4c76-af22-ef150e14981b-cabundle-cert\") pod \"router-with-refs-test-kserve-9c56b4c96-svrng\" (UID: \"bc5b869b-bcbd-4c76-af22-ef150e14981b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng"
Sep 17 22:42:29.341271 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.341213    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc5b869b-bcbd-4c76-af22-ef150e14981b-tmp-dir\") pod \"router-with-refs-test-kserve-9c56b4c96-svrng\" (UID: \"bc5b869b-bcbd-4c76-af22-ef150e14981b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng"
Sep 17 22:42:29.341271 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.341236    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxr6w\" (UniqueName: \"kubernetes.io/projected/bc5b869b-bcbd-4c76-af22-ef150e14981b-kube-api-access-zxr6w\") pod \"router-with-refs-test-kserve-9c56b4c96-svrng\" (UID: \"bc5b869b-bcbd-4c76-af22-ef150e14981b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng"
Sep 17 22:42:29.442018 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.441982    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc5b869b-bcbd-4c76-af22-ef150e14981b-home\") pod \"router-with-refs-test-kserve-9c56b4c96-svrng\" (UID: \"bc5b869b-bcbd-4c76-af22-ef150e14981b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng"
Sep 17 22:42:29.442522 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.442027    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc5b869b-bcbd-4c76-af22-ef150e14981b-dshm\") pod \"router-with-refs-test-kserve-9c56b4c96-svrng\" (UID: \"bc5b869b-bcbd-4c76-af22-ef150e14981b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng"
Sep 17 22:42:29.442522 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.442043    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bc5b869b-bcbd-4c76-af22-ef150e14981b-cabundle-cert\") pod \"router-with-refs-test-kserve-9c56b4c96-svrng\" (UID: \"bc5b869b-bcbd-4c76-af22-ef150e14981b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng"
Sep 17 22:42:29.442522 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.442072    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc5b869b-bcbd-4c76-af22-ef150e14981b-tmp-dir\") pod \"router-with-refs-test-kserve-9c56b4c96-svrng\" (UID: \"bc5b869b-bcbd-4c76-af22-ef150e14981b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng"
Sep 17 22:42:29.442522 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.442088    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxr6w\" (UniqueName: \"kubernetes.io/projected/bc5b869b-bcbd-4c76-af22-ef150e14981b-kube-api-access-zxr6w\") pod \"router-with-refs-test-kserve-9c56b4c96-svrng\" (UID: \"bc5b869b-bcbd-4c76-af22-ef150e14981b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng"
Sep 17 22:42:29.442522 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.442254    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc5b869b-bcbd-4c76-af22-ef150e14981b-kserve-provision-location\") pod \"router-with-refs-test-kserve-9c56b4c96-svrng\" (UID: \"bc5b869b-bcbd-4c76-af22-ef150e14981b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng"
Sep 17 22:42:29.442522 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.442302    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc5b869b-bcbd-4c76-af22-ef150e14981b-tls-certs\") pod \"router-with-refs-test-kserve-9c56b4c96-svrng\" (UID: \"bc5b869b-bcbd-4c76-af22-ef150e14981b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng"
Sep 17 22:42:29.442522 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.442344    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc5b869b-bcbd-4c76-af22-ef150e14981b-model-cache\") pod \"router-with-refs-test-kserve-9c56b4c96-svrng\" (UID: \"bc5b869b-bcbd-4c76-af22-ef150e14981b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng"
Sep 17 22:42:29.442804 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.442521    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc5b869b-bcbd-4c76-af22-ef150e14981b-home\") pod \"router-with-refs-test-kserve-9c56b4c96-svrng\" (UID: \"bc5b869b-bcbd-4c76-af22-ef150e14981b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng"
Sep 17 22:42:29.442804 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.442543    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc5b869b-bcbd-4c76-af22-ef150e14981b-tmp-dir\") pod \"router-with-refs-test-kserve-9c56b4c96-svrng\" (UID: \"bc5b869b-bcbd-4c76-af22-ef150e14981b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng"
Sep 17 22:42:29.442804 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.442613    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc5b869b-bcbd-4c76-af22-ef150e14981b-kserve-provision-location\") pod \"router-with-refs-test-kserve-9c56b4c96-svrng\" (UID: \"bc5b869b-bcbd-4c76-af22-ef150e14981b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng"
Sep 17 22:42:29.442804 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.442685    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc5b869b-bcbd-4c76-af22-ef150e14981b-model-cache\") pod \"router-with-refs-test-kserve-9c56b4c96-svrng\" (UID: \"bc5b869b-bcbd-4c76-af22-ef150e14981b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng"
Sep 17 22:42:29.442982 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.442892    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bc5b869b-bcbd-4c76-af22-ef150e14981b-cabundle-cert\") pod \"router-with-refs-test-kserve-9c56b4c96-svrng\" (UID: \"bc5b869b-bcbd-4c76-af22-ef150e14981b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng"
Sep 17 22:42:29.444624 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.444599    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc5b869b-bcbd-4c76-af22-ef150e14981b-dshm\") pod \"router-with-refs-test-kserve-9c56b4c96-svrng\" (UID: \"bc5b869b-bcbd-4c76-af22-ef150e14981b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng"
Sep 17 22:42:29.444946 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.444928    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc5b869b-bcbd-4c76-af22-ef150e14981b-tls-certs\") pod \"router-with-refs-test-kserve-9c56b4c96-svrng\" (UID: \"bc5b869b-bcbd-4c76-af22-ef150e14981b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng"
Sep 17 22:42:29.450072 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.450050    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxr6w\" (UniqueName: \"kubernetes.io/projected/bc5b869b-bcbd-4c76-af22-ef150e14981b-kube-api-access-zxr6w\") pod \"router-with-refs-test-kserve-9c56b4c96-svrng\" (UID: \"bc5b869b-bcbd-4c76-af22-ef150e14981b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng"
Sep 17 22:42:29.570131 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.570035    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng"
Sep 17 22:42:29.642293 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.642151    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:42:29.741710 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.741683    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:42:29.869695 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.869668    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng"]
Sep 17 22:42:29.871273 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:42:29.871241    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc5b869b_bcbd_4c76_af22_ef150e14981b.slice/crio-4daab57975813dd1336c803ceb8ded7c373b8b55d23668d8bfbde223d96a6f1e WatchSource:0}: Error finding container 4daab57975813dd1336c803ceb8ded7c373b8b55d23668d8bfbde223d96a6f1e: Status 404 returned error can't find the container with id 4daab57975813dd1336c803ceb8ded7c373b8b55d23668d8bfbde223d96a6f1e
Sep 17 22:42:29.969769 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.969737    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng" event={"ID":"bc5b869b-bcbd-4c76-af22-ef150e14981b","Type":"ContainerStarted","Data":"b58b84c9880b7a9670b68154e7aa4d7427a8dcd3a81cb54472499721eaa9d162"}
Sep 17 22:42:29.969885 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:29.969798    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng" event={"ID":"bc5b869b-bcbd-4c76-af22-ef150e14981b","Type":"ContainerStarted","Data":"4daab57975813dd1336c803ceb8ded7c373b8b55d23668d8bfbde223d96a6f1e"}
Sep 17 22:42:31.937830 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:31.937785    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:42:31.991719 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:31.991672    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:42:32.992699 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:32.992665    2555 generic.go:356] "Generic (PLEG): container finished" podID="bc5b869b-bcbd-4c76-af22-ef150e14981b" containerID="b58b84c9880b7a9670b68154e7aa4d7427a8dcd3a81cb54472499721eaa9d162" exitCode=0
Sep 17 22:42:32.993075 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:32.992738    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng" event={"ID":"bc5b869b-bcbd-4c76-af22-ef150e14981b","Type":"ContainerDied","Data":"b58b84c9880b7a9670b68154e7aa4d7427a8dcd3a81cb54472499721eaa9d162"}
Sep 17 22:42:34.000004 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:33.999968    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng" event={"ID":"bc5b869b-bcbd-4c76-af22-ef150e14981b","Type":"ContainerStarted","Data":"12b5e99b17f869b06ae40ab40275ee4bddf9f0f12799b99fa272702babce0aa8"}
Sep 17 22:42:34.021411 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:34.021359    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng" podStartSLOduration=2.69833956 podStartE2EDuration="5.021341359s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.323001799s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:42:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:42:34.019413844 +0000 UTC m=+2655.465459118" watchObservedRunningTime="2026-09-17 22:42:34.021341359 +0000 UTC m=+2655.467386632"
Sep 17 22:42:34.570654 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:34.570609    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng"
Sep 17 22:42:38.359976 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:38.359937    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp"
Sep 17 22:42:38.367437 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:38.367406    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp"
Sep 17 22:42:39.570794 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:39.570750    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng"
Sep 17 22:42:39.572619 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:39.572589    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng" podUID="bc5b869b-bcbd-4c76-af22-ef150e14981b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 17 22:42:48.495410 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:48.495365    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:42:49.571065 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:49.571022    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng" podUID="bc5b869b-bcbd-4c76-af22-ef150e14981b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 17 22:42:49.849646 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:49.849546    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp"]
Sep 17 22:42:49.864887 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:49.864857    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:42:49.931768 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:49.931734    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:42:50.024478 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:50.024443    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:42:59.570878 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:42:59.570832    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng" podUID="bc5b869b-bcbd-4c76-af22-ef150e14981b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 17 22:43:02.876205 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:02.876137    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:43:03.168529 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:03.168437    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:43:04.873539 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:04.873493    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp" podUID="576beb41-890e-41b2-b4c7-fba1c29a8385" containerName="main" containerID="cri-o://e52136aedcff314586f57620561ed1cbe6bc6a07a10f546f96ad511ee6f0d8a5" gracePeriod=45
Sep 17 22:43:05.360848 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:05.360795    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp" podUID="576beb41-890e-41b2-b4c7-fba1c29a8385" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Sep 17 22:43:05.629393 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:05.629364    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp"
Sep 17 22:43:05.685932 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:05.685895    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/576beb41-890e-41b2-b4c7-fba1c29a8385-kserve-provision-location\") pod \"576beb41-890e-41b2-b4c7-fba1c29a8385\" (UID: \"576beb41-890e-41b2-b4c7-fba1c29a8385\") "
Sep 17 22:43:05.686125 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:05.685951    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/576beb41-890e-41b2-b4c7-fba1c29a8385-tmp-dir\") pod \"576beb41-890e-41b2-b4c7-fba1c29a8385\" (UID: \"576beb41-890e-41b2-b4c7-fba1c29a8385\") "
Sep 17 22:43:05.686125 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:05.685978    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/576beb41-890e-41b2-b4c7-fba1c29a8385-tls-certs\") pod \"576beb41-890e-41b2-b4c7-fba1c29a8385\" (UID: \"576beb41-890e-41b2-b4c7-fba1c29a8385\") "
Sep 17 22:43:05.686125 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:05.686026    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/576beb41-890e-41b2-b4c7-fba1c29a8385-model-cache\") pod \"576beb41-890e-41b2-b4c7-fba1c29a8385\" (UID: \"576beb41-890e-41b2-b4c7-fba1c29a8385\") "
Sep 17 22:43:05.686125 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:05.686051    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/576beb41-890e-41b2-b4c7-fba1c29a8385-cabundle-cert\") pod \"576beb41-890e-41b2-b4c7-fba1c29a8385\" (UID: \"576beb41-890e-41b2-b4c7-fba1c29a8385\") "
Sep 17 22:43:05.686125 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:05.686091    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/576beb41-890e-41b2-b4c7-fba1c29a8385-home\") pod \"576beb41-890e-41b2-b4c7-fba1c29a8385\" (UID: \"576beb41-890e-41b2-b4c7-fba1c29a8385\") "
Sep 17 22:43:05.686427 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:05.686137    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9wph\" (UniqueName: \"kubernetes.io/projected/576beb41-890e-41b2-b4c7-fba1c29a8385-kube-api-access-j9wph\") pod \"576beb41-890e-41b2-b4c7-fba1c29a8385\" (UID: \"576beb41-890e-41b2-b4c7-fba1c29a8385\") "
Sep 17 22:43:05.686427 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:05.686223    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/576beb41-890e-41b2-b4c7-fba1c29a8385-dshm\") pod \"576beb41-890e-41b2-b4c7-fba1c29a8385\" (UID: \"576beb41-890e-41b2-b4c7-fba1c29a8385\") "
Sep 17 22:43:05.686427 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:05.686368    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576beb41-890e-41b2-b4c7-fba1c29a8385-model-cache" (OuterVolumeSpecName: "model-cache") pod "576beb41-890e-41b2-b4c7-fba1c29a8385" (UID: "576beb41-890e-41b2-b4c7-fba1c29a8385"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:43:05.686635 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:05.686560    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/576beb41-890e-41b2-b4c7-fba1c29a8385-model-cache\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:43:05.687438 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:05.687339    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576beb41-890e-41b2-b4c7-fba1c29a8385-home" (OuterVolumeSpecName: "home") pod "576beb41-890e-41b2-b4c7-fba1c29a8385" (UID: "576beb41-890e-41b2-b4c7-fba1c29a8385"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:43:05.688157 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:05.688123    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/576beb41-890e-41b2-b4c7-fba1c29a8385-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "576beb41-890e-41b2-b4c7-fba1c29a8385" (UID: "576beb41-890e-41b2-b4c7-fba1c29a8385"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:43:05.689241 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:05.689202    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/576beb41-890e-41b2-b4c7-fba1c29a8385-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "576beb41-890e-41b2-b4c7-fba1c29a8385" (UID: "576beb41-890e-41b2-b4c7-fba1c29a8385"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:43:05.689827 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:05.689795    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576beb41-890e-41b2-b4c7-fba1c29a8385-dshm" (OuterVolumeSpecName: "dshm") pod "576beb41-890e-41b2-b4c7-fba1c29a8385" (UID: "576beb41-890e-41b2-b4c7-fba1c29a8385"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:43:05.690530 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:05.690475    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/576beb41-890e-41b2-b4c7-fba1c29a8385-kube-api-access-j9wph" (OuterVolumeSpecName: "kube-api-access-j9wph") pod "576beb41-890e-41b2-b4c7-fba1c29a8385" (UID: "576beb41-890e-41b2-b4c7-fba1c29a8385"). InnerVolumeSpecName "kube-api-access-j9wph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:43:05.706254 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:05.706216    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576beb41-890e-41b2-b4c7-fba1c29a8385-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "576beb41-890e-41b2-b4c7-fba1c29a8385" (UID: "576beb41-890e-41b2-b4c7-fba1c29a8385"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:43:05.710214 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:05.710170    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576beb41-890e-41b2-b4c7-fba1c29a8385-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "576beb41-890e-41b2-b4c7-fba1c29a8385" (UID: "576beb41-890e-41b2-b4c7-fba1c29a8385"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:43:05.787360 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:05.787316    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/576beb41-890e-41b2-b4c7-fba1c29a8385-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:43:05.787360 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:05.787357    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/576beb41-890e-41b2-b4c7-fba1c29a8385-home\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:43:05.787360 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:05.787373    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9wph\" (UniqueName: \"kubernetes.io/projected/576beb41-890e-41b2-b4c7-fba1c29a8385-kube-api-access-j9wph\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:43:05.787685 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:05.787385    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/576beb41-890e-41b2-b4c7-fba1c29a8385-dshm\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:43:05.787685 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:05.787401    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/576beb41-890e-41b2-b4c7-fba1c29a8385-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:43:05.787685 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:05.787416    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/576beb41-890e-41b2-b4c7-fba1c29a8385-tmp-dir\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:43:05.787685 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:05.787429    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/576beb41-890e-41b2-b4c7-fba1c29a8385-tls-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:43:06.206470 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:06.206432    2555 generic.go:356] "Generic (PLEG): container finished" podID="576beb41-890e-41b2-b4c7-fba1c29a8385" containerID="e52136aedcff314586f57620561ed1cbe6bc6a07a10f546f96ad511ee6f0d8a5" exitCode=0
Sep 17 22:43:06.206957 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:06.206490    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp"
Sep 17 22:43:06.206957 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:06.206524    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp" event={"ID":"576beb41-890e-41b2-b4c7-fba1c29a8385","Type":"ContainerDied","Data":"e52136aedcff314586f57620561ed1cbe6bc6a07a10f546f96ad511ee6f0d8a5"}
Sep 17 22:43:06.206957 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:06.206599    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp" event={"ID":"576beb41-890e-41b2-b4c7-fba1c29a8385","Type":"ContainerDied","Data":"c0cbaa705ac4d0f3314a51d398efc0a056f6add5fd139b6fbf63b83831ebab5a"}
Sep 17 22:43:06.206957 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:06.206635    2555 scope.go:117] "RemoveContainer" containerID="e52136aedcff314586f57620561ed1cbe6bc6a07a10f546f96ad511ee6f0d8a5"
Sep 17 22:43:06.217081 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:06.217061    2555 scope.go:117] "RemoveContainer" containerID="d8afd82283751404f0664ddcc192c32e4f6774704e598217612cb8d8b8971aeb"
Sep 17 22:43:06.225802 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:06.225780    2555 scope.go:117] "RemoveContainer" containerID="e52136aedcff314586f57620561ed1cbe6bc6a07a10f546f96ad511ee6f0d8a5"
Sep 17 22:43:06.226069 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:43:06.226051    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e52136aedcff314586f57620561ed1cbe6bc6a07a10f546f96ad511ee6f0d8a5\": container with ID starting with e52136aedcff314586f57620561ed1cbe6bc6a07a10f546f96ad511ee6f0d8a5 not found: ID does not exist" containerID="e52136aedcff314586f57620561ed1cbe6bc6a07a10f546f96ad511ee6f0d8a5"
Sep 17 22:43:06.226122 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:06.226079    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e52136aedcff314586f57620561ed1cbe6bc6a07a10f546f96ad511ee6f0d8a5"} err="failed to get container status \"e52136aedcff314586f57620561ed1cbe6bc6a07a10f546f96ad511ee6f0d8a5\": rpc error: code = NotFound desc = could not find container \"e52136aedcff314586f57620561ed1cbe6bc6a07a10f546f96ad511ee6f0d8a5\": container with ID starting with e52136aedcff314586f57620561ed1cbe6bc6a07a10f546f96ad511ee6f0d8a5 not found: ID does not exist"
Sep 17 22:43:06.226122 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:06.226095    2555 scope.go:117] "RemoveContainer" containerID="d8afd82283751404f0664ddcc192c32e4f6774704e598217612cb8d8b8971aeb"
Sep 17 22:43:06.226379 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:43:06.226364    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8afd82283751404f0664ddcc192c32e4f6774704e598217612cb8d8b8971aeb\": container with ID starting with d8afd82283751404f0664ddcc192c32e4f6774704e598217612cb8d8b8971aeb not found: ID does not exist" containerID="d8afd82283751404f0664ddcc192c32e4f6774704e598217612cb8d8b8971aeb"
Sep 17 22:43:06.226439 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:06.226383    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8afd82283751404f0664ddcc192c32e4f6774704e598217612cb8d8b8971aeb"} err="failed to get container status \"d8afd82283751404f0664ddcc192c32e4f6774704e598217612cb8d8b8971aeb\": rpc error: code = NotFound desc = could not find container \"d8afd82283751404f0664ddcc192c32e4f6774704e598217612cb8d8b8971aeb\": container with ID starting with d8afd82283751404f0664ddcc192c32e4f6774704e598217612cb8d8b8971aeb not found: ID does not exist"
Sep 17 22:43:06.231521 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:06.231500    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp"]
Sep 17 22:43:06.236628 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:06.236609    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-mtlrp"]
Sep 17 22:43:06.679702 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:06.679668    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="576beb41-890e-41b2-b4c7-fba1c29a8385" path="/var/lib/kubelet/pods/576beb41-890e-41b2-b4c7-fba1c29a8385/volumes"
Sep 17 22:43:09.570739 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:09.570692    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng" podUID="bc5b869b-bcbd-4c76-af22-ef150e14981b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 17 22:43:19.448310 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:19.448266    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:43:19.570478 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:19.570433    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng" podUID="bc5b869b-bcbd-4c76-af22-ef150e14981b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 17 22:43:22.869195 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:22.869146    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-7njvj_a324c9ab-5d42-4d31-8ff7-449f619be774/console-operator/1.log"
Sep 17 22:43:22.873906 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:22.873879    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s7f5_4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf/ovn-acl-logging/0.log"
Sep 17 22:43:22.882154 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:22.882127    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-7njvj_a324c9ab-5d42-4d31-8ff7-449f619be774/console-operator/1.log"
Sep 17 22:43:22.886600 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:22.886583    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s7f5_4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf/ovn-acl-logging/0.log"
Sep 17 22:43:29.570985 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:29.570935    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng" podUID="bc5b869b-bcbd-4c76-af22-ef150e14981b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 17 22:43:30.056766 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:30.056728    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:43:39.571165 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:39.571105    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng" podUID="bc5b869b-bcbd-4c76-af22-ef150e14981b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 17 22:43:49.570826 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:49.570785    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng" podUID="bc5b869b-bcbd-4c76-af22-ef150e14981b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 17 22:43:59.570994 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:43:59.570955    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng" podUID="bc5b869b-bcbd-4c76-af22-ef150e14981b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 17 22:44:09.570683 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:09.570642    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng" podUID="bc5b869b-bcbd-4c76-af22-ef150e14981b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 17 22:44:19.571368 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:19.571319    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng" podUID="bc5b869b-bcbd-4c76-af22-ef150e14981b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 17 22:44:29.580634 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:29.580601    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng"
Sep 17 22:44:29.588553 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:29.588531    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng"
Sep 17 22:44:34.072879 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:34.072757    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:44:34.083496 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:34.083465    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng"]
Sep 17 22:44:34.129808 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:34.129781    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:44:38.451994 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:38.451960    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:44:49.108000 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:49.107957    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng" podUID="bc5b869b-bcbd-4c76-af22-ef150e14981b" containerName="main" containerID="cri-o://12b5e99b17f869b06ae40ab40275ee4bddf9f0f12799b99fa272702babce0aa8" gracePeriod=45
Sep 17 22:44:49.582121 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:49.582072    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng" podUID="bc5b869b-bcbd-4c76-af22-ef150e14981b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 17 22:44:49.803559 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:49.803523    2555 generic.go:356] "Generic (PLEG): container finished" podID="bc5b869b-bcbd-4c76-af22-ef150e14981b" containerID="12b5e99b17f869b06ae40ab40275ee4bddf9f0f12799b99fa272702babce0aa8" exitCode=0
Sep 17 22:44:49.803559 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:49.803551    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng" event={"ID":"bc5b869b-bcbd-4c76-af22-ef150e14981b","Type":"ContainerDied","Data":"12b5e99b17f869b06ae40ab40275ee4bddf9f0f12799b99fa272702babce0aa8"}
Sep 17 22:44:49.864985 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:49.864955    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng"
Sep 17 22:44:50.030295 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:50.030263    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc5b869b-bcbd-4c76-af22-ef150e14981b-tls-certs\") pod \"bc5b869b-bcbd-4c76-af22-ef150e14981b\" (UID: \"bc5b869b-bcbd-4c76-af22-ef150e14981b\") "
Sep 17 22:44:50.030463 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:50.030337    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc5b869b-bcbd-4c76-af22-ef150e14981b-kserve-provision-location\") pod \"bc5b869b-bcbd-4c76-af22-ef150e14981b\" (UID: \"bc5b869b-bcbd-4c76-af22-ef150e14981b\") "
Sep 17 22:44:50.030463 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:50.030360    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc5b869b-bcbd-4c76-af22-ef150e14981b-dshm\") pod \"bc5b869b-bcbd-4c76-af22-ef150e14981b\" (UID: \"bc5b869b-bcbd-4c76-af22-ef150e14981b\") "
Sep 17 22:44:50.030463 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:50.030379    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc5b869b-bcbd-4c76-af22-ef150e14981b-model-cache\") pod \"bc5b869b-bcbd-4c76-af22-ef150e14981b\" (UID: \"bc5b869b-bcbd-4c76-af22-ef150e14981b\") "
Sep 17 22:44:50.030463 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:50.030416    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bc5b869b-bcbd-4c76-af22-ef150e14981b-cabundle-cert\") pod \"bc5b869b-bcbd-4c76-af22-ef150e14981b\" (UID: \"bc5b869b-bcbd-4c76-af22-ef150e14981b\") "
Sep 17 22:44:50.030661 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:50.030465    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc5b869b-bcbd-4c76-af22-ef150e14981b-home\") pod \"bc5b869b-bcbd-4c76-af22-ef150e14981b\" (UID: \"bc5b869b-bcbd-4c76-af22-ef150e14981b\") "
Sep 17 22:44:50.030661 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:50.030498    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc5b869b-bcbd-4c76-af22-ef150e14981b-tmp-dir\") pod \"bc5b869b-bcbd-4c76-af22-ef150e14981b\" (UID: \"bc5b869b-bcbd-4c76-af22-ef150e14981b\") "
Sep 17 22:44:50.030661 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:50.030528    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxr6w\" (UniqueName: \"kubernetes.io/projected/bc5b869b-bcbd-4c76-af22-ef150e14981b-kube-api-access-zxr6w\") pod \"bc5b869b-bcbd-4c76-af22-ef150e14981b\" (UID: \"bc5b869b-bcbd-4c76-af22-ef150e14981b\") "
Sep 17 22:44:50.030828 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:50.030657    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc5b869b-bcbd-4c76-af22-ef150e14981b-model-cache" (OuterVolumeSpecName: "model-cache") pod "bc5b869b-bcbd-4c76-af22-ef150e14981b" (UID: "bc5b869b-bcbd-4c76-af22-ef150e14981b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:44:50.030890 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:50.030832    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc5b869b-bcbd-4c76-af22-ef150e14981b-model-cache\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:44:50.030890 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:50.030827    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc5b869b-bcbd-4c76-af22-ef150e14981b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bc5b869b-bcbd-4c76-af22-ef150e14981b" (UID: "bc5b869b-bcbd-4c76-af22-ef150e14981b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:44:50.031270 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:50.031243    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc5b869b-bcbd-4c76-af22-ef150e14981b-home" (OuterVolumeSpecName: "home") pod "bc5b869b-bcbd-4c76-af22-ef150e14981b" (UID: "bc5b869b-bcbd-4c76-af22-ef150e14981b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:44:50.032744 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:50.032718    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc5b869b-bcbd-4c76-af22-ef150e14981b-dshm" (OuterVolumeSpecName: "dshm") pod "bc5b869b-bcbd-4c76-af22-ef150e14981b" (UID: "bc5b869b-bcbd-4c76-af22-ef150e14981b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:44:50.032946 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:50.032928    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc5b869b-bcbd-4c76-af22-ef150e14981b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bc5b869b-bcbd-4c76-af22-ef150e14981b" (UID: "bc5b869b-bcbd-4c76-af22-ef150e14981b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:44:50.032997 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:50.032971    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc5b869b-bcbd-4c76-af22-ef150e14981b-kube-api-access-zxr6w" (OuterVolumeSpecName: "kube-api-access-zxr6w") pod "bc5b869b-bcbd-4c76-af22-ef150e14981b" (UID: "bc5b869b-bcbd-4c76-af22-ef150e14981b"). InnerVolumeSpecName "kube-api-access-zxr6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:44:50.050915 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:50.050887    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc5b869b-bcbd-4c76-af22-ef150e14981b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bc5b869b-bcbd-4c76-af22-ef150e14981b" (UID: "bc5b869b-bcbd-4c76-af22-ef150e14981b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:44:50.061439 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:50.061418    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc5b869b-bcbd-4c76-af22-ef150e14981b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bc5b869b-bcbd-4c76-af22-ef150e14981b" (UID: "bc5b869b-bcbd-4c76-af22-ef150e14981b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:44:50.132245 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:50.132227    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc5b869b-bcbd-4c76-af22-ef150e14981b-home\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:44:50.132245 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:50.132245    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc5b869b-bcbd-4c76-af22-ef150e14981b-tmp-dir\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:44:50.132575 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:50.132255    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxr6w\" (UniqueName: \"kubernetes.io/projected/bc5b869b-bcbd-4c76-af22-ef150e14981b-kube-api-access-zxr6w\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:44:50.132575 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:50.132263    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc5b869b-bcbd-4c76-af22-ef150e14981b-tls-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:44:50.132575 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:50.132274    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc5b869b-bcbd-4c76-af22-ef150e14981b-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:44:50.132575 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:50.132283    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc5b869b-bcbd-4c76-af22-ef150e14981b-dshm\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:44:50.132575 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:50.132292    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bc5b869b-bcbd-4c76-af22-ef150e14981b-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:44:50.810866 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:50.810825    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng" event={"ID":"bc5b869b-bcbd-4c76-af22-ef150e14981b","Type":"ContainerDied","Data":"4daab57975813dd1336c803ceb8ded7c373b8b55d23668d8bfbde223d96a6f1e"}
Sep 17 22:44:50.810866 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:50.810864    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng"
Sep 17 22:44:50.811124 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:50.810900    2555 scope.go:117] "RemoveContainer" containerID="12b5e99b17f869b06ae40ab40275ee4bddf9f0f12799b99fa272702babce0aa8"
Sep 17 22:44:50.820239 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:50.820217    2555 scope.go:117] "RemoveContainer" containerID="b58b84c9880b7a9670b68154e7aa4d7427a8dcd3a81cb54472499721eaa9d162"
Sep 17 22:44:50.830009 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:50.829987    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng"]
Sep 17 22:44:50.835744 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:50.835723    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-9c56b4c96-svrng"]
Sep 17 22:44:52.678647 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:52.678613    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc5b869b-bcbd-4c76-af22-ef150e14981b" path="/var/lib/kubelet/pods/bc5b869b-bcbd-4c76-af22-ef150e14981b/volumes"
Sep 17 22:44:55.006019 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:55.005981    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:44:55.096381 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:44:55.096350    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:45:24.194552 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:24.194509    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:45:24.292907 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:24.292875    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:45:29.246109 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:29.246070    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:45:32.393430 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.393397    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5"]
Sep 17 22:45:32.393840 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.393759    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc5b869b-bcbd-4c76-af22-ef150e14981b" containerName="main"
Sep 17 22:45:32.393840 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.393769    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc5b869b-bcbd-4c76-af22-ef150e14981b" containerName="main"
Sep 17 22:45:32.393840 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.393781    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="576beb41-890e-41b2-b4c7-fba1c29a8385" containerName="storage-initializer"
Sep 17 22:45:32.393840 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.393788    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="576beb41-890e-41b2-b4c7-fba1c29a8385" containerName="storage-initializer"
Sep 17 22:45:32.393840 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.393797    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc5b869b-bcbd-4c76-af22-ef150e14981b" containerName="storage-initializer"
Sep 17 22:45:32.393840 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.393803    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc5b869b-bcbd-4c76-af22-ef150e14981b" containerName="storage-initializer"
Sep 17 22:45:32.393840 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.393817    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="576beb41-890e-41b2-b4c7-fba1c29a8385" containerName="main"
Sep 17 22:45:32.393840 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.393822    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="576beb41-890e-41b2-b4c7-fba1c29a8385" containerName="main"
Sep 17 22:45:32.394122 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.393884    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="576beb41-890e-41b2-b4c7-fba1c29a8385" containerName="main"
Sep 17 22:45:32.394122 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.393892    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc5b869b-bcbd-4c76-af22-ef150e14981b" containerName="main"
Sep 17 22:45:32.398377 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.398357    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5"
Sep 17 22:45:32.400733 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.400704    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 17 22:45:32.400865 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.400790    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-qqzh5\""
Sep 17 22:45:32.400865 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.400799    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-dmgrv\""
Sep 17 22:45:32.400865 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.400854    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:45:32.401060 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.400933    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 17 22:45:32.401060 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.400939    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 17 22:45:32.401536 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.401517    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 17 22:45:32.412709 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.412686    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5"]
Sep 17 22:45:32.469816 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.469786    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:45:32.479918 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.479887    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5\" (UID: \"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5"
Sep 17 22:45:32.480063 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.479926    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5\" (UID: \"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5"
Sep 17 22:45:32.480063 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.479972    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5\" (UID: \"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5"
Sep 17 22:45:32.480063 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.479989    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5\" (UID: \"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5"
Sep 17 22:45:32.480063 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.480008    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfgbh\" (UniqueName: \"kubernetes.io/projected/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-kube-api-access-zfgbh\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5\" (UID: \"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5"
Sep 17 22:45:32.480063 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.480058    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5\" (UID: \"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5"
Sep 17 22:45:32.480303 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.480098    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5\" (UID: \"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5"
Sep 17 22:45:32.480303 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.480131    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5\" (UID: \"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5"
Sep 17 22:45:32.581444 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.581410    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5\" (UID: \"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5"
Sep 17 22:45:32.581444 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.581444    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5\" (UID: \"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5"
Sep 17 22:45:32.581716 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.581464    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfgbh\" (UniqueName: \"kubernetes.io/projected/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-kube-api-access-zfgbh\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5\" (UID: \"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5"
Sep 17 22:45:32.581716 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.581492    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5\" (UID: \"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5"
Sep 17 22:45:32.581716 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.581527    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5\" (UID: \"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5"
Sep 17 22:45:32.581716 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.581562    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5\" (UID: \"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5"
Sep 17 22:45:32.581716 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.581605    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5\" (UID: \"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5"
Sep 17 22:45:32.581716 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.581639    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5\" (UID: \"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5"
Sep 17 22:45:32.582033 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.581967    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5\" (UID: \"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5"
Sep 17 22:45:32.582108 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.582063    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5\" (UID: \"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5"
Sep 17 22:45:32.582108 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.582092    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5\" (UID: \"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5"
Sep 17 22:45:32.582229 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.582115    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5\" (UID: \"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5"
Sep 17 22:45:32.582385 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.582368    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5\" (UID: \"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5"
Sep 17 22:45:32.583814 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.583792    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5\" (UID: \"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5"
Sep 17 22:45:32.584157 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.584138    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5\" (UID: \"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5"
Sep 17 22:45:32.589429 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.589409    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfgbh\" (UniqueName: \"kubernetes.io/projected/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-kube-api-access-zfgbh\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5\" (UID: \"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5"
Sep 17 22:45:32.663526 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.663445    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:45:32.708600 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.708573    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5"
Sep 17 22:45:32.988116 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.988079    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5"]
Sep 17 22:45:32.989215 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:45:32.989168    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d82f409_aed3_49f0_8c0f_349a5f5ddc9e.slice/crio-ece4c7126f2b51f4a17268fbb5986215d60c3cee26e518be2897719e6ca99f31 WatchSource:0}: Error finding container ece4c7126f2b51f4a17268fbb5986215d60c3cee26e518be2897719e6ca99f31: Status 404 returned error can't find the container with id ece4c7126f2b51f4a17268fbb5986215d60c3cee26e518be2897719e6ca99f31
Sep 17 22:45:32.991647 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:32.991625    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 22:45:33.043262 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:33.043231    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5" event={"ID":"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e","Type":"ContainerStarted","Data":"ece4c7126f2b51f4a17268fbb5986215d60c3cee26e518be2897719e6ca99f31"}
Sep 17 22:45:35.056784 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:35.056682    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5" event={"ID":"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e","Type":"ContainerStarted","Data":"231ee7b2ccdd26672b4037e21071aaa4876f3b1baeb33facf4e99dfdeee60d16"}
Sep 17 22:45:35.057309 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:35.056797    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5"
Sep 17 22:45:36.064215 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:36.064161    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5" event={"ID":"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e","Type":"ContainerStarted","Data":"d7d3fd5cbf5f127a1162689d5fba2b6d17ef0153ae70941791156680f30d9257"}
Sep 17 22:45:38.077912 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:38.077863    2555 generic.go:356] "Generic (PLEG): container finished" podID="7d82f409-aed3-49f0-8c0f-349a5f5ddc9e" containerID="d7d3fd5cbf5f127a1162689d5fba2b6d17ef0153ae70941791156680f30d9257" exitCode=0
Sep 17 22:45:38.078414 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:38.077930    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5" event={"ID":"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e","Type":"ContainerDied","Data":"d7d3fd5cbf5f127a1162689d5fba2b6d17ef0153ae70941791156680f30d9257"}
Sep 17 22:45:39.087205 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:39.087140    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5" event={"ID":"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e","Type":"ContainerStarted","Data":"976a555a210b9e8c08e2633d5d917dbab3b4ae1b3ec720ce7c5e1d5efdaaf517"}
Sep 17 22:45:39.111983 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:39.111927    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5" podStartSLOduration=3.163963299 podStartE2EDuration="7.11190765s" totalImagesPullingTime="1.678810911s" totalInitContainerRuntime="2.26913344s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:45:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:45:39.109059223 +0000 UTC m=+2840.555104496" watchObservedRunningTime="2026-09-17 22:45:39.11190765 +0000 UTC m=+2840.557952924"
Sep 17 22:45:42.708839 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:42.708793    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5"
Sep 17 22:45:42.709254 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:42.708869    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5"
Sep 17 22:45:42.710450 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:42.710416    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5" podUID="7d82f409-aed3-49f0-8c0f-349a5f5ddc9e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8001/health\": dial tcp 10.132.0.81:8001: connect: connection refused"
Sep 17 22:45:44.890527 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:44.890462    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:45:44.995602 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:44.995554    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:45:45.091108 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:45.091070    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:45:51.459041 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:51.459008    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:45:52.009588 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:52.009555    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:45:52.248815 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:52.248782    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:45:52.709759 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:52.709713    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5" podUID="7d82f409-aed3-49f0-8c0f-349a5f5ddc9e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8001/health\": dial tcp 10.132.0.81:8001: connect: connection refused"
Sep 17 22:45:52.728692 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:52.728668    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5"
Sep 17 22:45:56.916334 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:56.916297    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:45:57.016721 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:57.016686    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:45:58.126063 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:58.126025    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:45:58.226639 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:45:58.226607    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:46:02.709306 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:02.709235    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5" podUID="7d82f409-aed3-49f0-8c0f-349a5f5ddc9e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8001/health\": dial tcp 10.132.0.81:8001: connect: connection refused"
Sep 17 22:46:09.051142 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:09.051059    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr"]
Sep 17 22:46:09.054798 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:09.054778    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr"
Sep 17 22:46:09.057237 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:09.057212    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Sep 17 22:46:09.057361 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:09.057331    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Sep 17 22:46:09.057434 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:09.057361    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Sep 17 22:46:09.057811 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:09.057784    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Sep 17 22:46:09.057916 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:09.057788    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:46:09.057916 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:09.057880    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-z5vpl\""
Sep 17 22:46:09.072236 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:09.072204    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr"]
Sep 17 22:46:09.115135 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:09.115094    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54a37dd5-9e5c-4134-833a-84ef4d113fa7-home\") pod \"lora-single-adapter-test-kserve-79676d7fcb-xklhr\" (UID: \"54a37dd5-9e5c-4134-833a-84ef4d113fa7\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr"
Sep 17 22:46:09.115343 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:09.115146    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54a37dd5-9e5c-4134-833a-84ef4d113fa7-model-cache\") pod \"lora-single-adapter-test-kserve-79676d7fcb-xklhr\" (UID: \"54a37dd5-9e5c-4134-833a-84ef4d113fa7\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr"
Sep 17 22:46:09.115343 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:09.115197    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54a37dd5-9e5c-4134-833a-84ef4d113fa7-tmp-dir\") pod \"lora-single-adapter-test-kserve-79676d7fcb-xklhr\" (UID: \"54a37dd5-9e5c-4134-833a-84ef4d113fa7\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr"
Sep 17 22:46:09.115343 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:09.115266    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54a37dd5-9e5c-4134-833a-84ef4d113fa7-dshm\") pod \"lora-single-adapter-test-kserve-79676d7fcb-xklhr\" (UID: \"54a37dd5-9e5c-4134-833a-84ef4d113fa7\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr"
Sep 17 22:46:09.115479 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:09.115394    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54a37dd5-9e5c-4134-833a-84ef4d113fa7-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-79676d7fcb-xklhr\" (UID: \"54a37dd5-9e5c-4134-833a-84ef4d113fa7\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr"
Sep 17 22:46:09.115479 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:09.115421    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54a37dd5-9e5c-4134-833a-84ef4d113fa7-cabundle-cert\") pod \"lora-single-adapter-test-kserve-79676d7fcb-xklhr\" (UID: \"54a37dd5-9e5c-4134-833a-84ef4d113fa7\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr"
Sep 17 22:46:09.115566 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:09.115497    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54a37dd5-9e5c-4134-833a-84ef4d113fa7-tls-certs\") pod \"lora-single-adapter-test-kserve-79676d7fcb-xklhr\" (UID: \"54a37dd5-9e5c-4134-833a-84ef4d113fa7\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr"
Sep 17 22:46:09.115566 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:09.115528    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t77jj\" (UniqueName: \"kubernetes.io/projected/54a37dd5-9e5c-4134-833a-84ef4d113fa7-kube-api-access-t77jj\") pod \"lora-single-adapter-test-kserve-79676d7fcb-xklhr\" (UID: \"54a37dd5-9e5c-4134-833a-84ef4d113fa7\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr"
Sep 17 22:46:09.215988 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:09.215951    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t77jj\" (UniqueName: \"kubernetes.io/projected/54a37dd5-9e5c-4134-833a-84ef4d113fa7-kube-api-access-t77jj\") pod \"lora-single-adapter-test-kserve-79676d7fcb-xklhr\" (UID: \"54a37dd5-9e5c-4134-833a-84ef4d113fa7\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr"
Sep 17 22:46:09.216199 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:09.216016    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54a37dd5-9e5c-4134-833a-84ef4d113fa7-home\") pod \"lora-single-adapter-test-kserve-79676d7fcb-xklhr\" (UID: \"54a37dd5-9e5c-4134-833a-84ef4d113fa7\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr"
Sep 17 22:46:09.216199 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:09.216047    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54a37dd5-9e5c-4134-833a-84ef4d113fa7-model-cache\") pod \"lora-single-adapter-test-kserve-79676d7fcb-xklhr\" (UID: \"54a37dd5-9e5c-4134-833a-84ef4d113fa7\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr"
Sep 17 22:46:09.216199 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:09.216071    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54a37dd5-9e5c-4134-833a-84ef4d113fa7-tmp-dir\") pod \"lora-single-adapter-test-kserve-79676d7fcb-xklhr\" (UID: \"54a37dd5-9e5c-4134-833a-84ef4d113fa7\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr"
Sep 17 22:46:09.216199 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:09.216101    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54a37dd5-9e5c-4134-833a-84ef4d113fa7-dshm\") pod \"lora-single-adapter-test-kserve-79676d7fcb-xklhr\" (UID: \"54a37dd5-9e5c-4134-833a-84ef4d113fa7\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr"
Sep 17 22:46:09.216441 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:09.216171    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54a37dd5-9e5c-4134-833a-84ef4d113fa7-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-79676d7fcb-xklhr\" (UID: \"54a37dd5-9e5c-4134-833a-84ef4d113fa7\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr"
Sep 17 22:46:09.216441 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:09.216229    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54a37dd5-9e5c-4134-833a-84ef4d113fa7-cabundle-cert\") pod \"lora-single-adapter-test-kserve-79676d7fcb-xklhr\" (UID: \"54a37dd5-9e5c-4134-833a-84ef4d113fa7\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr"
Sep 17 22:46:09.216441 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:09.216293    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54a37dd5-9e5c-4134-833a-84ef4d113fa7-tls-certs\") pod \"lora-single-adapter-test-kserve-79676d7fcb-xklhr\" (UID: \"54a37dd5-9e5c-4134-833a-84ef4d113fa7\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr"
Sep 17 22:46:09.216600 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:09.216494    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54a37dd5-9e5c-4134-833a-84ef4d113fa7-home\") pod \"lora-single-adapter-test-kserve-79676d7fcb-xklhr\" (UID: \"54a37dd5-9e5c-4134-833a-84ef4d113fa7\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr"
Sep 17 22:46:09.216600 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:09.216539    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54a37dd5-9e5c-4134-833a-84ef4d113fa7-model-cache\") pod \"lora-single-adapter-test-kserve-79676d7fcb-xklhr\" (UID: \"54a37dd5-9e5c-4134-833a-84ef4d113fa7\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr"
Sep 17 22:46:09.217050 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:09.217025    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54a37dd5-9e5c-4134-833a-84ef4d113fa7-tmp-dir\") pod \"lora-single-adapter-test-kserve-79676d7fcb-xklhr\" (UID: \"54a37dd5-9e5c-4134-833a-84ef4d113fa7\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr"
Sep 17 22:46:09.217169 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:09.217040    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54a37dd5-9e5c-4134-833a-84ef4d113fa7-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-79676d7fcb-xklhr\" (UID: \"54a37dd5-9e5c-4134-833a-84ef4d113fa7\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr"
Sep 17 22:46:09.217265 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:09.217248    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54a37dd5-9e5c-4134-833a-84ef4d113fa7-cabundle-cert\") pod \"lora-single-adapter-test-kserve-79676d7fcb-xklhr\" (UID: \"54a37dd5-9e5c-4134-833a-84ef4d113fa7\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr"
Sep 17 22:46:09.218970 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:09.218933    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54a37dd5-9e5c-4134-833a-84ef4d113fa7-dshm\") pod \"lora-single-adapter-test-kserve-79676d7fcb-xklhr\" (UID: \"54a37dd5-9e5c-4134-833a-84ef4d113fa7\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr"
Sep 17 22:46:09.219238 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:09.219220    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54a37dd5-9e5c-4134-833a-84ef4d113fa7-tls-certs\") pod \"lora-single-adapter-test-kserve-79676d7fcb-xklhr\" (UID: \"54a37dd5-9e5c-4134-833a-84ef4d113fa7\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr"
Sep 17 22:46:09.222233 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:09.222170    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:46:09.224506 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:09.224482    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t77jj\" (UniqueName: \"kubernetes.io/projected/54a37dd5-9e5c-4134-833a-84ef4d113fa7-kube-api-access-t77jj\") pod \"lora-single-adapter-test-kserve-79676d7fcb-xklhr\" (UID: \"54a37dd5-9e5c-4134-833a-84ef4d113fa7\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr"
Sep 17 22:46:09.367751 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:09.367657    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr"
Sep 17 22:46:09.655395 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:09.655362    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr"]
Sep 17 22:46:09.656913 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:46:09.656882    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54a37dd5_9e5c_4134_833a_84ef4d113fa7.slice/crio-370d5f2aed539db6b8ca11b66e005e4518e18b4b9d59e810b87034acbd61fe5f WatchSource:0}: Error finding container 370d5f2aed539db6b8ca11b66e005e4518e18b4b9d59e810b87034acbd61fe5f: Status 404 returned error can't find the container with id 370d5f2aed539db6b8ca11b66e005e4518e18b4b9d59e810b87034acbd61fe5f
Sep 17 22:46:10.277930 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:10.277885    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr" event={"ID":"54a37dd5-9e5c-4134-833a-84ef4d113fa7","Type":"ContainerStarted","Data":"8a1aacb66e012bc2d254e59979af7ba4583f4d49b72f6720ff456e468dc6a067"}
Sep 17 22:46:10.278472 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:10.277958    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr" event={"ID":"54a37dd5-9e5c-4134-833a-84ef4d113fa7","Type":"ContainerStarted","Data":"370d5f2aed539db6b8ca11b66e005e4518e18b4b9d59e810b87034acbd61fe5f"}
Sep 17 22:46:12.709384 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:12.709334    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5" podUID="7d82f409-aed3-49f0-8c0f-349a5f5ddc9e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8001/health\": dial tcp 10.132.0.81:8001: connect: connection refused"
Sep 17 22:46:13.131381 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:13.131314    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:46:13.303478 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:13.303434    2555 generic.go:356] "Generic (PLEG): container finished" podID="54a37dd5-9e5c-4134-833a-84ef4d113fa7" containerID="8a1aacb66e012bc2d254e59979af7ba4583f4d49b72f6720ff456e468dc6a067" exitCode=0
Sep 17 22:46:13.303682 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:13.303510    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr" event={"ID":"54a37dd5-9e5c-4134-833a-84ef4d113fa7","Type":"ContainerDied","Data":"8a1aacb66e012bc2d254e59979af7ba4583f4d49b72f6720ff456e468dc6a067"}
Sep 17 22:46:14.313524 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:14.313474    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr" event={"ID":"54a37dd5-9e5c-4134-833a-84ef4d113fa7","Type":"ContainerStarted","Data":"5547355c2a092eeeb2cf94ac961d13c729ccc7da818a885baa23bed6171912e2"}
Sep 17 22:46:14.337266 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:14.337175    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr" podStartSLOduration=2.2326212 podStartE2EDuration="5.337157935s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.104536735s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:46:14.333473686 +0000 UTC m=+2875.779518959" watchObservedRunningTime="2026-09-17 22:46:14.337157935 +0000 UTC m=+2875.783203266"
Sep 17 22:46:14.368315 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:14.368273    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr"
Sep 17 22:46:19.368858 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:19.368806    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr"
Sep 17 22:46:19.370597 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:19.370564    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr" podUID="54a37dd5-9e5c-4134-833a-84ef4d113fa7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Sep 17 22:46:20.925359 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:20.925319    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:46:20.988703 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:20.988651    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:46:22.709360 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:22.709314    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5" podUID="7d82f409-aed3-49f0-8c0f-349a5f5ddc9e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8001/health\": dial tcp 10.132.0.81:8001: connect: connection refused"
Sep 17 22:46:29.369049 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:29.369007    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr" podUID="54a37dd5-9e5c-4134-833a-84ef4d113fa7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Sep 17 22:46:32.709805 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:32.709754    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5" podUID="7d82f409-aed3-49f0-8c0f-349a5f5ddc9e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8001/health\": dial tcp 10.132.0.81:8001: connect: connection refused"
Sep 17 22:46:39.368922 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:39.368871    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr" podUID="54a37dd5-9e5c-4134-833a-84ef4d113fa7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Sep 17 22:46:42.709452 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:42.709406    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5" podUID="7d82f409-aed3-49f0-8c0f-349a5f5ddc9e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8001/health\": dial tcp 10.132.0.81:8001: connect: connection refused"
Sep 17 22:46:49.368790 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:49.368741    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr" podUID="54a37dd5-9e5c-4134-833a-84ef4d113fa7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Sep 17 22:46:52.709844 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:52.709788    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5" podUID="7d82f409-aed3-49f0-8c0f-349a5f5ddc9e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8001/health\": dial tcp 10.132.0.81:8001: connect: connection refused"
Sep 17 22:46:59.369155 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:46:59.369094    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr" podUID="54a37dd5-9e5c-4134-833a-84ef4d113fa7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Sep 17 22:47:02.709735 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:47:02.709685    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5" podUID="7d82f409-aed3-49f0-8c0f-349a5f5ddc9e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8001/health\": dial tcp 10.132.0.81:8001: connect: connection refused"
Sep 17 22:47:09.368119 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:47:09.368068    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr" podUID="54a37dd5-9e5c-4134-833a-84ef4d113fa7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Sep 17 22:47:12.709719 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:47:12.709671    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5" podUID="7d82f409-aed3-49f0-8c0f-349a5f5ddc9e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8001/health\": dial tcp 10.132.0.81:8001: connect: connection refused"
Sep 17 22:47:19.368779 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:47:19.368720    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr" podUID="54a37dd5-9e5c-4134-833a-84ef4d113fa7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Sep 17 22:47:22.709575 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:47:22.709527    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5" podUID="7d82f409-aed3-49f0-8c0f-349a5f5ddc9e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8001/health\": dial tcp 10.132.0.81:8001: connect: connection refused"
Sep 17 22:47:29.378536 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:47:29.378500    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr"
Sep 17 22:47:29.386658 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:47:29.386631    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr"
Sep 17 22:47:32.709827 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:47:32.709779    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5" podUID="7d82f409-aed3-49f0-8c0f-349a5f5ddc9e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8001/health\": dial tcp 10.132.0.81:8001: connect: connection refused"
Sep 17 22:47:42.718776 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:47:42.718694    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5"
Sep 17 22:47:42.741522 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:47:42.741303    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5"
Sep 17 22:47:54.723707 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:47:54.723669    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5"]
Sep 17 22:47:54.749139 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:47:54.749108    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:47:55.620170 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:47:55.620136    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:48:00.552643 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:00.552612    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-79676d7fcb-xklhr_54a37dd5-9e5c-4134-833a-84ef4d113fa7/storage-initializer/0.log"
Sep 17 22:48:00.561505 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:00.561482    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-79676d7fcb-xklhr_54a37dd5-9e5c-4134-833a-84ef4d113fa7/main/0.log"
Sep 17 22:48:01.871956 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:01.871916    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr"]
Sep 17 22:48:01.906771 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:01.906736    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:48:09.756484 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:09.756437    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5" podUID="7d82f409-aed3-49f0-8c0f-349a5f5ddc9e" containerName="main" containerID="cri-o://976a555a210b9e8c08e2633d5d917dbab3b4ae1b3ec720ce7c5e1d5efdaaf517" gracePeriod=45
Sep 17 22:48:10.293416 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:10.293369    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5" podUID="7d82f409-aed3-49f0-8c0f-349a5f5ddc9e" containerName="llm-d-routing-sidecar" containerID="cri-o://231ee7b2ccdd26672b4037e21071aaa4876f3b1baeb33facf4e99dfdeee60d16" gracePeriod=45
Sep 17 22:48:10.549148 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:10.549124    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5"
Sep 17 22:48:10.663074 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:10.662969    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-dshm\") pod \"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e\" (UID: \"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e\") "
Sep 17 22:48:10.663272 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:10.663149    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfgbh\" (UniqueName: \"kubernetes.io/projected/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-kube-api-access-zfgbh\") pod \"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e\" (UID: \"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e\") "
Sep 17 22:48:10.663272 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:10.663209    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-tmp-dir\") pod \"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e\" (UID: \"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e\") "
Sep 17 22:48:10.663360 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:10.663327    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-home\") pod \"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e\" (UID: \"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e\") "
Sep 17 22:48:10.663402 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:10.663379    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-kserve-provision-location\") pod \"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e\" (UID: \"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e\") "
Sep 17 22:48:10.663451 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:10.663433    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-model-cache\") pod \"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e\" (UID: \"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e\") "
Sep 17 22:48:10.663504 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:10.663473    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-tls-certs\") pod \"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e\" (UID: \"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e\") "
Sep 17 22:48:10.663553 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:10.663516    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-cabundle-cert\") pod \"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e\" (UID: \"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e\") "
Sep 17 22:48:10.664010 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:10.663661    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-model-cache" (OuterVolumeSpecName: "model-cache") pod "7d82f409-aed3-49f0-8c0f-349a5f5ddc9e" (UID: "7d82f409-aed3-49f0-8c0f-349a5f5ddc9e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:48:10.664010 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:10.663987    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-model-cache\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:48:10.664222 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:10.664119    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-home" (OuterVolumeSpecName: "home") pod "7d82f409-aed3-49f0-8c0f-349a5f5ddc9e" (UID: "7d82f409-aed3-49f0-8c0f-349a5f5ddc9e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:48:10.664302 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:10.664285    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7d82f409-aed3-49f0-8c0f-349a5f5ddc9e" (UID: "7d82f409-aed3-49f0-8c0f-349a5f5ddc9e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:48:10.665540 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:10.665501    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-dshm" (OuterVolumeSpecName: "dshm") pod "7d82f409-aed3-49f0-8c0f-349a5f5ddc9e" (UID: "7d82f409-aed3-49f0-8c0f-349a5f5ddc9e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:48:10.665662 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:10.665577    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-kube-api-access-zfgbh" (OuterVolumeSpecName: "kube-api-access-zfgbh") pod "7d82f409-aed3-49f0-8c0f-349a5f5ddc9e" (UID: "7d82f409-aed3-49f0-8c0f-349a5f5ddc9e"). InnerVolumeSpecName "kube-api-access-zfgbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:48:10.665951 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:10.665937    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7d82f409-aed3-49f0-8c0f-349a5f5ddc9e" (UID: "7d82f409-aed3-49f0-8c0f-349a5f5ddc9e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:48:10.676455 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:10.676427    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7d82f409-aed3-49f0-8c0f-349a5f5ddc9e" (UID: "7d82f409-aed3-49f0-8c0f-349a5f5ddc9e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:48:10.679846 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:10.679822    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7d82f409-aed3-49f0-8c0f-349a5f5ddc9e" (UID: "7d82f409-aed3-49f0-8c0f-349a5f5ddc9e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:48:10.765070 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:10.765045    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-home\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:48:10.765070 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:10.765068    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:48:10.765433 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:10.765078    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-tls-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:48:10.765433 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:10.765087    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:48:10.765433 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:10.765094    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-dshm\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:48:10.765433 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:10.765101    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfgbh\" (UniqueName: \"kubernetes.io/projected/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-kube-api-access-zfgbh\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:48:10.765433 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:10.765109    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e-tmp-dir\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:48:11.082081 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:11.082044    2555 generic.go:356] "Generic (PLEG): container finished" podID="7d82f409-aed3-49f0-8c0f-349a5f5ddc9e" containerID="976a555a210b9e8c08e2633d5d917dbab3b4ae1b3ec720ce7c5e1d5efdaaf517" exitCode=0
Sep 17 22:48:11.082081 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:11.082075    2555 generic.go:356] "Generic (PLEG): container finished" podID="7d82f409-aed3-49f0-8c0f-349a5f5ddc9e" containerID="231ee7b2ccdd26672b4037e21071aaa4876f3b1baeb33facf4e99dfdeee60d16" exitCode=0
Sep 17 22:48:11.082317 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:11.082093    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5"
Sep 17 22:48:11.082317 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:11.082130    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5" event={"ID":"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e","Type":"ContainerDied","Data":"976a555a210b9e8c08e2633d5d917dbab3b4ae1b3ec720ce7c5e1d5efdaaf517"}
Sep 17 22:48:11.082317 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:11.082200    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5" event={"ID":"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e","Type":"ContainerDied","Data":"231ee7b2ccdd26672b4037e21071aaa4876f3b1baeb33facf4e99dfdeee60d16"}
Sep 17 22:48:11.082317 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:11.082228    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5" event={"ID":"7d82f409-aed3-49f0-8c0f-349a5f5ddc9e","Type":"ContainerDied","Data":"ece4c7126f2b51f4a17268fbb5986215d60c3cee26e518be2897719e6ca99f31"}
Sep 17 22:48:11.082317 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:11.082249    2555 scope.go:117] "RemoveContainer" containerID="976a555a210b9e8c08e2633d5d917dbab3b4ae1b3ec720ce7c5e1d5efdaaf517"
Sep 17 22:48:11.091647 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:11.091631    2555 scope.go:117] "RemoveContainer" containerID="d7d3fd5cbf5f127a1162689d5fba2b6d17ef0153ae70941791156680f30d9257"
Sep 17 22:48:11.099487 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:11.099469    2555 scope.go:117] "RemoveContainer" containerID="231ee7b2ccdd26672b4037e21071aaa4876f3b1baeb33facf4e99dfdeee60d16"
Sep 17 22:48:11.105399 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:11.105376    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5"]
Sep 17 22:48:11.108971 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:11.108946    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f955b8c96-pskn5"]
Sep 17 22:48:11.110569 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:11.110551    2555 scope.go:117] "RemoveContainer" containerID="976a555a210b9e8c08e2633d5d917dbab3b4ae1b3ec720ce7c5e1d5efdaaf517"
Sep 17 22:48:11.110833 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:48:11.110817    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"976a555a210b9e8c08e2633d5d917dbab3b4ae1b3ec720ce7c5e1d5efdaaf517\": container with ID starting with 976a555a210b9e8c08e2633d5d917dbab3b4ae1b3ec720ce7c5e1d5efdaaf517 not found: ID does not exist" containerID="976a555a210b9e8c08e2633d5d917dbab3b4ae1b3ec720ce7c5e1d5efdaaf517"
Sep 17 22:48:11.110890 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:11.110841    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"976a555a210b9e8c08e2633d5d917dbab3b4ae1b3ec720ce7c5e1d5efdaaf517"} err="failed to get container status \"976a555a210b9e8c08e2633d5d917dbab3b4ae1b3ec720ce7c5e1d5efdaaf517\": rpc error: code = NotFound desc = could not find container \"976a555a210b9e8c08e2633d5d917dbab3b4ae1b3ec720ce7c5e1d5efdaaf517\": container with ID starting with 976a555a210b9e8c08e2633d5d917dbab3b4ae1b3ec720ce7c5e1d5efdaaf517 not found: ID does not exist"
Sep 17 22:48:11.110890 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:11.110857    2555 scope.go:117] "RemoveContainer" containerID="d7d3fd5cbf5f127a1162689d5fba2b6d17ef0153ae70941791156680f30d9257"
Sep 17 22:48:11.111102 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:48:11.111083    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7d3fd5cbf5f127a1162689d5fba2b6d17ef0153ae70941791156680f30d9257\": container with ID starting with d7d3fd5cbf5f127a1162689d5fba2b6d17ef0153ae70941791156680f30d9257 not found: ID does not exist" containerID="d7d3fd5cbf5f127a1162689d5fba2b6d17ef0153ae70941791156680f30d9257"
Sep 17 22:48:11.111148 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:11.111107    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7d3fd5cbf5f127a1162689d5fba2b6d17ef0153ae70941791156680f30d9257"} err="failed to get container status \"d7d3fd5cbf5f127a1162689d5fba2b6d17ef0153ae70941791156680f30d9257\": rpc error: code = NotFound desc = could not find container \"d7d3fd5cbf5f127a1162689d5fba2b6d17ef0153ae70941791156680f30d9257\": container with ID starting with d7d3fd5cbf5f127a1162689d5fba2b6d17ef0153ae70941791156680f30d9257 not found: ID does not exist"
Sep 17 22:48:11.111148 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:11.111120    2555 scope.go:117] "RemoveContainer" containerID="231ee7b2ccdd26672b4037e21071aaa4876f3b1baeb33facf4e99dfdeee60d16"
Sep 17 22:48:11.111388 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:48:11.111369    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"231ee7b2ccdd26672b4037e21071aaa4876f3b1baeb33facf4e99dfdeee60d16\": container with ID starting with 231ee7b2ccdd26672b4037e21071aaa4876f3b1baeb33facf4e99dfdeee60d16 not found: ID does not exist" containerID="231ee7b2ccdd26672b4037e21071aaa4876f3b1baeb33facf4e99dfdeee60d16"
Sep 17 22:48:11.111429 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:11.111393    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"231ee7b2ccdd26672b4037e21071aaa4876f3b1baeb33facf4e99dfdeee60d16"} err="failed to get container status \"231ee7b2ccdd26672b4037e21071aaa4876f3b1baeb33facf4e99dfdeee60d16\": rpc error: code = NotFound desc = could not find container \"231ee7b2ccdd26672b4037e21071aaa4876f3b1baeb33facf4e99dfdeee60d16\": container with ID starting with 231ee7b2ccdd26672b4037e21071aaa4876f3b1baeb33facf4e99dfdeee60d16 not found: ID does not exist"
Sep 17 22:48:11.111429 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:11.111408    2555 scope.go:117] "RemoveContainer" containerID="976a555a210b9e8c08e2633d5d917dbab3b4ae1b3ec720ce7c5e1d5efdaaf517"
Sep 17 22:48:11.111661 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:11.111645    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"976a555a210b9e8c08e2633d5d917dbab3b4ae1b3ec720ce7c5e1d5efdaaf517"} err="failed to get container status \"976a555a210b9e8c08e2633d5d917dbab3b4ae1b3ec720ce7c5e1d5efdaaf517\": rpc error: code = NotFound desc = could not find container \"976a555a210b9e8c08e2633d5d917dbab3b4ae1b3ec720ce7c5e1d5efdaaf517\": container with ID starting with 976a555a210b9e8c08e2633d5d917dbab3b4ae1b3ec720ce7c5e1d5efdaaf517 not found: ID does not exist"
Sep 17 22:48:11.111756 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:11.111661    2555 scope.go:117] "RemoveContainer" containerID="d7d3fd5cbf5f127a1162689d5fba2b6d17ef0153ae70941791156680f30d9257"
Sep 17 22:48:11.111923 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:11.111907    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7d3fd5cbf5f127a1162689d5fba2b6d17ef0153ae70941791156680f30d9257"} err="failed to get container status \"d7d3fd5cbf5f127a1162689d5fba2b6d17ef0153ae70941791156680f30d9257\": rpc error: code = NotFound desc = could not find container \"d7d3fd5cbf5f127a1162689d5fba2b6d17ef0153ae70941791156680f30d9257\": container with ID starting with d7d3fd5cbf5f127a1162689d5fba2b6d17ef0153ae70941791156680f30d9257 not found: ID does not exist"
Sep 17 22:48:11.111923 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:11.111922    2555 scope.go:117] "RemoveContainer" containerID="231ee7b2ccdd26672b4037e21071aaa4876f3b1baeb33facf4e99dfdeee60d16"
Sep 17 22:48:11.112159 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:11.112139    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"231ee7b2ccdd26672b4037e21071aaa4876f3b1baeb33facf4e99dfdeee60d16"} err="failed to get container status \"231ee7b2ccdd26672b4037e21071aaa4876f3b1baeb33facf4e99dfdeee60d16\": rpc error: code = NotFound desc = could not find container \"231ee7b2ccdd26672b4037e21071aaa4876f3b1baeb33facf4e99dfdeee60d16\": container with ID starting with 231ee7b2ccdd26672b4037e21071aaa4876f3b1baeb33facf4e99dfdeee60d16 not found: ID does not exist"
Sep 17 22:48:12.678949 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:12.678903    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d82f409-aed3-49f0-8c0f-349a5f5ddc9e" path="/var/lib/kubelet/pods/7d82f409-aed3-49f0-8c0f-349a5f5ddc9e/volumes"
Sep 17 22:48:16.897054 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:16.897014    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr" podUID="54a37dd5-9e5c-4134-833a-84ef4d113fa7" containerName="main" containerID="cri-o://5547355c2a092eeeb2cf94ac961d13c729ccc7da818a885baa23bed6171912e2" gracePeriod=45
Sep 17 22:48:17.379521 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:17.379485    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr" podUID="54a37dd5-9e5c-4134-833a-84ef4d113fa7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Sep 17 22:48:17.554480 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:17.554458    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr"
Sep 17 22:48:17.615337 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:17.615253    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54a37dd5-9e5c-4134-833a-84ef4d113fa7-home\") pod \"54a37dd5-9e5c-4134-833a-84ef4d113fa7\" (UID: \"54a37dd5-9e5c-4134-833a-84ef4d113fa7\") "
Sep 17 22:48:17.615337 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:17.615321    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54a37dd5-9e5c-4134-833a-84ef4d113fa7-tmp-dir\") pod \"54a37dd5-9e5c-4134-833a-84ef4d113fa7\" (UID: \"54a37dd5-9e5c-4134-833a-84ef4d113fa7\") "
Sep 17 22:48:17.615543 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:17.615366    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t77jj\" (UniqueName: \"kubernetes.io/projected/54a37dd5-9e5c-4134-833a-84ef4d113fa7-kube-api-access-t77jj\") pod \"54a37dd5-9e5c-4134-833a-84ef4d113fa7\" (UID: \"54a37dd5-9e5c-4134-833a-84ef4d113fa7\") "
Sep 17 22:48:17.615543 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:17.615386    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54a37dd5-9e5c-4134-833a-84ef4d113fa7-model-cache\") pod \"54a37dd5-9e5c-4134-833a-84ef4d113fa7\" (UID: \"54a37dd5-9e5c-4134-833a-84ef4d113fa7\") "
Sep 17 22:48:17.615543 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:17.615421    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54a37dd5-9e5c-4134-833a-84ef4d113fa7-cabundle-cert\") pod \"54a37dd5-9e5c-4134-833a-84ef4d113fa7\" (UID: \"54a37dd5-9e5c-4134-833a-84ef4d113fa7\") "
Sep 17 22:48:17.615543 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:17.615449    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54a37dd5-9e5c-4134-833a-84ef4d113fa7-kserve-provision-location\") pod \"54a37dd5-9e5c-4134-833a-84ef4d113fa7\" (UID: \"54a37dd5-9e5c-4134-833a-84ef4d113fa7\") "
Sep 17 22:48:17.615543 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:17.615472    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54a37dd5-9e5c-4134-833a-84ef4d113fa7-tls-certs\") pod \"54a37dd5-9e5c-4134-833a-84ef4d113fa7\" (UID: \"54a37dd5-9e5c-4134-833a-84ef4d113fa7\") "
Sep 17 22:48:17.615543 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:17.615503    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54a37dd5-9e5c-4134-833a-84ef4d113fa7-dshm\") pod \"54a37dd5-9e5c-4134-833a-84ef4d113fa7\" (UID: \"54a37dd5-9e5c-4134-833a-84ef4d113fa7\") "
Sep 17 22:48:17.615832 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:17.615651    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54a37dd5-9e5c-4134-833a-84ef4d113fa7-model-cache" (OuterVolumeSpecName: "model-cache") pod "54a37dd5-9e5c-4134-833a-84ef4d113fa7" (UID: "54a37dd5-9e5c-4134-833a-84ef4d113fa7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:48:17.615832 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:17.615785    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54a37dd5-9e5c-4134-833a-84ef4d113fa7-model-cache\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:48:17.615951 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:17.615853    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54a37dd5-9e5c-4134-833a-84ef4d113fa7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "54a37dd5-9e5c-4134-833a-84ef4d113fa7" (UID: "54a37dd5-9e5c-4134-833a-84ef4d113fa7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:48:17.616017 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:17.615993    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54a37dd5-9e5c-4134-833a-84ef4d113fa7-home" (OuterVolumeSpecName: "home") pod "54a37dd5-9e5c-4134-833a-84ef4d113fa7" (UID: "54a37dd5-9e5c-4134-833a-84ef4d113fa7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:48:17.617792 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:17.617758    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54a37dd5-9e5c-4134-833a-84ef4d113fa7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "54a37dd5-9e5c-4134-833a-84ef4d113fa7" (UID: "54a37dd5-9e5c-4134-833a-84ef4d113fa7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:48:17.617792 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:17.617770    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54a37dd5-9e5c-4134-833a-84ef4d113fa7-kube-api-access-t77jj" (OuterVolumeSpecName: "kube-api-access-t77jj") pod "54a37dd5-9e5c-4134-833a-84ef4d113fa7" (UID: "54a37dd5-9e5c-4134-833a-84ef4d113fa7"). InnerVolumeSpecName "kube-api-access-t77jj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:48:17.617981 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:17.617935    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54a37dd5-9e5c-4134-833a-84ef4d113fa7-dshm" (OuterVolumeSpecName: "dshm") pod "54a37dd5-9e5c-4134-833a-84ef4d113fa7" (UID: "54a37dd5-9e5c-4134-833a-84ef4d113fa7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:48:17.622101 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:17.622074    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54a37dd5-9e5c-4134-833a-84ef4d113fa7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "54a37dd5-9e5c-4134-833a-84ef4d113fa7" (UID: "54a37dd5-9e5c-4134-833a-84ef4d113fa7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:48:17.628596 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:17.628569    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54a37dd5-9e5c-4134-833a-84ef4d113fa7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "54a37dd5-9e5c-4134-833a-84ef4d113fa7" (UID: "54a37dd5-9e5c-4134-833a-84ef4d113fa7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:48:17.716306 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:17.716283    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54a37dd5-9e5c-4134-833a-84ef4d113fa7-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:48:17.716306 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:17.716304    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54a37dd5-9e5c-4134-833a-84ef4d113fa7-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:48:17.716447 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:17.716313    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54a37dd5-9e5c-4134-833a-84ef4d113fa7-tls-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:48:17.716447 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:17.716321    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54a37dd5-9e5c-4134-833a-84ef4d113fa7-dshm\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:48:17.716447 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:17.716328    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54a37dd5-9e5c-4134-833a-84ef4d113fa7-home\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:48:17.716447 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:17.716336    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54a37dd5-9e5c-4134-833a-84ef4d113fa7-tmp-dir\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:48:17.716447 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:17.716343    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t77jj\" (UniqueName: \"kubernetes.io/projected/54a37dd5-9e5c-4134-833a-84ef4d113fa7-kube-api-access-t77jj\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:48:18.121304 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:18.121269    2555 generic.go:356] "Generic (PLEG): container finished" podID="54a37dd5-9e5c-4134-833a-84ef4d113fa7" containerID="5547355c2a092eeeb2cf94ac961d13c729ccc7da818a885baa23bed6171912e2" exitCode=0
Sep 17 22:48:18.121800 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:18.121323    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr"
Sep 17 22:48:18.121800 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:18.121365    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr" event={"ID":"54a37dd5-9e5c-4134-833a-84ef4d113fa7","Type":"ContainerDied","Data":"5547355c2a092eeeb2cf94ac961d13c729ccc7da818a885baa23bed6171912e2"}
Sep 17 22:48:18.121800 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:18.121438    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr" event={"ID":"54a37dd5-9e5c-4134-833a-84ef4d113fa7","Type":"ContainerDied","Data":"370d5f2aed539db6b8ca11b66e005e4518e18b4b9d59e810b87034acbd61fe5f"}
Sep 17 22:48:18.121800 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:18.121478    2555 scope.go:117] "RemoveContainer" containerID="5547355c2a092eeeb2cf94ac961d13c729ccc7da818a885baa23bed6171912e2"
Sep 17 22:48:18.131568 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:18.131551    2555 scope.go:117] "RemoveContainer" containerID="8a1aacb66e012bc2d254e59979af7ba4583f4d49b72f6720ff456e468dc6a067"
Sep 17 22:48:18.140711 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:18.140691    2555 scope.go:117] "RemoveContainer" containerID="5547355c2a092eeeb2cf94ac961d13c729ccc7da818a885baa23bed6171912e2"
Sep 17 22:48:18.141028 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:48:18.141007    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5547355c2a092eeeb2cf94ac961d13c729ccc7da818a885baa23bed6171912e2\": container with ID starting with 5547355c2a092eeeb2cf94ac961d13c729ccc7da818a885baa23bed6171912e2 not found: ID does not exist" containerID="5547355c2a092eeeb2cf94ac961d13c729ccc7da818a885baa23bed6171912e2"
Sep 17 22:48:18.141094 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:18.141035    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5547355c2a092eeeb2cf94ac961d13c729ccc7da818a885baa23bed6171912e2"} err="failed to get container status \"5547355c2a092eeeb2cf94ac961d13c729ccc7da818a885baa23bed6171912e2\": rpc error: code = NotFound desc = could not find container \"5547355c2a092eeeb2cf94ac961d13c729ccc7da818a885baa23bed6171912e2\": container with ID starting with 5547355c2a092eeeb2cf94ac961d13c729ccc7da818a885baa23bed6171912e2 not found: ID does not exist"
Sep 17 22:48:18.141094 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:18.141051    2555 scope.go:117] "RemoveContainer" containerID="8a1aacb66e012bc2d254e59979af7ba4583f4d49b72f6720ff456e468dc6a067"
Sep 17 22:48:18.141382 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:48:18.141340    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a1aacb66e012bc2d254e59979af7ba4583f4d49b72f6720ff456e468dc6a067\": container with ID starting with 8a1aacb66e012bc2d254e59979af7ba4583f4d49b72f6720ff456e468dc6a067 not found: ID does not exist" containerID="8a1aacb66e012bc2d254e59979af7ba4583f4d49b72f6720ff456e468dc6a067"
Sep 17 22:48:18.141382 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:18.141372    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a1aacb66e012bc2d254e59979af7ba4583f4d49b72f6720ff456e468dc6a067"} err="failed to get container status \"8a1aacb66e012bc2d254e59979af7ba4583f4d49b72f6720ff456e468dc6a067\": rpc error: code = NotFound desc = could not find container \"8a1aacb66e012bc2d254e59979af7ba4583f4d49b72f6720ff456e468dc6a067\": container with ID starting with 8a1aacb66e012bc2d254e59979af7ba4583f4d49b72f6720ff456e468dc6a067 not found: ID does not exist"
Sep 17 22:48:18.144396 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:18.144375    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr"]
Sep 17 22:48:18.147663 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:18.147639    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-79676d7fcb-xklhr"]
Sep 17 22:48:18.686524 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:18.686481    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54a37dd5-9e5c-4134-833a-84ef4d113fa7" path="/var/lib/kubelet/pods/54a37dd5-9e5c-4134-833a-84ef4d113fa7/volumes"
Sep 17 22:48:18.994699 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:18.994612    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:48:19.096648 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:19.096607    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:48:22.057781 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:22.057747    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:48:22.158108 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:22.158073    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:48:22.259647 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:22.259618    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:48:22.909042 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:22.909016    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-7njvj_a324c9ab-5d42-4d31-8ff7-449f619be774/console-operator/1.log"
Sep 17 22:48:22.912928 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:22.912907    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s7f5_4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf/ovn-acl-logging/0.log"
Sep 17 22:48:22.922759 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:22.922741    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-7njvj_a324c9ab-5d42-4d31-8ff7-449f619be774/console-operator/1.log"
Sep 17 22:48:22.926288 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:22.926270    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s7f5_4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf/ovn-acl-logging/0.log"
Sep 17 22:48:27.686903 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:27.686870    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:48:27.746752 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:27.746726    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:48:37.252587 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.252548    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66"]
Sep 17 22:48:37.253174 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.253152    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d82f409-aed3-49f0-8c0f-349a5f5ddc9e" containerName="main"
Sep 17 22:48:37.253174 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.253195    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d82f409-aed3-49f0-8c0f-349a5f5ddc9e" containerName="main"
Sep 17 22:48:37.253309 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.253213    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54a37dd5-9e5c-4134-833a-84ef4d113fa7" containerName="main"
Sep 17 22:48:37.253309 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.253222    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="54a37dd5-9e5c-4134-833a-84ef4d113fa7" containerName="main"
Sep 17 22:48:37.253309 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.253236    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d82f409-aed3-49f0-8c0f-349a5f5ddc9e" containerName="llm-d-routing-sidecar"
Sep 17 22:48:37.253309 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.253246    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d82f409-aed3-49f0-8c0f-349a5f5ddc9e" containerName="llm-d-routing-sidecar"
Sep 17 22:48:37.253309 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.253259    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d82f409-aed3-49f0-8c0f-349a5f5ddc9e" containerName="storage-initializer"
Sep 17 22:48:37.253309 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.253267    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d82f409-aed3-49f0-8c0f-349a5f5ddc9e" containerName="storage-initializer"
Sep 17 22:48:37.253309 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.253298    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54a37dd5-9e5c-4134-833a-84ef4d113fa7" containerName="storage-initializer"
Sep 17 22:48:37.253309 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.253306    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="54a37dd5-9e5c-4134-833a-84ef4d113fa7" containerName="storage-initializer"
Sep 17 22:48:37.253561 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.253414    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="54a37dd5-9e5c-4134-833a-84ef4d113fa7" containerName="main"
Sep 17 22:48:37.253561 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.253428    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d82f409-aed3-49f0-8c0f-349a5f5ddc9e" containerName="llm-d-routing-sidecar"
Sep 17 22:48:37.253561 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.253438    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d82f409-aed3-49f0-8c0f-349a5f5ddc9e" containerName="main"
Sep 17 22:48:37.257442 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.257418    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66"
Sep 17 22:48:37.259691 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.259649    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:48:37.259827 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.259693    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Sep 17 22:48:37.259827 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.259782    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Sep 17 22:48:37.259941 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.259864    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Sep 17 22:48:37.259980 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.259964    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-wsn85\""
Sep 17 22:48:37.260026 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.259977    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Sep 17 22:48:37.265761 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.265614    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66"]
Sep 17 22:48:37.272031 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.272009    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:48:37.382087 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.382054    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/417f3a88-d036-49ab-a7df-3a4e1bc62e13-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-568bfb787f-dzt66\" (UID: \"417f3a88-d036-49ab-a7df-3a4e1bc62e13\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66"
Sep 17 22:48:37.382087 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.382094    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/417f3a88-d036-49ab-a7df-3a4e1bc62e13-model-cache\") pod \"lora-multiple-adapters-test-kserve-568bfb787f-dzt66\" (UID: \"417f3a88-d036-49ab-a7df-3a4e1bc62e13\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66"
Sep 17 22:48:37.382319 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.382111    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/417f3a88-d036-49ab-a7df-3a4e1bc62e13-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-568bfb787f-dzt66\" (UID: \"417f3a88-d036-49ab-a7df-3a4e1bc62e13\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66"
Sep 17 22:48:37.382319 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.382162    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/417f3a88-d036-49ab-a7df-3a4e1bc62e13-dshm\") pod \"lora-multiple-adapters-test-kserve-568bfb787f-dzt66\" (UID: \"417f3a88-d036-49ab-a7df-3a4e1bc62e13\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66"
Sep 17 22:48:37.382319 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.382220    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/417f3a88-d036-49ab-a7df-3a4e1bc62e13-tls-certs\") pod \"lora-multiple-adapters-test-kserve-568bfb787f-dzt66\" (UID: \"417f3a88-d036-49ab-a7df-3a4e1bc62e13\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66"
Sep 17 22:48:37.382319 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.382250    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/417f3a88-d036-49ab-a7df-3a4e1bc62e13-home\") pod \"lora-multiple-adapters-test-kserve-568bfb787f-dzt66\" (UID: \"417f3a88-d036-49ab-a7df-3a4e1bc62e13\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66"
Sep 17 22:48:37.382319 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.382285    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/417f3a88-d036-49ab-a7df-3a4e1bc62e13-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-568bfb787f-dzt66\" (UID: \"417f3a88-d036-49ab-a7df-3a4e1bc62e13\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66"
Sep 17 22:48:37.382484 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.382331    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrx8s\" (UniqueName: \"kubernetes.io/projected/417f3a88-d036-49ab-a7df-3a4e1bc62e13-kube-api-access-nrx8s\") pod \"lora-multiple-adapters-test-kserve-568bfb787f-dzt66\" (UID: \"417f3a88-d036-49ab-a7df-3a4e1bc62e13\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66"
Sep 17 22:48:37.483454 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.483411    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/417f3a88-d036-49ab-a7df-3a4e1bc62e13-tls-certs\") pod \"lora-multiple-adapters-test-kserve-568bfb787f-dzt66\" (UID: \"417f3a88-d036-49ab-a7df-3a4e1bc62e13\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66"
Sep 17 22:48:37.483454 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.483459    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/417f3a88-d036-49ab-a7df-3a4e1bc62e13-home\") pod \"lora-multiple-adapters-test-kserve-568bfb787f-dzt66\" (UID: \"417f3a88-d036-49ab-a7df-3a4e1bc62e13\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66"
Sep 17 22:48:37.483721 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.483507    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/417f3a88-d036-49ab-a7df-3a4e1bc62e13-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-568bfb787f-dzt66\" (UID: \"417f3a88-d036-49ab-a7df-3a4e1bc62e13\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66"
Sep 17 22:48:37.483721 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.483666    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrx8s\" (UniqueName: \"kubernetes.io/projected/417f3a88-d036-49ab-a7df-3a4e1bc62e13-kube-api-access-nrx8s\") pod \"lora-multiple-adapters-test-kserve-568bfb787f-dzt66\" (UID: \"417f3a88-d036-49ab-a7df-3a4e1bc62e13\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66"
Sep 17 22:48:37.483836 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.483764    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/417f3a88-d036-49ab-a7df-3a4e1bc62e13-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-568bfb787f-dzt66\" (UID: \"417f3a88-d036-49ab-a7df-3a4e1bc62e13\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66"
Sep 17 22:48:37.483836 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.483798    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/417f3a88-d036-49ab-a7df-3a4e1bc62e13-model-cache\") pod \"lora-multiple-adapters-test-kserve-568bfb787f-dzt66\" (UID: \"417f3a88-d036-49ab-a7df-3a4e1bc62e13\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66"
Sep 17 22:48:37.483836 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.483814    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/417f3a88-d036-49ab-a7df-3a4e1bc62e13-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-568bfb787f-dzt66\" (UID: \"417f3a88-d036-49ab-a7df-3a4e1bc62e13\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66"
Sep 17 22:48:37.483998 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.483843    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/417f3a88-d036-49ab-a7df-3a4e1bc62e13-dshm\") pod \"lora-multiple-adapters-test-kserve-568bfb787f-dzt66\" (UID: \"417f3a88-d036-49ab-a7df-3a4e1bc62e13\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66"
Sep 17 22:48:37.484056 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.483996    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/417f3a88-d036-49ab-a7df-3a4e1bc62e13-home\") pod \"lora-multiple-adapters-test-kserve-568bfb787f-dzt66\" (UID: \"417f3a88-d036-49ab-a7df-3a4e1bc62e13\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66"
Sep 17 22:48:37.484304 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.484115    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/417f3a88-d036-49ab-a7df-3a4e1bc62e13-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-568bfb787f-dzt66\" (UID: \"417f3a88-d036-49ab-a7df-3a4e1bc62e13\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66"
Sep 17 22:48:37.484304 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.484171    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/417f3a88-d036-49ab-a7df-3a4e1bc62e13-model-cache\") pod \"lora-multiple-adapters-test-kserve-568bfb787f-dzt66\" (UID: \"417f3a88-d036-49ab-a7df-3a4e1bc62e13\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66"
Sep 17 22:48:37.484473 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.484301    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/417f3a88-d036-49ab-a7df-3a4e1bc62e13-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-568bfb787f-dzt66\" (UID: \"417f3a88-d036-49ab-a7df-3a4e1bc62e13\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66"
Sep 17 22:48:37.484473 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.484337    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/417f3a88-d036-49ab-a7df-3a4e1bc62e13-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-568bfb787f-dzt66\" (UID: \"417f3a88-d036-49ab-a7df-3a4e1bc62e13\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66"
Sep 17 22:48:37.486130 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.486102    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/417f3a88-d036-49ab-a7df-3a4e1bc62e13-dshm\") pod \"lora-multiple-adapters-test-kserve-568bfb787f-dzt66\" (UID: \"417f3a88-d036-49ab-a7df-3a4e1bc62e13\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66"
Sep 17 22:48:37.486269 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.486249    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/417f3a88-d036-49ab-a7df-3a4e1bc62e13-tls-certs\") pod \"lora-multiple-adapters-test-kserve-568bfb787f-dzt66\" (UID: \"417f3a88-d036-49ab-a7df-3a4e1bc62e13\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66"
Sep 17 22:48:37.491607 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.491585    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrx8s\" (UniqueName: \"kubernetes.io/projected/417f3a88-d036-49ab-a7df-3a4e1bc62e13-kube-api-access-nrx8s\") pod \"lora-multiple-adapters-test-kserve-568bfb787f-dzt66\" (UID: \"417f3a88-d036-49ab-a7df-3a4e1bc62e13\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66"
Sep 17 22:48:37.569368 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.569271    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66"
Sep 17 22:48:37.857108 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:37.857074    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66"]
Sep 17 22:48:37.859275 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:48:37.859245    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod417f3a88_d036_49ab_a7df_3a4e1bc62e13.slice/crio-54639215bb1e3097eb96db91b84ba36ae30862ef937c3a8bb61905b3e44b077b WatchSource:0}: Error finding container 54639215bb1e3097eb96db91b84ba36ae30862ef937c3a8bb61905b3e44b077b: Status 404 returned error can't find the container with id 54639215bb1e3097eb96db91b84ba36ae30862ef937c3a8bb61905b3e44b077b
Sep 17 22:48:38.236090 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:38.236054    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66" event={"ID":"417f3a88-d036-49ab-a7df-3a4e1bc62e13","Type":"ContainerStarted","Data":"44bc1406f15ad3f8612d38e821f10c6bcf3cd80236fbb5235d87f79f53058a87"}
Sep 17 22:48:38.236340 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:38.236106    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66" event={"ID":"417f3a88-d036-49ab-a7df-3a4e1bc62e13","Type":"ContainerStarted","Data":"54639215bb1e3097eb96db91b84ba36ae30862ef937c3a8bb61905b3e44b077b"}
Sep 17 22:48:38.490089 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:38.489993    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:48:41.264010 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:41.263919    2555 generic.go:356] "Generic (PLEG): container finished" podID="417f3a88-d036-49ab-a7df-3a4e1bc62e13" containerID="44bc1406f15ad3f8612d38e821f10c6bcf3cd80236fbb5235d87f79f53058a87" exitCode=0
Sep 17 22:48:41.264411 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:41.263997    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66" event={"ID":"417f3a88-d036-49ab-a7df-3a4e1bc62e13","Type":"ContainerDied","Data":"44bc1406f15ad3f8612d38e821f10c6bcf3cd80236fbb5235d87f79f53058a87"}
Sep 17 22:48:42.272765 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:42.272725    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66" event={"ID":"417f3a88-d036-49ab-a7df-3a4e1bc62e13","Type":"ContainerStarted","Data":"ca5918e3fd1e9683a65ff75ba170baf08a206b152b6bd3d9353bbe339924ecdf"}
Sep 17 22:48:42.295721 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:42.295668    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66" podStartSLOduration=2.288509163 podStartE2EDuration="5.295653181s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.007144018s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:48:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:48:42.294691433 +0000 UTC m=+3023.740736706" watchObservedRunningTime="2026-09-17 22:48:42.295653181 +0000 UTC m=+3023.741698453"
Sep 17 22:48:42.569463 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:42.569382    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66"
Sep 17 22:48:44.660012 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:44.659980    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:48:44.757766 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:44.757725    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:48:47.570171 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:47.570120    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66"
Sep 17 22:48:47.572248 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:47.572216    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66" podUID="417f3a88-d036-49ab-a7df-3a4e1bc62e13" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Sep 17 22:48:50.657624 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:50.657591    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:48:56.335423 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:56.335389    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:48:56.405828 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:56.405791    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:48:57.570480 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:57.570435    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66" podUID="417f3a88-d036-49ab-a7df-3a4e1bc62e13" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Sep 17 22:48:57.596109 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:57.596079    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl"]
Sep 17 22:48:57.599883 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:57.599858    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl"
Sep 17 22:48:57.602477 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:57.602420    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 17 22:48:57.602599 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:57.602562    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 17 22:48:57.602705 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:57.602668    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 17 22:48:57.602915 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:57.602891    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-6j9zp\""
Sep 17 22:48:57.603009 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:57.602941    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:48:57.603082 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:57.603008    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 17 22:48:57.614336 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:57.614309    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl"]
Sep 17 22:48:57.633562 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:57.633534    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:48:57.782481 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:57.782440    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3d5f074-e341-4b9d-bb24-94ea2def1539-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl\" (UID: \"a3d5f074-e341-4b9d-bb24-94ea2def1539\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl"
Sep 17 22:48:57.782481 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:57.782474    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sh4w\" (UniqueName: \"kubernetes.io/projected/a3d5f074-e341-4b9d-bb24-94ea2def1539-kube-api-access-4sh4w\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl\" (UID: \"a3d5f074-e341-4b9d-bb24-94ea2def1539\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl"
Sep 17 22:48:57.782755 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:57.782496    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3d5f074-e341-4b9d-bb24-94ea2def1539-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl\" (UID: \"a3d5f074-e341-4b9d-bb24-94ea2def1539\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl"
Sep 17 22:48:57.782755 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:57.782514    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3d5f074-e341-4b9d-bb24-94ea2def1539-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl\" (UID: \"a3d5f074-e341-4b9d-bb24-94ea2def1539\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl"
Sep 17 22:48:57.782755 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:57.782673    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3d5f074-e341-4b9d-bb24-94ea2def1539-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl\" (UID: \"a3d5f074-e341-4b9d-bb24-94ea2def1539\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl"
Sep 17 22:48:57.782755 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:57.782742    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3d5f074-e341-4b9d-bb24-94ea2def1539-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl\" (UID: \"a3d5f074-e341-4b9d-bb24-94ea2def1539\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl"
Sep 17 22:48:57.782979 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:57.782769    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3d5f074-e341-4b9d-bb24-94ea2def1539-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl\" (UID: \"a3d5f074-e341-4b9d-bb24-94ea2def1539\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl"
Sep 17 22:48:57.782979 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:57.782907    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3d5f074-e341-4b9d-bb24-94ea2def1539-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl\" (UID: \"a3d5f074-e341-4b9d-bb24-94ea2def1539\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl"
Sep 17 22:48:57.837839 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:57.837752    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:48:57.883946 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:57.883911    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3d5f074-e341-4b9d-bb24-94ea2def1539-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl\" (UID: \"a3d5f074-e341-4b9d-bb24-94ea2def1539\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl"
Sep 17 22:48:57.884144 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:57.883961    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3d5f074-e341-4b9d-bb24-94ea2def1539-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl\" (UID: \"a3d5f074-e341-4b9d-bb24-94ea2def1539\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl"
Sep 17 22:48:57.884144 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:57.883989    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3d5f074-e341-4b9d-bb24-94ea2def1539-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl\" (UID: \"a3d5f074-e341-4b9d-bb24-94ea2def1539\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl"
Sep 17 22:48:57.884144 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:57.884039    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3d5f074-e341-4b9d-bb24-94ea2def1539-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl\" (UID: \"a3d5f074-e341-4b9d-bb24-94ea2def1539\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl"
Sep 17 22:48:57.884144 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:57.884113    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3d5f074-e341-4b9d-bb24-94ea2def1539-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl\" (UID: \"a3d5f074-e341-4b9d-bb24-94ea2def1539\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl"
Sep 17 22:48:57.884144 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:57.884140    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sh4w\" (UniqueName: \"kubernetes.io/projected/a3d5f074-e341-4b9d-bb24-94ea2def1539-kube-api-access-4sh4w\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl\" (UID: \"a3d5f074-e341-4b9d-bb24-94ea2def1539\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl"
Sep 17 22:48:57.884467 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:57.884168    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3d5f074-e341-4b9d-bb24-94ea2def1539-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl\" (UID: \"a3d5f074-e341-4b9d-bb24-94ea2def1539\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl"
Sep 17 22:48:57.884467 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:57.884224    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3d5f074-e341-4b9d-bb24-94ea2def1539-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl\" (UID: \"a3d5f074-e341-4b9d-bb24-94ea2def1539\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl"
Sep 17 22:48:57.884467 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:57.884427    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3d5f074-e341-4b9d-bb24-94ea2def1539-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl\" (UID: \"a3d5f074-e341-4b9d-bb24-94ea2def1539\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl"
Sep 17 22:48:57.884630 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:57.884517    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3d5f074-e341-4b9d-bb24-94ea2def1539-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl\" (UID: \"a3d5f074-e341-4b9d-bb24-94ea2def1539\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl"
Sep 17 22:48:57.884731 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:57.884704    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3d5f074-e341-4b9d-bb24-94ea2def1539-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl\" (UID: \"a3d5f074-e341-4b9d-bb24-94ea2def1539\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl"
Sep 17 22:48:57.884829 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:57.884806    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3d5f074-e341-4b9d-bb24-94ea2def1539-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl\" (UID: \"a3d5f074-e341-4b9d-bb24-94ea2def1539\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl"
Sep 17 22:48:57.884916 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:57.884893    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3d5f074-e341-4b9d-bb24-94ea2def1539-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl\" (UID: \"a3d5f074-e341-4b9d-bb24-94ea2def1539\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl"
Sep 17 22:48:57.886701 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:57.886674    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3d5f074-e341-4b9d-bb24-94ea2def1539-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl\" (UID: \"a3d5f074-e341-4b9d-bb24-94ea2def1539\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl"
Sep 17 22:48:57.886809 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:57.886773    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3d5f074-e341-4b9d-bb24-94ea2def1539-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl\" (UID: \"a3d5f074-e341-4b9d-bb24-94ea2def1539\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl"
Sep 17 22:48:57.892449 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:57.892421    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sh4w\" (UniqueName: \"kubernetes.io/projected/a3d5f074-e341-4b9d-bb24-94ea2def1539-kube-api-access-4sh4w\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl\" (UID: \"a3d5f074-e341-4b9d-bb24-94ea2def1539\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl"
Sep 17 22:48:57.913104 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:57.913079    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl"
Sep 17 22:48:58.197734 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:58.197706    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl"]
Sep 17 22:48:58.201014 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:48:58.200981    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3d5f074_e341_4b9d_bb24_94ea2def1539.slice/crio-7989654e66c7cd3ed64776208a2b57056560395419916b0cf77620d6c48e3ef4 WatchSource:0}: Error finding container 7989654e66c7cd3ed64776208a2b57056560395419916b0cf77620d6c48e3ef4: Status 404 returned error can't find the container with id 7989654e66c7cd3ed64776208a2b57056560395419916b0cf77620d6c48e3ef4
Sep 17 22:48:58.368491 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:58.368455    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl" event={"ID":"a3d5f074-e341-4b9d-bb24-94ea2def1539","Type":"ContainerStarted","Data":"8b2516e52eb1dae7af0fce3470acaa0e9ff4678cf4905390879092ac69716881"}
Sep 17 22:48:58.368687 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:48:58.368509    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl" event={"ID":"a3d5f074-e341-4b9d-bb24-94ea2def1539","Type":"ContainerStarted","Data":"7989654e66c7cd3ed64776208a2b57056560395419916b0cf77620d6c48e3ef4"}
Sep 17 22:49:01.396358 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:49:01.396310    2555 generic.go:356] "Generic (PLEG): container finished" podID="a3d5f074-e341-4b9d-bb24-94ea2def1539" containerID="8b2516e52eb1dae7af0fce3470acaa0e9ff4678cf4905390879092ac69716881" exitCode=0
Sep 17 22:49:01.396915 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:49:01.396381    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl" event={"ID":"a3d5f074-e341-4b9d-bb24-94ea2def1539","Type":"ContainerDied","Data":"8b2516e52eb1dae7af0fce3470acaa0e9ff4678cf4905390879092ac69716881"}
Sep 17 22:49:02.404137 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:49:02.404097    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl" event={"ID":"a3d5f074-e341-4b9d-bb24-94ea2def1539","Type":"ContainerStarted","Data":"8329ce19dfed9be5b6157b8c7b1f6ac13ed4282c4df5b33cbfd3370fc3896ca1"}
Sep 17 22:49:02.426453 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:49:02.426373    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl" podStartSLOduration=3.051079034 podStartE2EDuration="5.426355782s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.375276748s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:48:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:49:02.425232481 +0000 UTC m=+3043.871277754" watchObservedRunningTime="2026-09-17 22:49:02.426355782 +0000 UTC m=+3043.872401054"
Sep 17 22:49:07.570211 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:49:07.570094    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66" podUID="417f3a88-d036-49ab-a7df-3a4e1bc62e13" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Sep 17 22:49:07.913463 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:49:07.913424    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl"
Sep 17 22:49:07.913683 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:49:07.913496    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl"
Sep 17 22:49:07.915374 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:49:07.915336    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl" podUID="a3d5f074-e341-4b9d-bb24-94ea2def1539" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Sep 17 22:49:13.603512 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:49:13.603470    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:49:15.815840 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:49:15.815803    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:49:15.916243 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:49:15.916203    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:49:17.570331 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:49:17.570288    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66" podUID="417f3a88-d036-49ab-a7df-3a4e1bc62e13" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Sep 17 22:49:17.914204 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:49:17.914153    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl" podUID="a3d5f074-e341-4b9d-bb24-94ea2def1539" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Sep 17 22:49:27.569886 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:49:27.569820    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66" podUID="417f3a88-d036-49ab-a7df-3a4e1bc62e13" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Sep 17 22:49:27.914064 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:49:27.914014    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl" podUID="a3d5f074-e341-4b9d-bb24-94ea2def1539" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Sep 17 22:49:37.570782 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:49:37.570731    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66" podUID="417f3a88-d036-49ab-a7df-3a4e1bc62e13" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Sep 17 22:49:37.914498 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:49:37.914442    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl" podUID="a3d5f074-e341-4b9d-bb24-94ea2def1539" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Sep 17 22:49:47.570585 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:49:47.570539    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66" podUID="417f3a88-d036-49ab-a7df-3a4e1bc62e13" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Sep 17 22:49:47.913753 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:49:47.913703    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl" podUID="a3d5f074-e341-4b9d-bb24-94ea2def1539" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Sep 17 22:49:57.580478 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:49:57.580433    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66"
Sep 17 22:49:57.589001 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:49:57.588959    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66"
Sep 17 22:49:57.914308 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:49:57.914259    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl" podUID="a3d5f074-e341-4b9d-bb24-94ea2def1539" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Sep 17 22:50:07.913712 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:07.913654    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl" podUID="a3d5f074-e341-4b9d-bb24-94ea2def1539" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Sep 17 22:50:17.914150 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:17.914101    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl" podUID="a3d5f074-e341-4b9d-bb24-94ea2def1539" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Sep 17 22:50:27.914538 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:27.914490    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl" podUID="a3d5f074-e341-4b9d-bb24-94ea2def1539" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Sep 17 22:50:34.779461 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:34.779381    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-568bfb787f-dzt66_417f3a88-d036-49ab-a7df-3a4e1bc62e13/storage-initializer/0.log"
Sep 17 22:50:34.790053 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:34.790030    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-568bfb787f-dzt66_417f3a88-d036-49ab-a7df-3a4e1bc62e13/main/0.log"
Sep 17 22:50:35.866371 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:35.866331    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66"]
Sep 17 22:50:35.888580 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:35.888541    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:50:37.914506 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:37.914465    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl" podUID="a3d5f074-e341-4b9d-bb24-94ea2def1539" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Sep 17 22:50:47.913819 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:47.913767    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl" podUID="a3d5f074-e341-4b9d-bb24-94ea2def1539" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Sep 17 22:50:50.890912 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:50.890859    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66" podUID="417f3a88-d036-49ab-a7df-3a4e1bc62e13" containerName="main" containerID="cri-o://ca5918e3fd1e9683a65ff75ba170baf08a206b152b6bd3d9353bbe339924ecdf" gracePeriod=45
Sep 17 22:50:51.581073 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:51.581023    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66" podUID="417f3a88-d036-49ab-a7df-3a4e1bc62e13" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Sep 17 22:50:51.848982 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:51.848956    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66"
Sep 17 22:50:51.954149 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:51.954109    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/417f3a88-d036-49ab-a7df-3a4e1bc62e13-dshm\") pod \"417f3a88-d036-49ab-a7df-3a4e1bc62e13\" (UID: \"417f3a88-d036-49ab-a7df-3a4e1bc62e13\") "
Sep 17 22:50:51.954579 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:51.954217    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/417f3a88-d036-49ab-a7df-3a4e1bc62e13-cabundle-cert\") pod \"417f3a88-d036-49ab-a7df-3a4e1bc62e13\" (UID: \"417f3a88-d036-49ab-a7df-3a4e1bc62e13\") "
Sep 17 22:50:51.954579 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:51.954274    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrx8s\" (UniqueName: \"kubernetes.io/projected/417f3a88-d036-49ab-a7df-3a4e1bc62e13-kube-api-access-nrx8s\") pod \"417f3a88-d036-49ab-a7df-3a4e1bc62e13\" (UID: \"417f3a88-d036-49ab-a7df-3a4e1bc62e13\") "
Sep 17 22:50:51.954579 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:51.954442    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/417f3a88-d036-49ab-a7df-3a4e1bc62e13-home\") pod \"417f3a88-d036-49ab-a7df-3a4e1bc62e13\" (UID: \"417f3a88-d036-49ab-a7df-3a4e1bc62e13\") "
Sep 17 22:50:51.954579 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:51.954513    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/417f3a88-d036-49ab-a7df-3a4e1bc62e13-model-cache\") pod \"417f3a88-d036-49ab-a7df-3a4e1bc62e13\" (UID: \"417f3a88-d036-49ab-a7df-3a4e1bc62e13\") "
Sep 17 22:50:51.954579 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:51.954565    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/417f3a88-d036-49ab-a7df-3a4e1bc62e13-tmp-dir\") pod \"417f3a88-d036-49ab-a7df-3a4e1bc62e13\" (UID: \"417f3a88-d036-49ab-a7df-3a4e1bc62e13\") "
Sep 17 22:50:51.954870 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:51.954595    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/417f3a88-d036-49ab-a7df-3a4e1bc62e13-kserve-provision-location\") pod \"417f3a88-d036-49ab-a7df-3a4e1bc62e13\" (UID: \"417f3a88-d036-49ab-a7df-3a4e1bc62e13\") "
Sep 17 22:50:51.954870 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:51.954598    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/417f3a88-d036-49ab-a7df-3a4e1bc62e13-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "417f3a88-d036-49ab-a7df-3a4e1bc62e13" (UID: "417f3a88-d036-49ab-a7df-3a4e1bc62e13"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:50:51.954870 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:51.954619    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/417f3a88-d036-49ab-a7df-3a4e1bc62e13-tls-certs\") pod \"417f3a88-d036-49ab-a7df-3a4e1bc62e13\" (UID: \"417f3a88-d036-49ab-a7df-3a4e1bc62e13\") "
Sep 17 22:50:51.955021 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:51.954921    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/417f3a88-d036-49ab-a7df-3a4e1bc62e13-home" (OuterVolumeSpecName: "home") pod "417f3a88-d036-49ab-a7df-3a4e1bc62e13" (UID: "417f3a88-d036-49ab-a7df-3a4e1bc62e13"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:50:51.955021 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:51.954937    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/417f3a88-d036-49ab-a7df-3a4e1bc62e13-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:50:51.955499 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:51.955469    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/417f3a88-d036-49ab-a7df-3a4e1bc62e13-model-cache" (OuterVolumeSpecName: "model-cache") pod "417f3a88-d036-49ab-a7df-3a4e1bc62e13" (UID: "417f3a88-d036-49ab-a7df-3a4e1bc62e13"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:50:51.956984 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:51.956961    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/417f3a88-d036-49ab-a7df-3a4e1bc62e13-dshm" (OuterVolumeSpecName: "dshm") pod "417f3a88-d036-49ab-a7df-3a4e1bc62e13" (UID: "417f3a88-d036-49ab-a7df-3a4e1bc62e13"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:50:51.957149 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:51.957041    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/417f3a88-d036-49ab-a7df-3a4e1bc62e13-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "417f3a88-d036-49ab-a7df-3a4e1bc62e13" (UID: "417f3a88-d036-49ab-a7df-3a4e1bc62e13"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:50:51.957149 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:51.957065    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/417f3a88-d036-49ab-a7df-3a4e1bc62e13-kube-api-access-nrx8s" (OuterVolumeSpecName: "kube-api-access-nrx8s") pod "417f3a88-d036-49ab-a7df-3a4e1bc62e13" (UID: "417f3a88-d036-49ab-a7df-3a4e1bc62e13"). InnerVolumeSpecName "kube-api-access-nrx8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:50:51.968936 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:51.968906    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/417f3a88-d036-49ab-a7df-3a4e1bc62e13-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "417f3a88-d036-49ab-a7df-3a4e1bc62e13" (UID: "417f3a88-d036-49ab-a7df-3a4e1bc62e13"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:50:51.980786 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:51.980754    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/417f3a88-d036-49ab-a7df-3a4e1bc62e13-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "417f3a88-d036-49ab-a7df-3a4e1bc62e13" (UID: "417f3a88-d036-49ab-a7df-3a4e1bc62e13"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:50:52.056264 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:52.056225    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrx8s\" (UniqueName: \"kubernetes.io/projected/417f3a88-d036-49ab-a7df-3a4e1bc62e13-kube-api-access-nrx8s\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:50:52.056264 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:52.056259    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/417f3a88-d036-49ab-a7df-3a4e1bc62e13-home\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:50:52.056489 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:52.056274    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/417f3a88-d036-49ab-a7df-3a4e1bc62e13-model-cache\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:50:52.056489 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:52.056312    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/417f3a88-d036-49ab-a7df-3a4e1bc62e13-tmp-dir\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:50:52.056489 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:52.056324    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/417f3a88-d036-49ab-a7df-3a4e1bc62e13-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:50:52.056489 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:52.056335    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/417f3a88-d036-49ab-a7df-3a4e1bc62e13-tls-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:50:52.056489 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:52.056346    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/417f3a88-d036-49ab-a7df-3a4e1bc62e13-dshm\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:50:52.118084 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:52.117995    2555 generic.go:356] "Generic (PLEG): container finished" podID="417f3a88-d036-49ab-a7df-3a4e1bc62e13" containerID="ca5918e3fd1e9683a65ff75ba170baf08a206b152b6bd3d9353bbe339924ecdf" exitCode=0
Sep 17 22:50:52.118084 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:52.118040    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66"
Sep 17 22:50:52.118084 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:52.118077    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66" event={"ID":"417f3a88-d036-49ab-a7df-3a4e1bc62e13","Type":"ContainerDied","Data":"ca5918e3fd1e9683a65ff75ba170baf08a206b152b6bd3d9353bbe339924ecdf"}
Sep 17 22:50:52.118355 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:52.118133    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66" event={"ID":"417f3a88-d036-49ab-a7df-3a4e1bc62e13","Type":"ContainerDied","Data":"54639215bb1e3097eb96db91b84ba36ae30862ef937c3a8bb61905b3e44b077b"}
Sep 17 22:50:52.118355 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:52.118163    2555 scope.go:117] "RemoveContainer" containerID="ca5918e3fd1e9683a65ff75ba170baf08a206b152b6bd3d9353bbe339924ecdf"
Sep 17 22:50:52.128557 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:52.128535    2555 scope.go:117] "RemoveContainer" containerID="44bc1406f15ad3f8612d38e821f10c6bcf3cd80236fbb5235d87f79f53058a87"
Sep 17 22:50:52.137605 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:52.137583    2555 scope.go:117] "RemoveContainer" containerID="ca5918e3fd1e9683a65ff75ba170baf08a206b152b6bd3d9353bbe339924ecdf"
Sep 17 22:50:52.137894 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:50:52.137871    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca5918e3fd1e9683a65ff75ba170baf08a206b152b6bd3d9353bbe339924ecdf\": container with ID starting with ca5918e3fd1e9683a65ff75ba170baf08a206b152b6bd3d9353bbe339924ecdf not found: ID does not exist" containerID="ca5918e3fd1e9683a65ff75ba170baf08a206b152b6bd3d9353bbe339924ecdf"
Sep 17 22:50:52.137997 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:52.137901    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca5918e3fd1e9683a65ff75ba170baf08a206b152b6bd3d9353bbe339924ecdf"} err="failed to get container status \"ca5918e3fd1e9683a65ff75ba170baf08a206b152b6bd3d9353bbe339924ecdf\": rpc error: code = NotFound desc = could not find container \"ca5918e3fd1e9683a65ff75ba170baf08a206b152b6bd3d9353bbe339924ecdf\": container with ID starting with ca5918e3fd1e9683a65ff75ba170baf08a206b152b6bd3d9353bbe339924ecdf not found: ID does not exist"
Sep 17 22:50:52.137997 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:52.137918    2555 scope.go:117] "RemoveContainer" containerID="44bc1406f15ad3f8612d38e821f10c6bcf3cd80236fbb5235d87f79f53058a87"
Sep 17 22:50:52.138161 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:50:52.138141    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44bc1406f15ad3f8612d38e821f10c6bcf3cd80236fbb5235d87f79f53058a87\": container with ID starting with 44bc1406f15ad3f8612d38e821f10c6bcf3cd80236fbb5235d87f79f53058a87 not found: ID does not exist" containerID="44bc1406f15ad3f8612d38e821f10c6bcf3cd80236fbb5235d87f79f53058a87"
Sep 17 22:50:52.138236 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:52.138168    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44bc1406f15ad3f8612d38e821f10c6bcf3cd80236fbb5235d87f79f53058a87"} err="failed to get container status \"44bc1406f15ad3f8612d38e821f10c6bcf3cd80236fbb5235d87f79f53058a87\": rpc error: code = NotFound desc = could not find container \"44bc1406f15ad3f8612d38e821f10c6bcf3cd80236fbb5235d87f79f53058a87\": container with ID starting with 44bc1406f15ad3f8612d38e821f10c6bcf3cd80236fbb5235d87f79f53058a87 not found: ID does not exist"
Sep 17 22:50:52.143430 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:52.143404    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66"]
Sep 17 22:50:52.145921 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:52.145897    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-568bfb787f-dzt66"]
Sep 17 22:50:52.687638 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:52.687600    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="417f3a88-d036-49ab-a7df-3a4e1bc62e13" path="/var/lib/kubelet/pods/417f3a88-d036-49ab-a7df-3a4e1bc62e13/volumes"
Sep 17 22:50:56.076352 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:56.076321    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:50:56.178975 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:56.178940    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:50:56.280133 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:56.280105    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:50:56.359531 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:56.359447    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:50:56.424538 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:56.424507    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:50:57.914507 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:57.914466    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl" podUID="a3d5f074-e341-4b9d-bb24-94ea2def1539" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Sep 17 22:50:58.756115 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:58.756085    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:50:58.807712 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:58.807675    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:50:59.355021 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:59.354981    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:50:59.370846 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:59.370813    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xq2zq"]
Sep 17 22:50:59.371245 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:59.371230    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="417f3a88-d036-49ab-a7df-3a4e1bc62e13" containerName="main"
Sep 17 22:50:59.371291 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:59.371247    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="417f3a88-d036-49ab-a7df-3a4e1bc62e13" containerName="main"
Sep 17 22:50:59.371291 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:59.371275    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="417f3a88-d036-49ab-a7df-3a4e1bc62e13" containerName="storage-initializer"
Sep 17 22:50:59.371291 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:59.371282    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="417f3a88-d036-49ab-a7df-3a4e1bc62e13" containerName="storage-initializer"
Sep 17 22:50:59.371390 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:59.371347    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="417f3a88-d036-49ab-a7df-3a4e1bc62e13" containerName="main"
Sep 17 22:50:59.375678 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:59.375662    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xq2zq"
Sep 17 22:50:59.378424 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:59.378401    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Sep 17 22:50:59.378656 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:59.378639    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-8btxq\""
Sep 17 22:50:59.378857 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:59.378838    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Sep 17 22:50:59.378955 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:59.378850    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Sep 17 22:50:59.381144 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:59.381119    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xq2zq"]
Sep 17 22:50:59.456037 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:59.455998    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:50:59.526302 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:59.526262    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dcfn\" (UniqueName: \"kubernetes.io/projected/dce86c89-c1dc-4513-bf88-5bd913ebc676-kube-api-access-7dcfn\") pod \"e2e-pvc-lora-download-xq2zq\" (UID: \"dce86c89-c1dc-4513-bf88-5bd913ebc676\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xq2zq"
Sep 17 22:50:59.627523 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:59.627493    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dcfn\" (UniqueName: \"kubernetes.io/projected/dce86c89-c1dc-4513-bf88-5bd913ebc676-kube-api-access-7dcfn\") pod \"e2e-pvc-lora-download-xq2zq\" (UID: \"dce86c89-c1dc-4513-bf88-5bd913ebc676\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xq2zq"
Sep 17 22:50:59.635403 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:50:59.635376    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dcfn\" (UniqueName: \"kubernetes.io/projected/dce86c89-c1dc-4513-bf88-5bd913ebc676-kube-api-access-7dcfn\") pod \"e2e-pvc-lora-download-xq2zq\" (UID: \"dce86c89-c1dc-4513-bf88-5bd913ebc676\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xq2zq"
Sep 17 22:51:01.993293 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:01.993252    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:51:02.064957 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:02.064921    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:51:07.095883 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:07.095850    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-2f66f8a0-2e27-4acc-a637-98baaa34f261\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04e013e9e3c8a1904\") pod \"e2e-pvc-lora-download-xq2zq\" (UID: \"dce86c89-c1dc-4513-bf88-5bd913ebc676\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xq2zq"
Sep 17 22:51:07.103444 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:07.103416    2555 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-2f66f8a0-2e27-4acc-a637-98baaa34f261\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04e013e9e3c8a1904\") pod \"e2e-pvc-lora-download-xq2zq\" (UID: \"dce86c89-c1dc-4513-bf88-5bd913ebc676\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xq2zq"
Sep 17 22:51:07.197100 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:07.197070    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-2f66f8a0-2e27-4acc-a637-98baaa34f261\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04e013e9e3c8a1904\") pod \"e2e-pvc-lora-download-xq2zq\" (UID: \"dce86c89-c1dc-4513-bf88-5bd913ebc676\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xq2zq"
Sep 17 22:51:07.197281 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:07.197254    2555 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-2f66f8a0-2e27-4acc-a637-98baaa34f261\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04e013e9e3c8a1904\") pod \"e2e-pvc-lora-download-xq2zq\" (UID: \"dce86c89-c1dc-4513-bf88-5bd913ebc676\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xq2zq"
Sep 17 22:51:07.200221 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:07.200196    2555 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-2f66f8a0-2e27-4acc-a637-98baaa34f261\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04e013e9e3c8a1904\") pod \"e2e-pvc-lora-download-xq2zq\" (UID: \"dce86c89-c1dc-4513-bf88-5bd913ebc676\") DevicePath \"csi-b9f2b492fff5e22748471b807bad60723efa4c55583e5cca0f3921f3b4c8bbe0\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xq2zq"
Sep 17 22:51:07.431102 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:07.431075    2555 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-2f66f8a0-2e27-4acc-a637-98baaa34f261\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04e013e9e3c8a1904\") pod \"e2e-pvc-lora-download-xq2zq\" (UID: \"dce86c89-c1dc-4513-bf88-5bd913ebc676\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/f9e3370dd684f2b54ddf503398436e032311411398b9258837a9a87f0eb52f1b/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xq2zq"
Sep 17 22:51:07.445936 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:07.445903    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-2f66f8a0-2e27-4acc-a637-98baaa34f261\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04e013e9e3c8a1904\") pod \"e2e-pvc-lora-download-xq2zq\" (UID: \"dce86c89-c1dc-4513-bf88-5bd913ebc676\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xq2zq"
Sep 17 22:51:07.486546 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:07.486517    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xq2zq"
Sep 17 22:51:07.759247 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:07.759156    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xq2zq"]
Sep 17 22:51:07.762453 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:51:07.762424    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddce86c89_c1dc_4513_bf88_5bd913ebc676.slice/crio-39909c13c7fdc4cfb3343028fb769af76f9926d9f6d14cfbb3c85769d73cf900 WatchSource:0}: Error finding container 39909c13c7fdc4cfb3343028fb769af76f9926d9f6d14cfbb3c85769d73cf900: Status 404 returned error can't find the container with id 39909c13c7fdc4cfb3343028fb769af76f9926d9f6d14cfbb3c85769d73cf900
Sep 17 22:51:07.923527 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:07.923494    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl"
Sep 17 22:51:07.931495 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:07.931468    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl"
Sep 17 22:51:08.223692 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:08.223645    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xq2zq" event={"ID":"dce86c89-c1dc-4513-bf88-5bd913ebc676","Type":"ContainerStarted","Data":"e74dd28bafd87b8b038f3a5a5643413b24f0981c16e03a3a2c3b489ecf06ec98"}
Sep 17 22:51:08.224222 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:08.223707    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xq2zq" event={"ID":"dce86c89-c1dc-4513-bf88-5bd913ebc676","Type":"ContainerStarted","Data":"39909c13c7fdc4cfb3343028fb769af76f9926d9f6d14cfbb3c85769d73cf900"}
Sep 17 22:51:08.239668 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:08.239614    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xq2zq" podStartSLOduration=12.239596986 podStartE2EDuration="12.239596986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 22:50:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:51:08.23900271 +0000 UTC m=+3169.685047983" watchObservedRunningTime="2026-09-17 22:51:08.239596986 +0000 UTC m=+3169.685642258"
Sep 17 22:51:10.235739 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:10.235707    2555 generic.go:356] "Generic (PLEG): container finished" podID="dce86c89-c1dc-4513-bf88-5bd913ebc676" containerID="e74dd28bafd87b8b038f3a5a5643413b24f0981c16e03a3a2c3b489ecf06ec98" exitCode=0
Sep 17 22:51:10.236139 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:10.235782    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xq2zq" event={"ID":"dce86c89-c1dc-4513-bf88-5bd913ebc676","Type":"ContainerDied","Data":"e74dd28bafd87b8b038f3a5a5643413b24f0981c16e03a3a2c3b489ecf06ec98"}
Sep 17 22:51:11.380343 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:11.380318    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xq2zq"
Sep 17 22:51:11.536348 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:11.536254    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04e013e9e3c8a1904\") pod \"dce86c89-c1dc-4513-bf88-5bd913ebc676\" (UID: \"dce86c89-c1dc-4513-bf88-5bd913ebc676\") "
Sep 17 22:51:11.536510 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:11.536362    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dcfn\" (UniqueName: \"kubernetes.io/projected/dce86c89-c1dc-4513-bf88-5bd913ebc676-kube-api-access-7dcfn\") pod \"dce86c89-c1dc-4513-bf88-5bd913ebc676\" (UID: \"dce86c89-c1dc-4513-bf88-5bd913ebc676\") "
Sep 17 22:51:11.538818 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:11.538775    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce86c89-c1dc-4513-bf88-5bd913ebc676-kube-api-access-7dcfn" (OuterVolumeSpecName: "kube-api-access-7dcfn") pod "dce86c89-c1dc-4513-bf88-5bd913ebc676" (UID: "dce86c89-c1dc-4513-bf88-5bd913ebc676"). InnerVolumeSpecName "kube-api-access-7dcfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:51:11.540389 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:11.540364    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-04e013e9e3c8a1904" (OuterVolumeSpecName: "model-storage") pod "dce86c89-c1dc-4513-bf88-5bd913ebc676" (UID: "dce86c89-c1dc-4513-bf88-5bd913ebc676"). InnerVolumeSpecName "pvc-2f66f8a0-2e27-4acc-a637-98baaa34f261". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 17 22:51:11.637760 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:11.637731    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dcfn\" (UniqueName: \"kubernetes.io/projected/dce86c89-c1dc-4513-bf88-5bd913ebc676-kube-api-access-7dcfn\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:51:11.637760 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:11.637774    2555 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-2f66f8a0-2e27-4acc-a637-98baaa34f261\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04e013e9e3c8a1904\") on node \"ip-10-0-140-157.ec2.internal\" "
Sep 17 22:51:12.249616 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:12.249576    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xq2zq" event={"ID":"dce86c89-c1dc-4513-bf88-5bd913ebc676","Type":"ContainerDied","Data":"39909c13c7fdc4cfb3343028fb769af76f9926d9f6d14cfbb3c85769d73cf900"}
Sep 17 22:51:12.249616 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:12.249597    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xq2zq"
Sep 17 22:51:12.249616 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:12.249619    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39909c13c7fdc4cfb3343028fb769af76f9926d9f6d14cfbb3c85769d73cf900"
Sep 17 22:51:12.611629 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:12.611603    2555 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-2f66f8a0-2e27-4acc-a637-98baaa34f261" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-04e013e9e3c8a1904") on node "ip-10-0-140-157.ec2.internal"
Sep 17 22:51:12.648569 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:12.648536    2555 reconciler_common.go:299] "Volume detached for volume \"pvc-2f66f8a0-2e27-4acc-a637-98baaa34f261\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04e013e9e3c8a1904\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"csi-b9f2b492fff5e22748471b807bad60723efa4c55583e5cca0f3921f3b4c8bbe0\""
Sep 17 22:51:22.556332 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:22.556290    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"]
Sep 17 22:51:22.557159 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:22.557133    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dce86c89-c1dc-4513-bf88-5bd913ebc676" containerName="storage-initializer"
Sep 17 22:51:22.557159 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:22.557161    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="dce86c89-c1dc-4513-bf88-5bd913ebc676" containerName="storage-initializer"
Sep 17 22:51:22.557391 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:22.557375    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="dce86c89-c1dc-4513-bf88-5bd913ebc676" containerName="storage-initializer"
Sep 17 22:51:22.563886 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:22.563864    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"
Sep 17 22:51:22.566333 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:22.566307    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Sep 17 22:51:22.566333 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:22.566326    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:51:22.566566 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:22.566326    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Sep 17 22:51:22.566566 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:22.566405    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Sep 17 22:51:22.566566 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:22.566326    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"lora-pvc-single-adapter-test-kserve-self-signed-certs\""
Sep 17 22:51:22.566911 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:22.566896    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-8btxq\""
Sep 17 22:51:22.571125 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:22.571100    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"]
Sep 17 22:51:22.580230 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:22.580204    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:51:22.733736 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:22.733704    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9e40a60-0a35-47b9-bf21-4762112ec07e-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p\" (UID: \"f9e40a60-0a35-47b9-bf21-4762112ec07e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"
Sep 17 22:51:22.733918 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:22.733740    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9e40a60-0a35-47b9-bf21-4762112ec07e-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p\" (UID: \"f9e40a60-0a35-47b9-bf21-4762112ec07e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"
Sep 17 22:51:22.733918 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:22.733765    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9e40a60-0a35-47b9-bf21-4762112ec07e-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p\" (UID: \"f9e40a60-0a35-47b9-bf21-4762112ec07e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"
Sep 17 22:51:22.733918 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:22.733843    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9e40a60-0a35-47b9-bf21-4762112ec07e-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p\" (UID: \"f9e40a60-0a35-47b9-bf21-4762112ec07e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"
Sep 17 22:51:22.733918 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:22.733873    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-662ss\" (UniqueName: \"kubernetes.io/projected/f9e40a60-0a35-47b9-bf21-4762112ec07e-kube-api-access-662ss\") pod \"lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p\" (UID: \"f9e40a60-0a35-47b9-bf21-4762112ec07e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"
Sep 17 22:51:22.734067 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:22.733922    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9e40a60-0a35-47b9-bf21-4762112ec07e-home\") pod \"lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p\" (UID: \"f9e40a60-0a35-47b9-bf21-4762112ec07e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"
Sep 17 22:51:22.734067 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:22.733950    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9e40a60-0a35-47b9-bf21-4762112ec07e-dshm\") pod \"lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p\" (UID: \"f9e40a60-0a35-47b9-bf21-4762112ec07e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"
Sep 17 22:51:22.734067 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:22.733990    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9e40a60-0a35-47b9-bf21-4762112ec07e-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p\" (UID: \"f9e40a60-0a35-47b9-bf21-4762112ec07e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"
Sep 17 22:51:22.834808 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:22.834722    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9e40a60-0a35-47b9-bf21-4762112ec07e-home\") pod \"lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p\" (UID: \"f9e40a60-0a35-47b9-bf21-4762112ec07e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"
Sep 17 22:51:22.834808 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:22.834755    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9e40a60-0a35-47b9-bf21-4762112ec07e-dshm\") pod \"lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p\" (UID: \"f9e40a60-0a35-47b9-bf21-4762112ec07e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"
Sep 17 22:51:22.835059 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:22.834966    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9e40a60-0a35-47b9-bf21-4762112ec07e-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p\" (UID: \"f9e40a60-0a35-47b9-bf21-4762112ec07e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"
Sep 17 22:51:22.835131 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:22.835060    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9e40a60-0a35-47b9-bf21-4762112ec07e-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p\" (UID: \"f9e40a60-0a35-47b9-bf21-4762112ec07e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"
Sep 17 22:51:22.835131 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:22.835089    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9e40a60-0a35-47b9-bf21-4762112ec07e-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p\" (UID: \"f9e40a60-0a35-47b9-bf21-4762112ec07e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"
Sep 17 22:51:22.835131 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:22.835124    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9e40a60-0a35-47b9-bf21-4762112ec07e-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p\" (UID: \"f9e40a60-0a35-47b9-bf21-4762112ec07e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"
Sep 17 22:51:22.835332 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:22.835156    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9e40a60-0a35-47b9-bf21-4762112ec07e-home\") pod \"lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p\" (UID: \"f9e40a60-0a35-47b9-bf21-4762112ec07e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"
Sep 17 22:51:22.835332 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:22.835249    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9e40a60-0a35-47b9-bf21-4762112ec07e-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p\" (UID: \"f9e40a60-0a35-47b9-bf21-4762112ec07e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"
Sep 17 22:51:22.835332 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:22.835289    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-662ss\" (UniqueName: \"kubernetes.io/projected/f9e40a60-0a35-47b9-bf21-4762112ec07e-kube-api-access-662ss\") pod \"lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p\" (UID: \"f9e40a60-0a35-47b9-bf21-4762112ec07e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"
Sep 17 22:51:22.835496 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:22.835437    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9e40a60-0a35-47b9-bf21-4762112ec07e-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p\" (UID: \"f9e40a60-0a35-47b9-bf21-4762112ec07e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"
Sep 17 22:51:22.835552 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:22.835539    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9e40a60-0a35-47b9-bf21-4762112ec07e-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p\" (UID: \"f9e40a60-0a35-47b9-bf21-4762112ec07e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"
Sep 17 22:51:22.835674 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:22.835651    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9e40a60-0a35-47b9-bf21-4762112ec07e-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p\" (UID: \"f9e40a60-0a35-47b9-bf21-4762112ec07e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"
Sep 17 22:51:22.835758 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:22.835709    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9e40a60-0a35-47b9-bf21-4762112ec07e-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p\" (UID: \"f9e40a60-0a35-47b9-bf21-4762112ec07e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"
Sep 17 22:51:22.837254 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:22.837232    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9e40a60-0a35-47b9-bf21-4762112ec07e-dshm\") pod \"lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p\" (UID: \"f9e40a60-0a35-47b9-bf21-4762112ec07e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"
Sep 17 22:51:22.837708 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:22.837689    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9e40a60-0a35-47b9-bf21-4762112ec07e-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p\" (UID: \"f9e40a60-0a35-47b9-bf21-4762112ec07e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"
Sep 17 22:51:22.842818 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:22.842792    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-662ss\" (UniqueName: \"kubernetes.io/projected/f9e40a60-0a35-47b9-bf21-4762112ec07e-kube-api-access-662ss\") pod \"lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p\" (UID: \"f9e40a60-0a35-47b9-bf21-4762112ec07e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"
Sep 17 22:51:27.579155 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:27.579118    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-2f66f8a0-2e27-4acc-a637-98baaa34f261\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04e013e9e3c8a1904\") pod \"lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p\" (UID: \"f9e40a60-0a35-47b9-bf21-4762112ec07e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"
Sep 17 22:51:27.584441 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:27.584412    2555 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-2f66f8a0-2e27-4acc-a637-98baaa34f261\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04e013e9e3c8a1904\") pod \"lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p\" (UID: \"f9e40a60-0a35-47b9-bf21-4762112ec07e\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"
Sep 17 22:51:27.680237 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:27.680208    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-2f66f8a0-2e27-4acc-a637-98baaa34f261\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04e013e9e3c8a1904\") pod \"lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p\" (UID: \"f9e40a60-0a35-47b9-bf21-4762112ec07e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"
Sep 17 22:51:27.680391 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:27.680340    2555 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-2f66f8a0-2e27-4acc-a637-98baaa34f261\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04e013e9e3c8a1904\") pod \"lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p\" (UID: \"f9e40a60-0a35-47b9-bf21-4762112ec07e\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"
Sep 17 22:51:27.683209 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:27.683189    2555 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-2f66f8a0-2e27-4acc-a637-98baaa34f261\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04e013e9e3c8a1904\") pod \"lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p\" (UID: \"f9e40a60-0a35-47b9-bf21-4762112ec07e\") DevicePath \"csi-b9f2b492fff5e22748471b807bad60723efa4c55583e5cca0f3921f3b4c8bbe0\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"
Sep 17 22:51:27.774054 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:27.774015    2555 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-2f66f8a0-2e27-4acc-a637-98baaa34f261\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04e013e9e3c8a1904\") pod \"lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p\" (UID: \"f9e40a60-0a35-47b9-bf21-4762112ec07e\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/f9e3370dd684f2b54ddf503398436e032311411398b9258837a9a87f0eb52f1b/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"
Sep 17 22:51:27.794504 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:27.794479    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-2f66f8a0-2e27-4acc-a637-98baaa34f261\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04e013e9e3c8a1904\") pod \"lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p\" (UID: \"f9e40a60-0a35-47b9-bf21-4762112ec07e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"
Sep 17 22:51:27.977738 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:27.977701    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"
Sep 17 22:51:28.262012 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:51:28.261977    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9e40a60_0a35_47b9_bf21_4762112ec07e.slice/crio-255ec586f7b58064cff6eff6ef97760579b95240dec281a0cc7b7f59fe0a248e WatchSource:0}: Error finding container 255ec586f7b58064cff6eff6ef97760579b95240dec281a0cc7b7f59fe0a248e: Status 404 returned error can't find the container with id 255ec586f7b58064cff6eff6ef97760579b95240dec281a0cc7b7f59fe0a248e
Sep 17 22:51:28.262142 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:28.262122    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"]
Sep 17 22:51:28.338822 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:28.338789    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p" event={"ID":"f9e40a60-0a35-47b9-bf21-4762112ec07e","Type":"ContainerStarted","Data":"d0fac35bb0d1e5471f70009434a10f547785e4bdc0d0ccdfdce2e74972c684e6"}
Sep 17 22:51:28.338934 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:28.338849    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p" event={"ID":"f9e40a60-0a35-47b9-bf21-4762112ec07e","Type":"ContainerStarted","Data":"255ec586f7b58064cff6eff6ef97760579b95240dec281a0cc7b7f59fe0a248e"}
Sep 17 22:51:31.357165 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:31.357129    2555 generic.go:356] "Generic (PLEG): container finished" podID="f9e40a60-0a35-47b9-bf21-4762112ec07e" containerID="d0fac35bb0d1e5471f70009434a10f547785e4bdc0d0ccdfdce2e74972c684e6" exitCode=0
Sep 17 22:51:31.357165 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:31.357173    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p" event={"ID":"f9e40a60-0a35-47b9-bf21-4762112ec07e","Type":"ContainerDied","Data":"d0fac35bb0d1e5471f70009434a10f547785e4bdc0d0ccdfdce2e74972c684e6"}
Sep 17 22:51:32.365289 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:32.365243    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p" event={"ID":"f9e40a60-0a35-47b9-bf21-4762112ec07e","Type":"ContainerStarted","Data":"72053bbd7cd2708ced81a1f812cc59513f8c8ca8db26c6af3c5ac59b274f3046"}
Sep 17 22:51:32.387154 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:32.387097    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p" podStartSLOduration=8.099270871 podStartE2EDuration="10.387079816s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.287808945s" isStatefulPod=true podCreationTimestamp="2026-09-17 22:51:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:51:32.385756518 +0000 UTC m=+3193.831801815" watchObservedRunningTime="2026-09-17 22:51:32.387079816 +0000 UTC m=+3193.833125088"
Sep 17 22:51:32.977956 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:32.977896    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"
Sep 17 22:51:37.776636 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:37.776604    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl"]
Sep 17 22:51:37.789943 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:37.789911    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:51:37.854610 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:37.854569    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:51:37.978081 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:37.978040    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"
Sep 17 22:51:37.980136 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:37.980100    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p" podUID="f9e40a60-0a35-47b9-bf21-4762112ec07e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": dial tcp 10.132.0.86:8000: connect: connection refused"
Sep 17 22:51:47.979016 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:47.978951    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p" podUID="f9e40a60-0a35-47b9-bf21-4762112ec07e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": dial tcp 10.132.0.86:8000: connect: connection refused"
Sep 17 22:51:52.810879 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:52.810836    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl" podUID="a3d5f074-e341-4b9d-bb24-94ea2def1539" containerName="main" containerID="cri-o://8329ce19dfed9be5b6157b8c7b1f6ac13ed4282c4df5b33cbfd3370fc3896ca1" gracePeriod=45
Sep 17 22:51:53.501713 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:53.501672    2555 generic.go:356] "Generic (PLEG): container finished" podID="a3d5f074-e341-4b9d-bb24-94ea2def1539" containerID="8329ce19dfed9be5b6157b8c7b1f6ac13ed4282c4df5b33cbfd3370fc3896ca1" exitCode=0
Sep 17 22:51:53.501906 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:53.501715    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl" event={"ID":"a3d5f074-e341-4b9d-bb24-94ea2def1539","Type":"ContainerDied","Data":"8329ce19dfed9be5b6157b8c7b1f6ac13ed4282c4df5b33cbfd3370fc3896ca1"}
Sep 17 22:51:53.571731 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:53.571703    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl"
Sep 17 22:51:53.613361 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:53.613292    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3d5f074-e341-4b9d-bb24-94ea2def1539-cabundle-cert\") pod \"a3d5f074-e341-4b9d-bb24-94ea2def1539\" (UID: \"a3d5f074-e341-4b9d-bb24-94ea2def1539\") "
Sep 17 22:51:53.613361 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:53.613335    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3d5f074-e341-4b9d-bb24-94ea2def1539-tls-certs\") pod \"a3d5f074-e341-4b9d-bb24-94ea2def1539\" (UID: \"a3d5f074-e341-4b9d-bb24-94ea2def1539\") "
Sep 17 22:51:53.613570 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:53.613363    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3d5f074-e341-4b9d-bb24-94ea2def1539-kserve-provision-location\") pod \"a3d5f074-e341-4b9d-bb24-94ea2def1539\" (UID: \"a3d5f074-e341-4b9d-bb24-94ea2def1539\") "
Sep 17 22:51:53.613570 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:53.613399    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3d5f074-e341-4b9d-bb24-94ea2def1539-home\") pod \"a3d5f074-e341-4b9d-bb24-94ea2def1539\" (UID: \"a3d5f074-e341-4b9d-bb24-94ea2def1539\") "
Sep 17 22:51:53.613570 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:53.613472    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4sh4w\" (UniqueName: \"kubernetes.io/projected/a3d5f074-e341-4b9d-bb24-94ea2def1539-kube-api-access-4sh4w\") pod \"a3d5f074-e341-4b9d-bb24-94ea2def1539\" (UID: \"a3d5f074-e341-4b9d-bb24-94ea2def1539\") "
Sep 17 22:51:53.613570 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:53.613505    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3d5f074-e341-4b9d-bb24-94ea2def1539-tmp-dir\") pod \"a3d5f074-e341-4b9d-bb24-94ea2def1539\" (UID: \"a3d5f074-e341-4b9d-bb24-94ea2def1539\") "
Sep 17 22:51:53.613796 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:53.613577    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3d5f074-e341-4b9d-bb24-94ea2def1539-dshm\") pod \"a3d5f074-e341-4b9d-bb24-94ea2def1539\" (UID: \"a3d5f074-e341-4b9d-bb24-94ea2def1539\") "
Sep 17 22:51:53.613796 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:53.613601    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3d5f074-e341-4b9d-bb24-94ea2def1539-model-cache\") pod \"a3d5f074-e341-4b9d-bb24-94ea2def1539\" (UID: \"a3d5f074-e341-4b9d-bb24-94ea2def1539\") "
Sep 17 22:51:53.613796 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:53.613713    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3d5f074-e341-4b9d-bb24-94ea2def1539-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a3d5f074-e341-4b9d-bb24-94ea2def1539" (UID: "a3d5f074-e341-4b9d-bb24-94ea2def1539"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:51:53.613952 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:53.613915    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3d5f074-e341-4b9d-bb24-94ea2def1539-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:51:53.614798 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:53.614759    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d5f074-e341-4b9d-bb24-94ea2def1539-home" (OuterVolumeSpecName: "home") pod "a3d5f074-e341-4b9d-bb24-94ea2def1539" (UID: "a3d5f074-e341-4b9d-bb24-94ea2def1539"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:51:53.616065 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:53.615988    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d5f074-e341-4b9d-bb24-94ea2def1539-model-cache" (OuterVolumeSpecName: "model-cache") pod "a3d5f074-e341-4b9d-bb24-94ea2def1539" (UID: "a3d5f074-e341-4b9d-bb24-94ea2def1539"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:51:53.616269 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:53.616056    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3d5f074-e341-4b9d-bb24-94ea2def1539-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a3d5f074-e341-4b9d-bb24-94ea2def1539" (UID: "a3d5f074-e341-4b9d-bb24-94ea2def1539"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:51:53.616345 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:53.616325    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d5f074-e341-4b9d-bb24-94ea2def1539-dshm" (OuterVolumeSpecName: "dshm") pod "a3d5f074-e341-4b9d-bb24-94ea2def1539" (UID: "a3d5f074-e341-4b9d-bb24-94ea2def1539"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:51:53.616786 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:53.616750    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3d5f074-e341-4b9d-bb24-94ea2def1539-kube-api-access-4sh4w" (OuterVolumeSpecName: "kube-api-access-4sh4w") pod "a3d5f074-e341-4b9d-bb24-94ea2def1539" (UID: "a3d5f074-e341-4b9d-bb24-94ea2def1539"). InnerVolumeSpecName "kube-api-access-4sh4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:51:53.629847 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:53.629818    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d5f074-e341-4b9d-bb24-94ea2def1539-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a3d5f074-e341-4b9d-bb24-94ea2def1539" (UID: "a3d5f074-e341-4b9d-bb24-94ea2def1539"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:51:53.631108 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:53.631078    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d5f074-e341-4b9d-bb24-94ea2def1539-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a3d5f074-e341-4b9d-bb24-94ea2def1539" (UID: "a3d5f074-e341-4b9d-bb24-94ea2def1539"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:51:53.714801 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:53.714761    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3d5f074-e341-4b9d-bb24-94ea2def1539-home\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:51:53.714801 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:53.714793    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sh4w\" (UniqueName: \"kubernetes.io/projected/a3d5f074-e341-4b9d-bb24-94ea2def1539-kube-api-access-4sh4w\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:51:53.714801 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:53.714808    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3d5f074-e341-4b9d-bb24-94ea2def1539-tmp-dir\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:51:53.715035 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:53.714819    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3d5f074-e341-4b9d-bb24-94ea2def1539-dshm\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:51:53.715035 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:53.714830    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3d5f074-e341-4b9d-bb24-94ea2def1539-model-cache\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:51:53.715035 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:53.714840    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3d5f074-e341-4b9d-bb24-94ea2def1539-tls-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:51:53.715035 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:53.714851    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3d5f074-e341-4b9d-bb24-94ea2def1539-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:51:54.517422 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:54.517384    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl"
Sep 17 22:51:54.517810 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:54.517391    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl" event={"ID":"a3d5f074-e341-4b9d-bb24-94ea2def1539","Type":"ContainerDied","Data":"7989654e66c7cd3ed64776208a2b57056560395419916b0cf77620d6c48e3ef4"}
Sep 17 22:51:54.517810 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:54.517537    2555 scope.go:117] "RemoveContainer" containerID="8329ce19dfed9be5b6157b8c7b1f6ac13ed4282c4df5b33cbfd3370fc3896ca1"
Sep 17 22:51:54.529024 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:54.529004    2555 scope.go:117] "RemoveContainer" containerID="8b2516e52eb1dae7af0fce3470acaa0e9ff4678cf4905390879092ac69716881"
Sep 17 22:51:54.540746 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:54.540716    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl"]
Sep 17 22:51:54.544664 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:54.544639    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bc757c44c-c2rzl"]
Sep 17 22:51:54.679475 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:54.679435    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3d5f074-e341-4b9d-bb24-94ea2def1539" path="/var/lib/kubelet/pods/a3d5f074-e341-4b9d-bb24-94ea2def1539/volumes"
Sep 17 22:51:57.978576 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:51:57.978525    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p" podUID="f9e40a60-0a35-47b9-bf21-4762112ec07e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": dial tcp 10.132.0.86:8000: connect: connection refused"
Sep 17 22:52:07.978671 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:07.978566    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p" podUID="f9e40a60-0a35-47b9-bf21-4762112ec07e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": dial tcp 10.132.0.86:8000: connect: connection refused"
Sep 17 22:52:17.978165 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:17.978114    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p" podUID="f9e40a60-0a35-47b9-bf21-4762112ec07e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": dial tcp 10.132.0.86:8000: connect: connection refused"
Sep 17 22:52:27.979130 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:27.979083    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p" podUID="f9e40a60-0a35-47b9-bf21-4762112ec07e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": dial tcp 10.132.0.86:8000: connect: connection refused"
Sep 17 22:52:28.103479 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:28.103446    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:52:28.202925 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:28.202881    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:52:28.356090 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:28.356046    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:52:28.420263 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:28.420222    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:52:31.775869 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:31.775835    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:52:34.426046 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:34.426011    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:52:37.978579 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:37.978521    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p" podUID="f9e40a60-0a35-47b9-bf21-4762112ec07e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": dial tcp 10.132.0.86:8000: connect: connection refused"
Sep 17 22:52:39.427799 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:39.427742    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57"]
Sep 17 22:52:39.428319 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:39.428143    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3d5f074-e341-4b9d-bb24-94ea2def1539" containerName="main"
Sep 17 22:52:39.428319 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:39.428155    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3d5f074-e341-4b9d-bb24-94ea2def1539" containerName="main"
Sep 17 22:52:39.428319 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:39.428177    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3d5f074-e341-4b9d-bb24-94ea2def1539" containerName="storage-initializer"
Sep 17 22:52:39.428319 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:39.428197    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3d5f074-e341-4b9d-bb24-94ea2def1539" containerName="storage-initializer"
Sep 17 22:52:39.428319 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:39.428249    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3d5f074-e341-4b9d-bb24-94ea2def1539" containerName="main"
Sep 17 22:52:39.431672 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:39.431650    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57"
Sep 17 22:52:39.434175 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:39.434148    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 17 22:52:39.434366 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:39.434151    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 17 22:52:39.434366 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:39.434291    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-xs6r6\""
Sep 17 22:52:39.434520 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:39.434374    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:52:39.434625 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:39.434604    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 17 22:52:39.434690 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:39.434663    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 17 22:52:39.442493 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:39.442470    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57"]
Sep 17 22:52:39.522560 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:39.522526    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:52:39.530707 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:39.530677    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b9ad214-52e4-49d1-b001-cb26a7319a9f-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57\" (UID: \"8b9ad214-52e4-49d1-b001-cb26a7319a9f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57"
Sep 17 22:52:39.530866 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:39.530722    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b9ad214-52e4-49d1-b001-cb26a7319a9f-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57\" (UID: \"8b9ad214-52e4-49d1-b001-cb26a7319a9f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57"
Sep 17 22:52:39.530866 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:39.530767    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b9ad214-52e4-49d1-b001-cb26a7319a9f-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57\" (UID: \"8b9ad214-52e4-49d1-b001-cb26a7319a9f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57"
Sep 17 22:52:39.530866 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:39.530794    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b9ad214-52e4-49d1-b001-cb26a7319a9f-home\") pod \"router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57\" (UID: \"8b9ad214-52e4-49d1-b001-cb26a7319a9f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57"
Sep 17 22:52:39.530866 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:39.530822    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b9ad214-52e4-49d1-b001-cb26a7319a9f-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57\" (UID: \"8b9ad214-52e4-49d1-b001-cb26a7319a9f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57"
Sep 17 22:52:39.530866 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:39.530845    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b9ad214-52e4-49d1-b001-cb26a7319a9f-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57\" (UID: \"8b9ad214-52e4-49d1-b001-cb26a7319a9f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57"
Sep 17 22:52:39.531164 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:39.531001    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49kjp\" (UniqueName: \"kubernetes.io/projected/8b9ad214-52e4-49d1-b001-cb26a7319a9f-kube-api-access-49kjp\") pod \"router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57\" (UID: \"8b9ad214-52e4-49d1-b001-cb26a7319a9f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57"
Sep 17 22:52:39.531164 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:39.531050    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b9ad214-52e4-49d1-b001-cb26a7319a9f-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57\" (UID: \"8b9ad214-52e4-49d1-b001-cb26a7319a9f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57"
Sep 17 22:52:39.631572 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:39.631536    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b9ad214-52e4-49d1-b001-cb26a7319a9f-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57\" (UID: \"8b9ad214-52e4-49d1-b001-cb26a7319a9f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57"
Sep 17 22:52:39.631572 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:39.631572    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b9ad214-52e4-49d1-b001-cb26a7319a9f-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57\" (UID: \"8b9ad214-52e4-49d1-b001-cb26a7319a9f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57"
Sep 17 22:52:39.631814 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:39.631656    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49kjp\" (UniqueName: \"kubernetes.io/projected/8b9ad214-52e4-49d1-b001-cb26a7319a9f-kube-api-access-49kjp\") pod \"router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57\" (UID: \"8b9ad214-52e4-49d1-b001-cb26a7319a9f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57"
Sep 17 22:52:39.631814 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:39.631696    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b9ad214-52e4-49d1-b001-cb26a7319a9f-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57\" (UID: \"8b9ad214-52e4-49d1-b001-cb26a7319a9f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57"
Sep 17 22:52:39.631814 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:39.631732    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b9ad214-52e4-49d1-b001-cb26a7319a9f-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57\" (UID: \"8b9ad214-52e4-49d1-b001-cb26a7319a9f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57"
Sep 17 22:52:39.631814 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:39.631763    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b9ad214-52e4-49d1-b001-cb26a7319a9f-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57\" (UID: \"8b9ad214-52e4-49d1-b001-cb26a7319a9f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57"
Sep 17 22:52:39.631814 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:39.631788    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b9ad214-52e4-49d1-b001-cb26a7319a9f-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57\" (UID: \"8b9ad214-52e4-49d1-b001-cb26a7319a9f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57"
Sep 17 22:52:39.631814 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:39.631815    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b9ad214-52e4-49d1-b001-cb26a7319a9f-home\") pod \"router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57\" (UID: \"8b9ad214-52e4-49d1-b001-cb26a7319a9f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57"
Sep 17 22:52:39.632155 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:39.632002    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b9ad214-52e4-49d1-b001-cb26a7319a9f-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57\" (UID: \"8b9ad214-52e4-49d1-b001-cb26a7319a9f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57"
Sep 17 22:52:39.632155 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:39.632085    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b9ad214-52e4-49d1-b001-cb26a7319a9f-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57\" (UID: \"8b9ad214-52e4-49d1-b001-cb26a7319a9f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57"
Sep 17 22:52:39.632300 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:39.632151    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b9ad214-52e4-49d1-b001-cb26a7319a9f-home\") pod \"router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57\" (UID: \"8b9ad214-52e4-49d1-b001-cb26a7319a9f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57"
Sep 17 22:52:39.632407 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:39.632381    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b9ad214-52e4-49d1-b001-cb26a7319a9f-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57\" (UID: \"8b9ad214-52e4-49d1-b001-cb26a7319a9f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57"
Sep 17 22:52:39.632407 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:39.632397    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b9ad214-52e4-49d1-b001-cb26a7319a9f-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57\" (UID: \"8b9ad214-52e4-49d1-b001-cb26a7319a9f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57"
Sep 17 22:52:39.634498 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:39.634465    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b9ad214-52e4-49d1-b001-cb26a7319a9f-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57\" (UID: \"8b9ad214-52e4-49d1-b001-cb26a7319a9f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57"
Sep 17 22:52:39.634807 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:39.634780    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b9ad214-52e4-49d1-b001-cb26a7319a9f-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57\" (UID: \"8b9ad214-52e4-49d1-b001-cb26a7319a9f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57"
Sep 17 22:52:39.639887 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:39.639860    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49kjp\" (UniqueName: \"kubernetes.io/projected/8b9ad214-52e4-49d1-b001-cb26a7319a9f-kube-api-access-49kjp\") pod \"router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57\" (UID: \"8b9ad214-52e4-49d1-b001-cb26a7319a9f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57"
Sep 17 22:52:39.742984 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:39.742890    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57"
Sep 17 22:52:39.777023 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:39.776918    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:52:39.842868 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:39.842809    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:52:40.034506 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:40.034473    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57"]
Sep 17 22:52:40.035163 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:52:40.035139    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b9ad214_52e4_49d1_b001_cb26a7319a9f.slice/crio-66623890bed5f5e7eba8e343586e42a1bdb2681e1d9dccaca5e0c15088e7f418 WatchSource:0}: Error finding container 66623890bed5f5e7eba8e343586e42a1bdb2681e1d9dccaca5e0c15088e7f418: Status 404 returned error can't find the container with id 66623890bed5f5e7eba8e343586e42a1bdb2681e1d9dccaca5e0c15088e7f418
Sep 17 22:52:40.490929 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:40.490891    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:52:40.564793 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:40.564748    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:52:40.812515 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:40.812472    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57" event={"ID":"8b9ad214-52e4-49d1-b001-cb26a7319a9f","Type":"ContainerStarted","Data":"6d5c1adca58cbd0509cf21e8708bd5b9d30d117d3385d164e792716bdd92fe61"}
Sep 17 22:52:40.812710 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:40.812548    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57" event={"ID":"8b9ad214-52e4-49d1-b001-cb26a7319a9f","Type":"ContainerStarted","Data":"66623890bed5f5e7eba8e343586e42a1bdb2681e1d9dccaca5e0c15088e7f418"}
Sep 17 22:52:42.831815 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:42.831728    2555 generic.go:356] "Generic (PLEG): container finished" podID="8b9ad214-52e4-49d1-b001-cb26a7319a9f" containerID="6d5c1adca58cbd0509cf21e8708bd5b9d30d117d3385d164e792716bdd92fe61" exitCode=0
Sep 17 22:52:42.832203 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:42.831807    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57" event={"ID":"8b9ad214-52e4-49d1-b001-cb26a7319a9f","Type":"ContainerDied","Data":"6d5c1adca58cbd0509cf21e8708bd5b9d30d117d3385d164e792716bdd92fe61"}
Sep 17 22:52:43.845690 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:43.845648    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57" event={"ID":"8b9ad214-52e4-49d1-b001-cb26a7319a9f","Type":"ContainerStarted","Data":"84b3e0afcf79ac02d7a6762973e86aee061c199e33cb4d53cec8befbf5ba6c4f"}
Sep 17 22:52:43.868744 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:43.868689    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57" podStartSLOduration=2.416617973 podStartE2EDuration="4.868674571s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.452056598s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:52:43.867395231 +0000 UTC m=+3265.313440505" watchObservedRunningTime="2026-09-17 22:52:43.868674571 +0000 UTC m=+3265.314719842"
Sep 17 22:52:47.989079 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:47.989044    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"
Sep 17 22:52:47.996953 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:47.996920    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"
Sep 17 22:52:49.743811 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:49.743773    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57"
Sep 17 22:52:49.744229 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:49.743842    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57"
Sep 17 22:52:49.745525 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:49.745492    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57" podUID="8b9ad214-52e4-49d1-b001-cb26a7319a9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": dial tcp 10.132.0.87:8000: connect: connection refused"
Sep 17 22:52:59.743995 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:52:59.743940    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57" podUID="8b9ad214-52e4-49d1-b001-cb26a7319a9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": dial tcp 10.132.0.87:8000: connect: connection refused"
Sep 17 22:53:09.743533 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:09.743487    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57" podUID="8b9ad214-52e4-49d1-b001-cb26a7319a9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": dial tcp 10.132.0.87:8000: connect: connection refused"
Sep 17 22:53:19.743458 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:19.743418    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57" podUID="8b9ad214-52e4-49d1-b001-cb26a7319a9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": dial tcp 10.132.0.87:8000: connect: connection refused"
Sep 17 22:53:21.243979 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:21.243946    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p_f9e40a60-0a35-47b9-bf21-4762112ec07e/storage-initializer/0.log"
Sep 17 22:53:21.254637 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:21.254613    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p_f9e40a60-0a35-47b9-bf21-4762112ec07e/main/0.log"
Sep 17 22:53:22.267417 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:22.267381    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"]
Sep 17 22:53:22.290014 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:22.289982    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:53:22.954322 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:22.954290    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-7njvj_a324c9ab-5d42-4d31-8ff7-449f619be774/console-operator/1.log"
Sep 17 22:53:22.958804 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:22.958779    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s7f5_4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf/ovn-acl-logging/0.log"
Sep 17 22:53:22.970575 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:22.970555    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-7njvj_a324c9ab-5d42-4d31-8ff7-449f619be774/console-operator/1.log"
Sep 17 22:53:22.974594 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:22.974574    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s7f5_4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf/ovn-acl-logging/0.log"
Sep 17 22:53:29.744001 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:29.743952    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57" podUID="8b9ad214-52e4-49d1-b001-cb26a7319a9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": dial tcp 10.132.0.87:8000: connect: connection refused"
Sep 17 22:53:37.292339 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:37.292236    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p" podUID="f9e40a60-0a35-47b9-bf21-4762112ec07e" containerName="main" containerID="cri-o://72053bbd7cd2708ced81a1f812cc59513f8c8ca8db26c6af3c5ac59b274f3046" gracePeriod=45
Sep 17 22:53:37.941590 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:37.941561    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"
Sep 17 22:53:37.981392 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:37.981353    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-662ss\" (UniqueName: \"kubernetes.io/projected/f9e40a60-0a35-47b9-bf21-4762112ec07e-kube-api-access-662ss\") pod \"f9e40a60-0a35-47b9-bf21-4762112ec07e\" (UID: \"f9e40a60-0a35-47b9-bf21-4762112ec07e\") "
Sep 17 22:53:37.981561 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:37.981487    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"lora-pvc-lora-adapter-1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04e013e9e3c8a1904\") pod \"f9e40a60-0a35-47b9-bf21-4762112ec07e\" (UID: \"f9e40a60-0a35-47b9-bf21-4762112ec07e\") "
Sep 17 22:53:37.981561 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:37.981520    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9e40a60-0a35-47b9-bf21-4762112ec07e-kserve-provision-location\") pod \"f9e40a60-0a35-47b9-bf21-4762112ec07e\" (UID: \"f9e40a60-0a35-47b9-bf21-4762112ec07e\") "
Sep 17 22:53:37.981561 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:37.981541    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9e40a60-0a35-47b9-bf21-4762112ec07e-dshm\") pod \"f9e40a60-0a35-47b9-bf21-4762112ec07e\" (UID: \"f9e40a60-0a35-47b9-bf21-4762112ec07e\") "
Sep 17 22:53:37.981721 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:37.981588    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9e40a60-0a35-47b9-bf21-4762112ec07e-home\") pod \"f9e40a60-0a35-47b9-bf21-4762112ec07e\" (UID: \"f9e40a60-0a35-47b9-bf21-4762112ec07e\") "
Sep 17 22:53:37.981721 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:37.981681    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9e40a60-0a35-47b9-bf21-4762112ec07e-tls-certs\") pod \"f9e40a60-0a35-47b9-bf21-4762112ec07e\" (UID: \"f9e40a60-0a35-47b9-bf21-4762112ec07e\") "
Sep 17 22:53:37.981721 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:37.981710    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9e40a60-0a35-47b9-bf21-4762112ec07e-model-cache\") pod \"f9e40a60-0a35-47b9-bf21-4762112ec07e\" (UID: \"f9e40a60-0a35-47b9-bf21-4762112ec07e\") "
Sep 17 22:53:37.981876 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:37.981737    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9e40a60-0a35-47b9-bf21-4762112ec07e-cabundle-cert\") pod \"f9e40a60-0a35-47b9-bf21-4762112ec07e\" (UID: \"f9e40a60-0a35-47b9-bf21-4762112ec07e\") "
Sep 17 22:53:37.981876 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:37.981766    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9e40a60-0a35-47b9-bf21-4762112ec07e-tmp-dir\") pod \"f9e40a60-0a35-47b9-bf21-4762112ec07e\" (UID: \"f9e40a60-0a35-47b9-bf21-4762112ec07e\") "
Sep 17 22:53:37.983285 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:37.982665    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9e40a60-0a35-47b9-bf21-4762112ec07e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f9e40a60-0a35-47b9-bf21-4762112ec07e" (UID: "f9e40a60-0a35-47b9-bf21-4762112ec07e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:53:37.983285 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:37.982909    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9e40a60-0a35-47b9-bf21-4762112ec07e-model-cache" (OuterVolumeSpecName: "model-cache") pod "f9e40a60-0a35-47b9-bf21-4762112ec07e" (UID: "f9e40a60-0a35-47b9-bf21-4762112ec07e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:53:37.983285 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:37.983255    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9e40a60-0a35-47b9-bf21-4762112ec07e-home" (OuterVolumeSpecName: "home") pod "f9e40a60-0a35-47b9-bf21-4762112ec07e" (UID: "f9e40a60-0a35-47b9-bf21-4762112ec07e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:53:37.986079 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:37.986045    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9e40a60-0a35-47b9-bf21-4762112ec07e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f9e40a60-0a35-47b9-bf21-4762112ec07e" (UID: "f9e40a60-0a35-47b9-bf21-4762112ec07e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:53:37.986326 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:37.986136    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9e40a60-0a35-47b9-bf21-4762112ec07e-dshm" (OuterVolumeSpecName: "dshm") pod "f9e40a60-0a35-47b9-bf21-4762112ec07e" (UID: "f9e40a60-0a35-47b9-bf21-4762112ec07e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:53:37.986326 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:37.986280    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9e40a60-0a35-47b9-bf21-4762112ec07e-kube-api-access-662ss" (OuterVolumeSpecName: "kube-api-access-662ss") pod "f9e40a60-0a35-47b9-bf21-4762112ec07e" (UID: "f9e40a60-0a35-47b9-bf21-4762112ec07e"). InnerVolumeSpecName "kube-api-access-662ss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:53:37.988874 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:37.988847    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-04e013e9e3c8a1904" (OuterVolumeSpecName: "lora-pvc-lora-adapter-1") pod "f9e40a60-0a35-47b9-bf21-4762112ec07e" (UID: "f9e40a60-0a35-47b9-bf21-4762112ec07e"). InnerVolumeSpecName "pvc-2f66f8a0-2e27-4acc-a637-98baaa34f261". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 17 22:53:37.994604 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:37.994550    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9e40a60-0a35-47b9-bf21-4762112ec07e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f9e40a60-0a35-47b9-bf21-4762112ec07e" (UID: "f9e40a60-0a35-47b9-bf21-4762112ec07e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:53:38.019084 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:38.019048    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9e40a60-0a35-47b9-bf21-4762112ec07e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f9e40a60-0a35-47b9-bf21-4762112ec07e" (UID: "f9e40a60-0a35-47b9-bf21-4762112ec07e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:53:38.083368 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:38.083336    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9e40a60-0a35-47b9-bf21-4762112ec07e-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:53:38.083500 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:38.083370    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9e40a60-0a35-47b9-bf21-4762112ec07e-tmp-dir\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:53:38.083500 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:38.083386    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-662ss\" (UniqueName: \"kubernetes.io/projected/f9e40a60-0a35-47b9-bf21-4762112ec07e-kube-api-access-662ss\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:53:38.083500 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:38.083427    2555 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-2f66f8a0-2e27-4acc-a637-98baaa34f261\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04e013e9e3c8a1904\") on node \"ip-10-0-140-157.ec2.internal\" "
Sep 17 22:53:38.083500 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:38.083446    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9e40a60-0a35-47b9-bf21-4762112ec07e-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:53:38.083500 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:38.083461    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9e40a60-0a35-47b9-bf21-4762112ec07e-dshm\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:53:38.083500 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:38.083474    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9e40a60-0a35-47b9-bf21-4762112ec07e-home\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:53:38.083500 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:38.083488    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9e40a60-0a35-47b9-bf21-4762112ec07e-tls-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:53:38.083500 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:38.083501    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9e40a60-0a35-47b9-bf21-4762112ec07e-model-cache\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:53:38.093474 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:38.093454    2555 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-2f66f8a0-2e27-4acc-a637-98baaa34f261" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-04e013e9e3c8a1904") on node "ip-10-0-140-157.ec2.internal"
Sep 17 22:53:38.184336 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:38.184312    2555 reconciler_common.go:299] "Volume detached for volume \"pvc-2f66f8a0-2e27-4acc-a637-98baaa34f261\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04e013e9e3c8a1904\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"csi-b9f2b492fff5e22748471b807bad60723efa4c55583e5cca0f3921f3b4c8bbe0\""
Sep 17 22:53:38.188478 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:38.188444    2555 generic.go:356] "Generic (PLEG): container finished" podID="f9e40a60-0a35-47b9-bf21-4762112ec07e" containerID="72053bbd7cd2708ced81a1f812cc59513f8c8ca8db26c6af3c5ac59b274f3046" exitCode=0
Sep 17 22:53:38.188584 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:38.188506    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"
Sep 17 22:53:38.188584 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:38.188532    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p" event={"ID":"f9e40a60-0a35-47b9-bf21-4762112ec07e","Type":"ContainerDied","Data":"72053bbd7cd2708ced81a1f812cc59513f8c8ca8db26c6af3c5ac59b274f3046"}
Sep 17 22:53:38.188684 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:38.188587    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p" event={"ID":"f9e40a60-0a35-47b9-bf21-4762112ec07e","Type":"ContainerDied","Data":"255ec586f7b58064cff6eff6ef97760579b95240dec281a0cc7b7f59fe0a248e"}
Sep 17 22:53:38.188684 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:38.188618    2555 scope.go:117] "RemoveContainer" containerID="72053bbd7cd2708ced81a1f812cc59513f8c8ca8db26c6af3c5ac59b274f3046"
Sep 17 22:53:38.199362 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:38.199335    2555 scope.go:117] "RemoveContainer" containerID="d0fac35bb0d1e5471f70009434a10f547785e4bdc0d0ccdfdce2e74972c684e6"
Sep 17 22:53:38.207514 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:38.207496    2555 scope.go:117] "RemoveContainer" containerID="72053bbd7cd2708ced81a1f812cc59513f8c8ca8db26c6af3c5ac59b274f3046"
Sep 17 22:53:38.207762 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:53:38.207745    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72053bbd7cd2708ced81a1f812cc59513f8c8ca8db26c6af3c5ac59b274f3046\": container with ID starting with 72053bbd7cd2708ced81a1f812cc59513f8c8ca8db26c6af3c5ac59b274f3046 not found: ID does not exist" containerID="72053bbd7cd2708ced81a1f812cc59513f8c8ca8db26c6af3c5ac59b274f3046"
Sep 17 22:53:38.207831 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:38.207767    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72053bbd7cd2708ced81a1f812cc59513f8c8ca8db26c6af3c5ac59b274f3046"} err="failed to get container status \"72053bbd7cd2708ced81a1f812cc59513f8c8ca8db26c6af3c5ac59b274f3046\": rpc error: code = NotFound desc = could not find container \"72053bbd7cd2708ced81a1f812cc59513f8c8ca8db26c6af3c5ac59b274f3046\": container with ID starting with 72053bbd7cd2708ced81a1f812cc59513f8c8ca8db26c6af3c5ac59b274f3046 not found: ID does not exist"
Sep 17 22:53:38.207831 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:38.207781    2555 scope.go:117] "RemoveContainer" containerID="d0fac35bb0d1e5471f70009434a10f547785e4bdc0d0ccdfdce2e74972c684e6"
Sep 17 22:53:38.208036 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:53:38.208019    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0fac35bb0d1e5471f70009434a10f547785e4bdc0d0ccdfdce2e74972c684e6\": container with ID starting with d0fac35bb0d1e5471f70009434a10f547785e4bdc0d0ccdfdce2e74972c684e6 not found: ID does not exist" containerID="d0fac35bb0d1e5471f70009434a10f547785e4bdc0d0ccdfdce2e74972c684e6"
Sep 17 22:53:38.208075 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:38.208041    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0fac35bb0d1e5471f70009434a10f547785e4bdc0d0ccdfdce2e74972c684e6"} err="failed to get container status \"d0fac35bb0d1e5471f70009434a10f547785e4bdc0d0ccdfdce2e74972c684e6\": rpc error: code = NotFound desc = could not find container \"d0fac35bb0d1e5471f70009434a10f547785e4bdc0d0ccdfdce2e74972c684e6\": container with ID starting with d0fac35bb0d1e5471f70009434a10f547785e4bdc0d0ccdfdce2e74972c684e6 not found: ID does not exist"
Sep 17 22:53:38.212820 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:38.212790    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"]
Sep 17 22:53:38.216931 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:38.216907    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-65c6d87774-2qp4p"]
Sep 17 22:53:38.679298 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:38.679262    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9e40a60-0a35-47b9-bf21-4762112ec07e" path="/var/lib/kubelet/pods/f9e40a60-0a35-47b9-bf21-4762112ec07e/volumes"
Sep 17 22:53:39.744195 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:39.744137    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57" podUID="8b9ad214-52e4-49d1-b001-cb26a7319a9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": dial tcp 10.132.0.87:8000: connect: connection refused"
Sep 17 22:53:43.344363 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:43.344326    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:53:43.445569 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:43.445532    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:53:45.118249 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:45.118216    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:53:45.205247 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:45.205216    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:53:45.876512 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:45.876468    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-jt89z"]
Sep 17 22:53:45.876961 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:45.876942    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9e40a60-0a35-47b9-bf21-4762112ec07e" containerName="storage-initializer"
Sep 17 22:53:45.877058 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:45.876963    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9e40a60-0a35-47b9-bf21-4762112ec07e" containerName="storage-initializer"
Sep 17 22:53:45.877058 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:45.876979    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9e40a60-0a35-47b9-bf21-4762112ec07e" containerName="main"
Sep 17 22:53:45.877058 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:45.876987    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9e40a60-0a35-47b9-bf21-4762112ec07e" containerName="main"
Sep 17 22:53:45.877227 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:45.877116    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9e40a60-0a35-47b9-bf21-4762112ec07e" containerName="main"
Sep 17 22:53:45.881802 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:45.881781    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-jt89z"
Sep 17 22:53:45.884335 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:45.884312    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-26pcl\""
Sep 17 22:53:45.884525 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:45.884378    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 17 22:53:45.884600 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:45.884389    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"seaweedfs-s3-creds\""
Sep 17 22:53:45.885140 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:45.885122    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 17 22:53:45.885950 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:45.885930    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-jt89z"]
Sep 17 22:53:45.955958 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:45.955928    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlqlm\" (UniqueName: \"kubernetes.io/projected/dd83092c-7729-4f86-9c4c-2722a11b8aab-kube-api-access-nlqlm\") pod \"e2e-pvc-lora-download-jt89z\" (UID: \"dd83092c-7729-4f86-9c4c-2722a11b8aab\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-jt89z"
Sep 17 22:53:45.978434 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:45.978410    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:53:46.056851 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:46.056815    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlqlm\" (UniqueName: \"kubernetes.io/projected/dd83092c-7729-4f86-9c4c-2722a11b8aab-kube-api-access-nlqlm\") pod \"e2e-pvc-lora-download-jt89z\" (UID: \"dd83092c-7729-4f86-9c4c-2722a11b8aab\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-jt89z"
Sep 17 22:53:46.065138 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:46.065114    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlqlm\" (UniqueName: \"kubernetes.io/projected/dd83092c-7729-4f86-9c4c-2722a11b8aab-kube-api-access-nlqlm\") pod \"e2e-pvc-lora-download-jt89z\" (UID: \"dd83092c-7729-4f86-9c4c-2722a11b8aab\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-jt89z"
Sep 17 22:53:47.549654 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:47.549607    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xq2zq"]
Sep 17 22:53:47.552907 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:47.552878    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xq2zq"]
Sep 17 22:53:48.436152 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:48.436111    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:53:48.535591 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:48.535553    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:53:48.644604 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:48.644566    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:53:48.680522 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:48.680485    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dce86c89-c1dc-4513-bf88-5bd913ebc676" path="/var/lib/kubelet/pods/dce86c89-c1dc-4513-bf88-5bd913ebc676/volumes"
Sep 17 22:53:48.698475 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:48.698397    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:53:49.743654 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:49.743602    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57" podUID="8b9ad214-52e4-49d1-b001-cb26a7319a9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": dial tcp 10.132.0.87:8000: connect: connection refused"
Sep 17 22:53:50.294464 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:50.294426    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-1745cf5c-2ab1-4673-b3a1-58ebcacefbc5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f467ab85ae1c531e\") pod \"e2e-pvc-lora-download-jt89z\" (UID: \"dd83092c-7729-4f86-9c4c-2722a11b8aab\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-jt89z"
Sep 17 22:53:50.299374 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:50.299336    2555 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-1745cf5c-2ab1-4673-b3a1-58ebcacefbc5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f467ab85ae1c531e\") pod \"e2e-pvc-lora-download-jt89z\" (UID: \"dd83092c-7729-4f86-9c4c-2722a11b8aab\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-jt89z"
Sep 17 22:53:50.394935 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:50.394891    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-1745cf5c-2ab1-4673-b3a1-58ebcacefbc5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f467ab85ae1c531e\") pod \"e2e-pvc-lora-download-jt89z\" (UID: \"dd83092c-7729-4f86-9c4c-2722a11b8aab\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-jt89z"
Sep 17 22:53:50.395139 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:50.395027    2555 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-1745cf5c-2ab1-4673-b3a1-58ebcacefbc5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f467ab85ae1c531e\") pod \"e2e-pvc-lora-download-jt89z\" (UID: \"dd83092c-7729-4f86-9c4c-2722a11b8aab\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-jt89z"
Sep 17 22:53:50.397877 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:50.397852    2555 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-1745cf5c-2ab1-4673-b3a1-58ebcacefbc5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f467ab85ae1c531e\") pod \"e2e-pvc-lora-download-jt89z\" (UID: \"dd83092c-7729-4f86-9c4c-2722a11b8aab\") DevicePath \"csi-f5c98a41544b090f4de3a2c550ab88940aec62ec09b3339210f465eca2617c28\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-jt89z"
Sep 17 22:53:50.610166 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:50.610125    2555 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-1745cf5c-2ab1-4673-b3a1-58ebcacefbc5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f467ab85ae1c531e\") pod \"e2e-pvc-lora-download-jt89z\" (UID: \"dd83092c-7729-4f86-9c4c-2722a11b8aab\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/68be14a7b51aceb62f964f391fb35fc333ac63549aa25d1bd2ae490712d340f8/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-jt89z"
Sep 17 22:53:50.623936 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:50.623907    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-1745cf5c-2ab1-4673-b3a1-58ebcacefbc5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f467ab85ae1c531e\") pod \"e2e-pvc-lora-download-jt89z\" (UID: \"dd83092c-7729-4f86-9c4c-2722a11b8aab\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-jt89z"
Sep 17 22:53:50.693285 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:50.693252    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-jt89z"
Sep 17 22:53:50.966643 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:50.966610    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-jt89z"]
Sep 17 22:53:50.968579 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:53:50.968554    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd83092c_7729_4f86_9c4c_2722a11b8aab.slice/crio-3a91cb9e15d8a934783942b137f24343b25e82b957aca74f08ec69052bde9d62 WatchSource:0}: Error finding container 3a91cb9e15d8a934783942b137f24343b25e82b957aca74f08ec69052bde9d62: Status 404 returned error can't find the container with id 3a91cb9e15d8a934783942b137f24343b25e82b957aca74f08ec69052bde9d62
Sep 17 22:53:51.267637 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:51.267527    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-jt89z" event={"ID":"dd83092c-7729-4f86-9c4c-2722a11b8aab","Type":"ContainerStarted","Data":"4a9e7bbbed0675af6d36a549fabcacce695f1beee5aa9b1e8443ac1bc9db7427"}
Sep 17 22:53:51.267637 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:51.267587    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-jt89z" event={"ID":"dd83092c-7729-4f86-9c4c-2722a11b8aab","Type":"ContainerStarted","Data":"3a91cb9e15d8a934783942b137f24343b25e82b957aca74f08ec69052bde9d62"}
Sep 17 22:53:51.285267 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:51.285208    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-jt89z" podStartSLOduration=9.285169684 podStartE2EDuration="9.285169684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 22:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:53:51.281030215 +0000 UTC m=+3332.727075485" watchObservedRunningTime="2026-09-17 22:53:51.285169684 +0000 UTC m=+3332.731214961"
Sep 17 22:53:53.283897 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:53.283810    2555 generic.go:356] "Generic (PLEG): container finished" podID="dd83092c-7729-4f86-9c4c-2722a11b8aab" containerID="4a9e7bbbed0675af6d36a549fabcacce695f1beee5aa9b1e8443ac1bc9db7427" exitCode=0
Sep 17 22:53:53.283897 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:53.283856    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-jt89z" event={"ID":"dd83092c-7729-4f86-9c4c-2722a11b8aab","Type":"ContainerDied","Data":"4a9e7bbbed0675af6d36a549fabcacce695f1beee5aa9b1e8443ac1bc9db7427"}
Sep 17 22:53:54.052098 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:54.052061    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:53:54.152342 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:54.152310    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:53:54.442171 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:54.442146    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-jt89z"
Sep 17 22:53:54.533841 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:54.533807    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f467ab85ae1c531e\") pod \"dd83092c-7729-4f86-9c4c-2722a11b8aab\" (UID: \"dd83092c-7729-4f86-9c4c-2722a11b8aab\") "
Sep 17 22:53:54.534006 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:54.533914    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlqlm\" (UniqueName: \"kubernetes.io/projected/dd83092c-7729-4f86-9c4c-2722a11b8aab-kube-api-access-nlqlm\") pod \"dd83092c-7729-4f86-9c4c-2722a11b8aab\" (UID: \"dd83092c-7729-4f86-9c4c-2722a11b8aab\") "
Sep 17 22:53:54.536355 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:54.536312    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd83092c-7729-4f86-9c4c-2722a11b8aab-kube-api-access-nlqlm" (OuterVolumeSpecName: "kube-api-access-nlqlm") pod "dd83092c-7729-4f86-9c4c-2722a11b8aab" (UID: "dd83092c-7729-4f86-9c4c-2722a11b8aab"). InnerVolumeSpecName "kube-api-access-nlqlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:53:54.536619 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:54.536597    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0f467ab85ae1c531e" (OuterVolumeSpecName: "model-storage") pod "dd83092c-7729-4f86-9c4c-2722a11b8aab" (UID: "dd83092c-7729-4f86-9c4c-2722a11b8aab"). InnerVolumeSpecName "pvc-1745cf5c-2ab1-4673-b3a1-58ebcacefbc5". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 17 22:53:54.635287 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:54.635259    2555 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-1745cf5c-2ab1-4673-b3a1-58ebcacefbc5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f467ab85ae1c531e\") on node \"ip-10-0-140-157.ec2.internal\" "
Sep 17 22:53:54.635287 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:54.635288    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlqlm\" (UniqueName: \"kubernetes.io/projected/dd83092c-7729-4f86-9c4c-2722a11b8aab-kube-api-access-nlqlm\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:53:55.296967 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:55.296934    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-jt89z"
Sep 17 22:53:55.297170 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:55.296976    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-jt89z" event={"ID":"dd83092c-7729-4f86-9c4c-2722a11b8aab","Type":"ContainerDied","Data":"3a91cb9e15d8a934783942b137f24343b25e82b957aca74f08ec69052bde9d62"}
Sep 17 22:53:55.297170 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:55.297017    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a91cb9e15d8a934783942b137f24343b25e82b957aca74f08ec69052bde9d62"
Sep 17 22:53:55.606040 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:55.606014    2555 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-1745cf5c-2ab1-4673-b3a1-58ebcacefbc5" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0f467ab85ae1c531e") on node "ip-10-0-140-157.ec2.internal"
Sep 17 22:53:55.643131 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:55.643107    2555 reconciler_common.go:299] "Volume detached for volume \"pvc-1745cf5c-2ab1-4673-b3a1-58ebcacefbc5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f467ab85ae1c531e\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"csi-f5c98a41544b090f4de3a2c550ab88940aec62ec09b3339210f465eca2617c28\""
Sep 17 22:53:59.744461 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:53:59.744410    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57" podUID="8b9ad214-52e4-49d1-b001-cb26a7319a9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": dial tcp 10.132.0.87:8000: connect: connection refused"
Sep 17 22:54:08.139201 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:08.139146    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk"]
Sep 17 22:54:08.139610 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:08.139594    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd83092c-7729-4f86-9c4c-2722a11b8aab" containerName="storage-initializer"
Sep 17 22:54:08.139664 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:08.139612    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd83092c-7729-4f86-9c4c-2722a11b8aab" containerName="storage-initializer"
Sep 17 22:54:08.139702 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:08.139667    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd83092c-7729-4f86-9c4c-2722a11b8aab" containerName="storage-initializer"
Sep 17 22:54:08.142903 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:08.142877    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk"
Sep 17 22:54:08.145694 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:08.145668    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 17 22:54:08.145812 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:08.145668    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-26pcl\""
Sep 17 22:54:08.145812 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:08.145670    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 17 22:54:08.145812 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:08.145674    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"lora-pvc-adapters-test-kserve-self-signed-certs\""
Sep 17 22:54:08.153589 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:08.153565    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk"]
Sep 17 22:54:08.162339 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:08.162307    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:54:08.255454 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:08.255418    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e881f616-0001-401d-9526-402705cdd9b3-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-czjbk\" (UID: \"e881f616-0001-401d-9526-402705cdd9b3\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk"
Sep 17 22:54:08.255632 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:08.255482    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e881f616-0001-401d-9526-402705cdd9b3-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-czjbk\" (UID: \"e881f616-0001-401d-9526-402705cdd9b3\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk"
Sep 17 22:54:08.255632 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:08.255581    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ks59\" (UniqueName: \"kubernetes.io/projected/e881f616-0001-401d-9526-402705cdd9b3-kube-api-access-9ks59\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-czjbk\" (UID: \"e881f616-0001-401d-9526-402705cdd9b3\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk"
Sep 17 22:54:08.255711 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:08.255636    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e881f616-0001-401d-9526-402705cdd9b3-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-czjbk\" (UID: \"e881f616-0001-401d-9526-402705cdd9b3\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk"
Sep 17 22:54:08.255711 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:08.255688    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e881f616-0001-401d-9526-402705cdd9b3-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-czjbk\" (UID: \"e881f616-0001-401d-9526-402705cdd9b3\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk"
Sep 17 22:54:08.255776 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:08.255749    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e881f616-0001-401d-9526-402705cdd9b3-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-czjbk\" (UID: \"e881f616-0001-401d-9526-402705cdd9b3\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk"
Sep 17 22:54:08.356964 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:08.356917    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e881f616-0001-401d-9526-402705cdd9b3-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-czjbk\" (UID: \"e881f616-0001-401d-9526-402705cdd9b3\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk"
Sep 17 22:54:08.357137 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:08.356981    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e881f616-0001-401d-9526-402705cdd9b3-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-czjbk\" (UID: \"e881f616-0001-401d-9526-402705cdd9b3\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk"
Sep 17 22:54:08.357137 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:08.357030    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e881f616-0001-401d-9526-402705cdd9b3-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-czjbk\" (UID: \"e881f616-0001-401d-9526-402705cdd9b3\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk"
Sep 17 22:54:08.357137 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:08.357108    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e881f616-0001-401d-9526-402705cdd9b3-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-czjbk\" (UID: \"e881f616-0001-401d-9526-402705cdd9b3\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk"
Sep 17 22:54:08.357346 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:08.357162    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ks59\" (UniqueName: \"kubernetes.io/projected/e881f616-0001-401d-9526-402705cdd9b3-kube-api-access-9ks59\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-czjbk\" (UID: \"e881f616-0001-401d-9526-402705cdd9b3\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk"
Sep 17 22:54:08.357346 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:08.357215    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e881f616-0001-401d-9526-402705cdd9b3-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-czjbk\" (UID: \"e881f616-0001-401d-9526-402705cdd9b3\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk"
Sep 17 22:54:08.357487 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:08.357464    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e881f616-0001-401d-9526-402705cdd9b3-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-czjbk\" (UID: \"e881f616-0001-401d-9526-402705cdd9b3\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk"
Sep 17 22:54:08.357544 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:08.357501    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e881f616-0001-401d-9526-402705cdd9b3-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-czjbk\" (UID: \"e881f616-0001-401d-9526-402705cdd9b3\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk"
Sep 17 22:54:08.357614 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:08.357598    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e881f616-0001-401d-9526-402705cdd9b3-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-czjbk\" (UID: \"e881f616-0001-401d-9526-402705cdd9b3\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk"
Sep 17 22:54:08.359369 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:08.359343    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e881f616-0001-401d-9526-402705cdd9b3-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-czjbk\" (UID: \"e881f616-0001-401d-9526-402705cdd9b3\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk"
Sep 17 22:54:08.359927 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:08.359905    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e881f616-0001-401d-9526-402705cdd9b3-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-czjbk\" (UID: \"e881f616-0001-401d-9526-402705cdd9b3\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk"
Sep 17 22:54:08.365149 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:08.365124    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ks59\" (UniqueName: \"kubernetes.io/projected/e881f616-0001-401d-9526-402705cdd9b3-kube-api-access-9ks59\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-czjbk\" (UID: \"e881f616-0001-401d-9526-402705cdd9b3\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk"
Sep 17 22:54:09.743633 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:09.743587    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57" podUID="8b9ad214-52e4-49d1-b001-cb26a7319a9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": dial tcp 10.132.0.87:8000: connect: connection refused"
Sep 17 22:54:10.783774 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:10.783722    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-1745cf5c-2ab1-4673-b3a1-58ebcacefbc5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f467ab85ae1c531e\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-czjbk\" (UID: \"e881f616-0001-401d-9526-402705cdd9b3\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk"
Sep 17 22:54:10.789481 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:10.789449    2555 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-1745cf5c-2ab1-4673-b3a1-58ebcacefbc5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f467ab85ae1c531e\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-czjbk\" (UID: \"e881f616-0001-401d-9526-402705cdd9b3\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk"
Sep 17 22:54:10.885242 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:10.885160    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-1745cf5c-2ab1-4673-b3a1-58ebcacefbc5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f467ab85ae1c531e\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-czjbk\" (UID: \"e881f616-0001-401d-9526-402705cdd9b3\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk"
Sep 17 22:54:10.885811 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:10.885570    2555 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-1745cf5c-2ab1-4673-b3a1-58ebcacefbc5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f467ab85ae1c531e\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-czjbk\" (UID: \"e881f616-0001-401d-9526-402705cdd9b3\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk"
Sep 17 22:54:10.888662 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:10.888631    2555 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-1745cf5c-2ab1-4673-b3a1-58ebcacefbc5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f467ab85ae1c531e\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-czjbk\" (UID: \"e881f616-0001-401d-9526-402705cdd9b3\") DevicePath \"csi-f5c98a41544b090f4de3a2c550ab88940aec62ec09b3339210f465eca2617c28\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk"
Sep 17 22:54:10.996391 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:10.996317    2555 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-1745cf5c-2ab1-4673-b3a1-58ebcacefbc5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f467ab85ae1c531e\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-czjbk\" (UID: \"e881f616-0001-401d-9526-402705cdd9b3\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/68be14a7b51aceb62f964f391fb35fc333ac63549aa25d1bd2ae490712d340f8/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk"
Sep 17 22:54:11.025858 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:11.025833    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-1745cf5c-2ab1-4673-b3a1-58ebcacefbc5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f467ab85ae1c531e\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-czjbk\" (UID: \"e881f616-0001-401d-9526-402705cdd9b3\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk"
Sep 17 22:54:11.155254 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:11.155213    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk"
Sep 17 22:54:11.464210 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:54:11.464108    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode881f616_0001_401d_9526_402705cdd9b3.slice/crio-32a661f1ef2bec7d1432fdb64fa54666337343fd10428aa47e4658be0d9941c2 WatchSource:0}: Error finding container 32a661f1ef2bec7d1432fdb64fa54666337343fd10428aa47e4658be0d9941c2: Status 404 returned error can't find the container with id 32a661f1ef2bec7d1432fdb64fa54666337343fd10428aa47e4658be0d9941c2
Sep 17 22:54:11.464761 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:11.464725    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk"]
Sep 17 22:54:12.422165 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:12.422106    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk" event={"ID":"e881f616-0001-401d-9526-402705cdd9b3","Type":"ContainerStarted","Data":"bce0e85b2e6c1f85740b6f7e7a4bfdc6eb84d3e6135b6da2a9d643a48e5c8f12"}
Sep 17 22:54:12.422681 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:12.422213    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk" event={"ID":"e881f616-0001-401d-9526-402705cdd9b3","Type":"ContainerStarted","Data":"32a661f1ef2bec7d1432fdb64fa54666337343fd10428aa47e4658be0d9941c2"}
Sep 17 22:54:12.448832 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:12.448766    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk" podStartSLOduration=4.448749775 podStartE2EDuration="4.448749775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 22:54:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:54:12.445263006 +0000 UTC m=+3353.891308277" watchObservedRunningTime="2026-09-17 22:54:12.448749775 +0000 UTC m=+3353.894795045"
Sep 17 22:54:13.155670 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:13.155636    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk"
Sep 17 22:54:19.744281 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:19.744238    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57" podUID="8b9ad214-52e4-49d1-b001-cb26a7319a9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": dial tcp 10.132.0.87:8000: connect: connection refused"
Sep 17 22:54:21.155861 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:21.155819    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk"
Sep 17 22:54:21.157611 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:21.157566    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk" podUID="e881f616-0001-401d-9526-402705cdd9b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": dial tcp 10.132.0.89:8000: connect: connection refused"
Sep 17 22:54:29.744051 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:29.744004    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57" podUID="8b9ad214-52e4-49d1-b001-cb26a7319a9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": dial tcp 10.132.0.87:8000: connect: connection refused"
Sep 17 22:54:31.155656 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:31.155615    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk" podUID="e881f616-0001-401d-9526-402705cdd9b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": dial tcp 10.132.0.89:8000: connect: connection refused"
Sep 17 22:54:39.756319 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:39.756280    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57"
Sep 17 22:54:39.765061 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:39.765034    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57"
Sep 17 22:54:41.156424 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:41.156371    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk" podUID="e881f616-0001-401d-9526-402705cdd9b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": dial tcp 10.132.0.89:8000: connect: connection refused"
Sep 17 22:54:48.328890 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:48.328852    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:54:48.344873 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:48.344847    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57"]
Sep 17 22:54:48.396533 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:48.396500    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:54:51.155937 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:54:51.155897    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk" podUID="e881f616-0001-401d-9526-402705cdd9b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": dial tcp 10.132.0.89:8000: connect: connection refused"
Sep 17 22:55:01.155787 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:01.155738    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk" podUID="e881f616-0001-401d-9526-402705cdd9b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": dial tcp 10.132.0.89:8000: connect: connection refused"
Sep 17 22:55:03.368553 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:03.368447    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57" podUID="8b9ad214-52e4-49d1-b001-cb26a7319a9f" containerName="main" containerID="cri-o://84b3e0afcf79ac02d7a6762973e86aee061c199e33cb4d53cec8befbf5ba6c4f" gracePeriod=45
Sep 17 22:55:04.125274 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:04.125200    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57"
Sep 17 22:55:04.219763 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:04.219722    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b9ad214-52e4-49d1-b001-cb26a7319a9f-dshm\") pod \"8b9ad214-52e4-49d1-b001-cb26a7319a9f\" (UID: \"8b9ad214-52e4-49d1-b001-cb26a7319a9f\") "
Sep 17 22:55:04.219763 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:04.219769    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b9ad214-52e4-49d1-b001-cb26a7319a9f-home\") pod \"8b9ad214-52e4-49d1-b001-cb26a7319a9f\" (UID: \"8b9ad214-52e4-49d1-b001-cb26a7319a9f\") "
Sep 17 22:55:04.220064 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:04.219812    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b9ad214-52e4-49d1-b001-cb26a7319a9f-model-cache\") pod \"8b9ad214-52e4-49d1-b001-cb26a7319a9f\" (UID: \"8b9ad214-52e4-49d1-b001-cb26a7319a9f\") "
Sep 17 22:55:04.220064 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:04.219840    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b9ad214-52e4-49d1-b001-cb26a7319a9f-kserve-provision-location\") pod \"8b9ad214-52e4-49d1-b001-cb26a7319a9f\" (UID: \"8b9ad214-52e4-49d1-b001-cb26a7319a9f\") "
Sep 17 22:55:04.220064 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:04.219937    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b9ad214-52e4-49d1-b001-cb26a7319a9f-cabundle-cert\") pod \"8b9ad214-52e4-49d1-b001-cb26a7319a9f\" (UID: \"8b9ad214-52e4-49d1-b001-cb26a7319a9f\") "
Sep 17 22:55:04.220064 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:04.219970    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49kjp\" (UniqueName: \"kubernetes.io/projected/8b9ad214-52e4-49d1-b001-cb26a7319a9f-kube-api-access-49kjp\") pod \"8b9ad214-52e4-49d1-b001-cb26a7319a9f\" (UID: \"8b9ad214-52e4-49d1-b001-cb26a7319a9f\") "
Sep 17 22:55:04.220064 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:04.219997    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b9ad214-52e4-49d1-b001-cb26a7319a9f-tmp-dir\") pod \"8b9ad214-52e4-49d1-b001-cb26a7319a9f\" (UID: \"8b9ad214-52e4-49d1-b001-cb26a7319a9f\") "
Sep 17 22:55:04.220064 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:04.220023    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b9ad214-52e4-49d1-b001-cb26a7319a9f-tls-certs\") pod \"8b9ad214-52e4-49d1-b001-cb26a7319a9f\" (UID: \"8b9ad214-52e4-49d1-b001-cb26a7319a9f\") "
Sep 17 22:55:04.220392 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:04.220149    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b9ad214-52e4-49d1-b001-cb26a7319a9f-model-cache" (OuterVolumeSpecName: "model-cache") pod "8b9ad214-52e4-49d1-b001-cb26a7319a9f" (UID: "8b9ad214-52e4-49d1-b001-cb26a7319a9f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:55:04.220392 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:04.220318    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b9ad214-52e4-49d1-b001-cb26a7319a9f-model-cache\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:55:04.220913 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:04.220665    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b9ad214-52e4-49d1-b001-cb26a7319a9f-home" (OuterVolumeSpecName: "home") pod "8b9ad214-52e4-49d1-b001-cb26a7319a9f" (UID: "8b9ad214-52e4-49d1-b001-cb26a7319a9f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:55:04.221275 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:04.221215    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b9ad214-52e4-49d1-b001-cb26a7319a9f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8b9ad214-52e4-49d1-b001-cb26a7319a9f" (UID: "8b9ad214-52e4-49d1-b001-cb26a7319a9f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:55:04.222929 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:04.222896    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b9ad214-52e4-49d1-b001-cb26a7319a9f-dshm" (OuterVolumeSpecName: "dshm") pod "8b9ad214-52e4-49d1-b001-cb26a7319a9f" (UID: "8b9ad214-52e4-49d1-b001-cb26a7319a9f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:55:04.222929 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:04.222910    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b9ad214-52e4-49d1-b001-cb26a7319a9f-kube-api-access-49kjp" (OuterVolumeSpecName: "kube-api-access-49kjp") pod "8b9ad214-52e4-49d1-b001-cb26a7319a9f" (UID: "8b9ad214-52e4-49d1-b001-cb26a7319a9f"). InnerVolumeSpecName "kube-api-access-49kjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:55:04.223089 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:04.222988    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b9ad214-52e4-49d1-b001-cb26a7319a9f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8b9ad214-52e4-49d1-b001-cb26a7319a9f" (UID: "8b9ad214-52e4-49d1-b001-cb26a7319a9f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:55:04.241147 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:04.241105    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b9ad214-52e4-49d1-b001-cb26a7319a9f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8b9ad214-52e4-49d1-b001-cb26a7319a9f" (UID: "8b9ad214-52e4-49d1-b001-cb26a7319a9f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:55:04.242908 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:04.242876    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b9ad214-52e4-49d1-b001-cb26a7319a9f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8b9ad214-52e4-49d1-b001-cb26a7319a9f" (UID: "8b9ad214-52e4-49d1-b001-cb26a7319a9f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:55:04.321160 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:04.321126    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b9ad214-52e4-49d1-b001-cb26a7319a9f-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:55:04.321160 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:04.321154    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49kjp\" (UniqueName: \"kubernetes.io/projected/8b9ad214-52e4-49d1-b001-cb26a7319a9f-kube-api-access-49kjp\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:55:04.321381 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:04.321200    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b9ad214-52e4-49d1-b001-cb26a7319a9f-tmp-dir\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:55:04.321381 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:04.321215    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b9ad214-52e4-49d1-b001-cb26a7319a9f-tls-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:55:04.321381 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:04.321223    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b9ad214-52e4-49d1-b001-cb26a7319a9f-dshm\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:55:04.321381 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:04.321230    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b9ad214-52e4-49d1-b001-cb26a7319a9f-home\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:55:04.321381 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:04.321237    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b9ad214-52e4-49d1-b001-cb26a7319a9f-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:55:04.750525 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:04.750483    2555 generic.go:356] "Generic (PLEG): container finished" podID="8b9ad214-52e4-49d1-b001-cb26a7319a9f" containerID="84b3e0afcf79ac02d7a6762973e86aee061c199e33cb4d53cec8befbf5ba6c4f" exitCode=0
Sep 17 22:55:04.751009 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:04.750546    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57"
Sep 17 22:55:04.751009 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:04.750579    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57" event={"ID":"8b9ad214-52e4-49d1-b001-cb26a7319a9f","Type":"ContainerDied","Data":"84b3e0afcf79ac02d7a6762973e86aee061c199e33cb4d53cec8befbf5ba6c4f"}
Sep 17 22:55:04.751009 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:04.750661    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57" event={"ID":"8b9ad214-52e4-49d1-b001-cb26a7319a9f","Type":"ContainerDied","Data":"66623890bed5f5e7eba8e343586e42a1bdb2681e1d9dccaca5e0c15088e7f418"}
Sep 17 22:55:04.751009 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:04.750707    2555 scope.go:117] "RemoveContainer" containerID="84b3e0afcf79ac02d7a6762973e86aee061c199e33cb4d53cec8befbf5ba6c4f"
Sep 17 22:55:04.760632 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:04.760610    2555 scope.go:117] "RemoveContainer" containerID="6d5c1adca58cbd0509cf21e8708bd5b9d30d117d3385d164e792716bdd92fe61"
Sep 17 22:55:04.767997 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:04.767970    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57"]
Sep 17 22:55:04.769918 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:04.769862    2555 scope.go:117] "RemoveContainer" containerID="84b3e0afcf79ac02d7a6762973e86aee061c199e33cb4d53cec8befbf5ba6c4f"
Sep 17 22:55:04.770650 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:55:04.770360    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84b3e0afcf79ac02d7a6762973e86aee061c199e33cb4d53cec8befbf5ba6c4f\": container with ID starting with 84b3e0afcf79ac02d7a6762973e86aee061c199e33cb4d53cec8befbf5ba6c4f not found: ID does not exist" containerID="84b3e0afcf79ac02d7a6762973e86aee061c199e33cb4d53cec8befbf5ba6c4f"
Sep 17 22:55:04.770650 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:04.770398    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84b3e0afcf79ac02d7a6762973e86aee061c199e33cb4d53cec8befbf5ba6c4f"} err="failed to get container status \"84b3e0afcf79ac02d7a6762973e86aee061c199e33cb4d53cec8befbf5ba6c4f\": rpc error: code = NotFound desc = could not find container \"84b3e0afcf79ac02d7a6762973e86aee061c199e33cb4d53cec8befbf5ba6c4f\": container with ID starting with 84b3e0afcf79ac02d7a6762973e86aee061c199e33cb4d53cec8befbf5ba6c4f not found: ID does not exist"
Sep 17 22:55:04.770650 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:04.770423    2555 scope.go:117] "RemoveContainer" containerID="6d5c1adca58cbd0509cf21e8708bd5b9d30d117d3385d164e792716bdd92fe61"
Sep 17 22:55:04.770971 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:55:04.770944    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d5c1adca58cbd0509cf21e8708bd5b9d30d117d3385d164e792716bdd92fe61\": container with ID starting with 6d5c1adca58cbd0509cf21e8708bd5b9d30d117d3385d164e792716bdd92fe61 not found: ID does not exist" containerID="6d5c1adca58cbd0509cf21e8708bd5b9d30d117d3385d164e792716bdd92fe61"
Sep 17 22:55:04.771086 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:04.770979    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d5c1adca58cbd0509cf21e8708bd5b9d30d117d3385d164e792716bdd92fe61"} err="failed to get container status \"6d5c1adca58cbd0509cf21e8708bd5b9d30d117d3385d164e792716bdd92fe61\": rpc error: code = NotFound desc = could not find container \"6d5c1adca58cbd0509cf21e8708bd5b9d30d117d3385d164e792716bdd92fe61\": container with ID starting with 6d5c1adca58cbd0509cf21e8708bd5b9d30d117d3385d164e792716bdd92fe61 not found: ID does not exist"
Sep 17 22:55:04.772362 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:04.772342    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5b966d57cc-rtc57"]
Sep 17 22:55:06.678784 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:06.678738    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b9ad214-52e4-49d1-b001-cb26a7319a9f" path="/var/lib/kubelet/pods/8b9ad214-52e4-49d1-b001-cb26a7319a9f/volumes"
Sep 17 22:55:11.155841 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:11.155789    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk" podUID="e881f616-0001-401d-9526-402705cdd9b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": dial tcp 10.132.0.89:8000: connect: connection refused"
Sep 17 22:55:21.155773 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:21.155722    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk" podUID="e881f616-0001-401d-9526-402705cdd9b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": dial tcp 10.132.0.89:8000: connect: connection refused"
Sep 17 22:55:22.282309 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:22.282273    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:55:31.165802 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:31.165769    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk"
Sep 17 22:55:31.173379 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:31.173350    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk"
Sep 17 22:55:35.247773 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:35.247739    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:55:35.339687 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:35.339646    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:55:38.443007 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:38.442967    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:55:38.536287 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:38.536250    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:55:43.492321 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:43.492278    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:55:59.119193 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:59.119097    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-a290d4b0_lora-pvc-adapters-test-kserve-8cfd5947f-czjbk_e881f616-0001-401d-9526-402705cdd9b3/main/0.log"
Sep 17 22:55:59.673631 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:55:59.673597    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:56:04.965824 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:04.965792    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:56:05.438900 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:05.438870    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:56:10.886417 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:10.886381    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:56:14.429486 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:14.429452    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk"]
Sep 17 22:56:14.450103 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:14.450072    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:56:26.684518 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:26.684487    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:56:29.454246 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:29.454201    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk" podUID="e881f616-0001-401d-9526-402705cdd9b3" containerName="main" containerID="cri-o://bce0e85b2e6c1f85740b6f7e7a4bfdc6eb84d3e6135b6da2a9d643a48e5c8f12" gracePeriod=45
Sep 17 22:56:30.114914 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:30.114888    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk"
Sep 17 22:56:30.253977 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:30.253875    2555 generic.go:356] "Generic (PLEG): container finished" podID="e881f616-0001-401d-9526-402705cdd9b3" containerID="bce0e85b2e6c1f85740b6f7e7a4bfdc6eb84d3e6135b6da2a9d643a48e5c8f12" exitCode=0
Sep 17 22:56:30.253977 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:30.253922    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk"
Sep 17 22:56:30.254170 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:30.253969    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk" event={"ID":"e881f616-0001-401d-9526-402705cdd9b3","Type":"ContainerDied","Data":"bce0e85b2e6c1f85740b6f7e7a4bfdc6eb84d3e6135b6da2a9d643a48e5c8f12"}
Sep 17 22:56:30.254170 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:30.254050    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk" event={"ID":"e881f616-0001-401d-9526-402705cdd9b3","Type":"ContainerDied","Data":"32a661f1ef2bec7d1432fdb64fa54666337343fd10428aa47e4658be0d9941c2"}
Sep 17 22:56:30.254170 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:30.254073    2555 scope.go:117] "RemoveContainer" containerID="bce0e85b2e6c1f85740b6f7e7a4bfdc6eb84d3e6135b6da2a9d643a48e5c8f12"
Sep 17 22:56:30.263094 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:30.263072    2555 scope.go:117] "RemoveContainer" containerID="bce0e85b2e6c1f85740b6f7e7a4bfdc6eb84d3e6135b6da2a9d643a48e5c8f12"
Sep 17 22:56:30.263402 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:56:30.263385    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bce0e85b2e6c1f85740b6f7e7a4bfdc6eb84d3e6135b6da2a9d643a48e5c8f12\": container with ID starting with bce0e85b2e6c1f85740b6f7e7a4bfdc6eb84d3e6135b6da2a9d643a48e5c8f12 not found: ID does not exist" containerID="bce0e85b2e6c1f85740b6f7e7a4bfdc6eb84d3e6135b6da2a9d643a48e5c8f12"
Sep 17 22:56:30.263471 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:30.263412    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bce0e85b2e6c1f85740b6f7e7a4bfdc6eb84d3e6135b6da2a9d643a48e5c8f12"} err="failed to get container status \"bce0e85b2e6c1f85740b6f7e7a4bfdc6eb84d3e6135b6da2a9d643a48e5c8f12\": rpc error: code = NotFound desc = could not find container \"bce0e85b2e6c1f85740b6f7e7a4bfdc6eb84d3e6135b6da2a9d643a48e5c8f12\": container with ID starting with bce0e85b2e6c1f85740b6f7e7a4bfdc6eb84d3e6135b6da2a9d643a48e5c8f12 not found: ID does not exist"
Sep 17 22:56:30.269690 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:30.269672    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e881f616-0001-401d-9526-402705cdd9b3-tls-certs\") pod \"e881f616-0001-401d-9526-402705cdd9b3\" (UID: \"e881f616-0001-401d-9526-402705cdd9b3\") "
Sep 17 22:56:30.269769 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:30.269702    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e881f616-0001-401d-9526-402705cdd9b3-model-cache\") pod \"e881f616-0001-401d-9526-402705cdd9b3\" (UID: \"e881f616-0001-401d-9526-402705cdd9b3\") "
Sep 17 22:56:30.269769 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:30.269742    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e881f616-0001-401d-9526-402705cdd9b3-tmp-dir\") pod \"e881f616-0001-401d-9526-402705cdd9b3\" (UID: \"e881f616-0001-401d-9526-402705cdd9b3\") "
Sep 17 22:56:30.269889 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:30.269813    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f467ab85ae1c531e\") pod \"e881f616-0001-401d-9526-402705cdd9b3\" (UID: \"e881f616-0001-401d-9526-402705cdd9b3\") "
Sep 17 22:56:30.269889 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:30.269838    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e881f616-0001-401d-9526-402705cdd9b3-home\") pod \"e881f616-0001-401d-9526-402705cdd9b3\" (UID: \"e881f616-0001-401d-9526-402705cdd9b3\") "
Sep 17 22:56:30.269889 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:30.269855    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e881f616-0001-401d-9526-402705cdd9b3-dshm\") pod \"e881f616-0001-401d-9526-402705cdd9b3\" (UID: \"e881f616-0001-401d-9526-402705cdd9b3\") "
Sep 17 22:56:30.270041 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:30.269891    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ks59\" (UniqueName: \"kubernetes.io/projected/e881f616-0001-401d-9526-402705cdd9b3-kube-api-access-9ks59\") pod \"e881f616-0001-401d-9526-402705cdd9b3\" (UID: \"e881f616-0001-401d-9526-402705cdd9b3\") "
Sep 17 22:56:30.270324 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:30.270286    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e881f616-0001-401d-9526-402705cdd9b3-home" (OuterVolumeSpecName: "home") pod "e881f616-0001-401d-9526-402705cdd9b3" (UID: "e881f616-0001-401d-9526-402705cdd9b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:56:30.270324 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:30.270310    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e881f616-0001-401d-9526-402705cdd9b3-model-cache" (OuterVolumeSpecName: "model-cache") pod "e881f616-0001-401d-9526-402705cdd9b3" (UID: "e881f616-0001-401d-9526-402705cdd9b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:56:30.272195 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:30.272151    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e881f616-0001-401d-9526-402705cdd9b3-dshm" (OuterVolumeSpecName: "dshm") pod "e881f616-0001-401d-9526-402705cdd9b3" (UID: "e881f616-0001-401d-9526-402705cdd9b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:56:30.272311 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:30.272172    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e881f616-0001-401d-9526-402705cdd9b3-kube-api-access-9ks59" (OuterVolumeSpecName: "kube-api-access-9ks59") pod "e881f616-0001-401d-9526-402705cdd9b3" (UID: "e881f616-0001-401d-9526-402705cdd9b3"). InnerVolumeSpecName "kube-api-access-9ks59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:56:30.272311 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:30.272298    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e881f616-0001-401d-9526-402705cdd9b3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e881f616-0001-401d-9526-402705cdd9b3" (UID: "e881f616-0001-401d-9526-402705cdd9b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:56:30.277696 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:30.277677    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0f467ab85ae1c531e" (OuterVolumeSpecName: "kserve-pvc-source") pod "e881f616-0001-401d-9526-402705cdd9b3" (UID: "e881f616-0001-401d-9526-402705cdd9b3"). InnerVolumeSpecName "pvc-1745cf5c-2ab1-4673-b3a1-58ebcacefbc5". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 17 22:56:30.281788 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:30.281764    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e881f616-0001-401d-9526-402705cdd9b3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e881f616-0001-401d-9526-402705cdd9b3" (UID: "e881f616-0001-401d-9526-402705cdd9b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:56:30.371475 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:30.371449    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e881f616-0001-401d-9526-402705cdd9b3-tls-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:56:30.371475 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:30.371473    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e881f616-0001-401d-9526-402705cdd9b3-model-cache\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:56:30.371608 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:30.371481    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e881f616-0001-401d-9526-402705cdd9b3-tmp-dir\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:56:30.371608 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:30.371506    2555 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-1745cf5c-2ab1-4673-b3a1-58ebcacefbc5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f467ab85ae1c531e\") on node \"ip-10-0-140-157.ec2.internal\" "
Sep 17 22:56:30.371608 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:30.371517    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e881f616-0001-401d-9526-402705cdd9b3-home\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:56:30.371608 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:30.371526    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e881f616-0001-401d-9526-402705cdd9b3-dshm\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:56:30.371608 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:30.371536    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ks59\" (UniqueName: \"kubernetes.io/projected/e881f616-0001-401d-9526-402705cdd9b3-kube-api-access-9ks59\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:56:30.403985 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:30.403967    2555 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-1745cf5c-2ab1-4673-b3a1-58ebcacefbc5" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0f467ab85ae1c531e") on node "ip-10-0-140-157.ec2.internal"
Sep 17 22:56:30.472050 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:30.472025    2555 reconciler_common.go:299] "Volume detached for volume \"pvc-1745cf5c-2ab1-4673-b3a1-58ebcacefbc5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f467ab85ae1c531e\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"csi-f5c98a41544b090f4de3a2c550ab88940aec62ec09b3339210f465eca2617c28\""
Sep 17 22:56:30.576647 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:30.576617    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk"]
Sep 17 22:56:30.582237 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:30.582213    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-czjbk"]
Sep 17 22:56:30.680681 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:30.680647    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e881f616-0001-401d-9526-402705cdd9b3" path="/var/lib/kubelet/pods/e881f616-0001-401d-9526-402705cdd9b3/volumes"
Sep 17 22:56:30.922161 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:30.922136    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:56:31.020749 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:31.020713    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:56:33.718441 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:33.718354    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:56:33.818315 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:33.818282    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:56:35.902046 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:35.901999    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-jt89z"]
Sep 17 22:56:35.904234 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:35.904212    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-jt89z"]
Sep 17 22:56:36.100335 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:36.100304    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:56:36.198363 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:36.198281    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:56:36.370575 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:36.370545    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:56:36.680168 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:36.680138    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd83092c-7729-4f86-9c4c-2722a11b8aab" path="/var/lib/kubelet/pods/dd83092c-7729-4f86-9c4c-2722a11b8aab/volumes"
Sep 17 22:56:36.817975 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:36.817944    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:56:37.260010 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:37.259979    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:56:37.360242 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:37.360207    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:56:42.359218 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:42.359161    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:56:42.459742 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:42.459711    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:56:47.549360 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:47.549324    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q"]
Sep 17 22:56:47.549823 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:47.549697    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b9ad214-52e4-49d1-b001-cb26a7319a9f" containerName="main"
Sep 17 22:56:47.549823 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:47.549709    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b9ad214-52e4-49d1-b001-cb26a7319a9f" containerName="main"
Sep 17 22:56:47.549823 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:47.549727    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b9ad214-52e4-49d1-b001-cb26a7319a9f" containerName="storage-initializer"
Sep 17 22:56:47.549823 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:47.549733    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b9ad214-52e4-49d1-b001-cb26a7319a9f" containerName="storage-initializer"
Sep 17 22:56:47.549823 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:47.549746    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e881f616-0001-401d-9526-402705cdd9b3" containerName="main"
Sep 17 22:56:47.549823 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:47.549751    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e881f616-0001-401d-9526-402705cdd9b3" containerName="main"
Sep 17 22:56:47.549823 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:47.549813    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b9ad214-52e4-49d1-b001-cb26a7319a9f" containerName="main"
Sep 17 22:56:47.549823 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:47.549819    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e881f616-0001-401d-9526-402705cdd9b3" containerName="main"
Sep 17 22:56:47.554477 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:47.554459    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q"
Sep 17 22:56:47.557008 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:47.556989    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 17 22:56:47.557125 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:47.557063    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"seaweedfs-s3-creds\""
Sep 17 22:56:47.557125 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:47.557085    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 17 22:56:47.557283 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:47.557128    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:56:47.557283 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:47.557145    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 17 22:56:47.557283 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:47.557146    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-w9d94\""
Sep 17 22:56:47.562873 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:47.562851    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q"]
Sep 17 22:56:47.612240 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:47.612210    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-tmp-dir\") pod \"tls-verification-test-kserve-5c798c6c78-q6v7q\" (UID: \"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q"
Sep 17 22:56:47.612436 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:47.612247    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb69x\" (UniqueName: \"kubernetes.io/projected/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-kube-api-access-pb69x\") pod \"tls-verification-test-kserve-5c798c6c78-q6v7q\" (UID: \"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q"
Sep 17 22:56:47.612436 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:47.612311    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-kserve-provision-location\") pod \"tls-verification-test-kserve-5c798c6c78-q6v7q\" (UID: \"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q"
Sep 17 22:56:47.612555 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:47.612451    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-tls-certs\") pod \"tls-verification-test-kserve-5c798c6c78-q6v7q\" (UID: \"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q"
Sep 17 22:56:47.612555 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:47.612482    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-cabundle-cert\") pod \"tls-verification-test-kserve-5c798c6c78-q6v7q\" (UID: \"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q"
Sep 17 22:56:47.612555 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:47.612514    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-model-cache\") pod \"tls-verification-test-kserve-5c798c6c78-q6v7q\" (UID: \"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q"
Sep 17 22:56:47.612660 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:47.612602    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-home\") pod \"tls-verification-test-kserve-5c798c6c78-q6v7q\" (UID: \"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q"
Sep 17 22:56:47.612660 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:47.612644    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-dshm\") pod \"tls-verification-test-kserve-5c798c6c78-q6v7q\" (UID: \"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q"
Sep 17 22:56:47.629656 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:47.629631    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:56:47.713577 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:47.713545    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-tmp-dir\") pod \"tls-verification-test-kserve-5c798c6c78-q6v7q\" (UID: \"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q"
Sep 17 22:56:47.713753 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:47.713588    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb69x\" (UniqueName: \"kubernetes.io/projected/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-kube-api-access-pb69x\") pod \"tls-verification-test-kserve-5c798c6c78-q6v7q\" (UID: \"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q"
Sep 17 22:56:47.713753 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:47.713621    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-kserve-provision-location\") pod \"tls-verification-test-kserve-5c798c6c78-q6v7q\" (UID: \"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q"
Sep 17 22:56:47.713753 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:47.713692    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-tls-certs\") pod \"tls-verification-test-kserve-5c798c6c78-q6v7q\" (UID: \"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q"
Sep 17 22:56:47.713753 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:47.713715    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-cabundle-cert\") pod \"tls-verification-test-kserve-5c798c6c78-q6v7q\" (UID: \"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q"
Sep 17 22:56:47.714011 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:47.713755    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-model-cache\") pod \"tls-verification-test-kserve-5c798c6c78-q6v7q\" (UID: \"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q"
Sep 17 22:56:47.714011 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:47.713783    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-home\") pod \"tls-verification-test-kserve-5c798c6c78-q6v7q\" (UID: \"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q"
Sep 17 22:56:47.714011 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:47.713828    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-dshm\") pod \"tls-verification-test-kserve-5c798c6c78-q6v7q\" (UID: \"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q"
Sep 17 22:56:47.714166 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:47.714033    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-tmp-dir\") pod \"tls-verification-test-kserve-5c798c6c78-q6v7q\" (UID: \"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q"
Sep 17 22:56:47.714273 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:47.714248    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-kserve-provision-location\") pod \"tls-verification-test-kserve-5c798c6c78-q6v7q\" (UID: \"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q"
Sep 17 22:56:47.714346 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:47.714267    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-model-cache\") pod \"tls-verification-test-kserve-5c798c6c78-q6v7q\" (UID: \"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q"
Sep 17 22:56:47.714405 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:47.714376    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-home\") pod \"tls-verification-test-kserve-5c798c6c78-q6v7q\" (UID: \"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q"
Sep 17 22:56:47.714619 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:47.714598    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-cabundle-cert\") pod \"tls-verification-test-kserve-5c798c6c78-q6v7q\" (UID: \"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q"
Sep 17 22:56:47.716128 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:47.716100    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-dshm\") pod \"tls-verification-test-kserve-5c798c6c78-q6v7q\" (UID: \"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q"
Sep 17 22:56:47.716472 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:47.716456    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-tls-certs\") pod \"tls-verification-test-kserve-5c798c6c78-q6v7q\" (UID: \"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q"
Sep 17 22:56:47.721278 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:47.721249    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb69x\" (UniqueName: \"kubernetes.io/projected/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-kube-api-access-pb69x\") pod \"tls-verification-test-kserve-5c798c6c78-q6v7q\" (UID: \"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q"
Sep 17 22:56:47.806428 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:47.806342    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:56:47.866348 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:47.866317    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q"
Sep 17 22:56:48.149648 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:48.149622    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q"]
Sep 17 22:56:48.151199 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:56:48.151150    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda882a1c3_a875_4eb0_a09f_c32cee1b4fbb.slice/crio-3f5615e76e1e6f0b009a351af994427ed2198f7487263e6bfb2af6cec3577df4 WatchSource:0}: Error finding container 3f5615e76e1e6f0b009a351af994427ed2198f7487263e6bfb2af6cec3577df4: Status 404 returned error can't find the container with id 3f5615e76e1e6f0b009a351af994427ed2198f7487263e6bfb2af6cec3577df4
Sep 17 22:56:48.358839 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:48.358802    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q" event={"ID":"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb","Type":"ContainerStarted","Data":"c4805edc5d2bb9d24be81acb8bcad67a0b324b206388a0501017294b56036d4c"}
Sep 17 22:56:48.359031 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:48.358858    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q" event={"ID":"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb","Type":"ContainerStarted","Data":"3f5615e76e1e6f0b009a351af994427ed2198f7487263e6bfb2af6cec3577df4"}
Sep 17 22:56:51.377790 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:51.377753    2555 generic.go:356] "Generic (PLEG): container finished" podID="a882a1c3-a875-4eb0-a09f-c32cee1b4fbb" containerID="c4805edc5d2bb9d24be81acb8bcad67a0b324b206388a0501017294b56036d4c" exitCode=0
Sep 17 22:56:51.378279 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:51.377825    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q" event={"ID":"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb","Type":"ContainerDied","Data":"c4805edc5d2bb9d24be81acb8bcad67a0b324b206388a0501017294b56036d4c"}
Sep 17 22:56:52.386237 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:52.386166    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q" event={"ID":"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb","Type":"ContainerStarted","Data":"09cd0f092ccf75ba8f42a5fcf349bed08c8ddd173d230bbeffcd8c3934f29efd"}
Sep 17 22:56:52.409229 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:52.409141    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q" podStartSLOduration=3.144933589 podStartE2EDuration="5.409121686s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.264188097s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:56:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:56:52.407336825 +0000 UTC m=+3513.853382098" watchObservedRunningTime="2026-09-17 22:56:52.409121686 +0000 UTC m=+3513.855166961"
Sep 17 22:56:52.866966 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:52.866881    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q"
Sep 17 22:56:57.866836 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:57.866796    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q"
Sep 17 22:56:57.868494 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:57.868470    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q" podUID="a882a1c3-a875-4eb0-a09f-c32cee1b4fbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": dial tcp 10.132.0.90:8000: connect: connection refused"
Sep 17 22:56:58.925787 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:56:58.925756    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:57:07.867654 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:57:07.867596    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q" podUID="a882a1c3-a875-4eb0-a09f-c32cee1b4fbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": dial tcp 10.132.0.90:8000: connect: connection refused"
Sep 17 22:57:17.867478 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:57:17.867435    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q" podUID="a882a1c3-a875-4eb0-a09f-c32cee1b4fbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": dial tcp 10.132.0.90:8000: connect: connection refused"
Sep 17 22:57:23.112365 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:57:23.112327    2555 scope.go:117] "RemoveContainer" containerID="e74dd28bafd87b8b038f3a5a5643413b24f0981c16e03a3a2c3b489ecf06ec98"
Sep 17 22:57:27.866807 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:57:27.866762    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q" podUID="a882a1c3-a875-4eb0-a09f-c32cee1b4fbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": dial tcp 10.132.0.90:8000: connect: connection refused"
Sep 17 22:57:37.867575 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:57:37.867528    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q" podUID="a882a1c3-a875-4eb0-a09f-c32cee1b4fbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": dial tcp 10.132.0.90:8000: connect: connection refused"
Sep 17 22:57:47.867000 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:57:47.866949    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q" podUID="a882a1c3-a875-4eb0-a09f-c32cee1b4fbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": dial tcp 10.132.0.90:8000: connect: connection refused"
Sep 17 22:57:54.671703 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:57:54.671669    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:57:57.867586 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:57:57.867547    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q" podUID="a882a1c3-a875-4eb0-a09f-c32cee1b4fbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": dial tcp 10.132.0.90:8000: connect: connection refused"
Sep 17 22:58:07.867176 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:07.867071    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q" podUID="a882a1c3-a875-4eb0-a09f-c32cee1b4fbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": dial tcp 10.132.0.90:8000: connect: connection refused"
Sep 17 22:58:14.852952 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:14.852920    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:58:14.951753 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:14.951721    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:58:17.160056 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:17.160020    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:58:17.281400 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:17.281368    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:58:17.417022 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:17.416891    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:58:17.443764 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:17.443732    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 22:58:17.447605 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:17.447581    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 22:58:17.450576 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:17.450551    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 17 22:58:17.450782 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:17.450572    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 17 22:58:17.451504 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:17.451478    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-zmcpl\""
Sep 17 22:58:17.451628 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:17.451547    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-74x5g\""
Sep 17 22:58:17.451628 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:17.451599    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:58:17.451759 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:17.451739    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 17 22:58:17.451817 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:17.451521    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 17 22:58:17.463600 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:17.463580    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 22:58:17.506006 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:17.505975    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:58:17.606869 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:17.606837    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5lfw\" (UniqueName: \"kubernetes.io/projected/6c707f08-4328-42c3-a2e4-b8231b9a0b16-kube-api-access-v5lfw\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6c707f08-4328-42c3-a2e4-b8231b9a0b16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 22:58:17.607039 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:17.606882    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c707f08-4328-42c3-a2e4-b8231b9a0b16-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6c707f08-4328-42c3-a2e4-b8231b9a0b16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 22:58:17.607039 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:17.606909    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c707f08-4328-42c3-a2e4-b8231b9a0b16-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6c707f08-4328-42c3-a2e4-b8231b9a0b16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 22:58:17.607039 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:17.606993    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c707f08-4328-42c3-a2e4-b8231b9a0b16-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6c707f08-4328-42c3-a2e4-b8231b9a0b16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 22:58:17.607039 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:17.607021    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c707f08-4328-42c3-a2e4-b8231b9a0b16-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6c707f08-4328-42c3-a2e4-b8231b9a0b16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 22:58:17.607240 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:17.607087    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c707f08-4328-42c3-a2e4-b8231b9a0b16-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6c707f08-4328-42c3-a2e4-b8231b9a0b16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 22:58:17.607240 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:17.607106    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c707f08-4328-42c3-a2e4-b8231b9a0b16-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6c707f08-4328-42c3-a2e4-b8231b9a0b16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 22:58:17.607240 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:17.607203    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c707f08-4328-42c3-a2e4-b8231b9a0b16-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6c707f08-4328-42c3-a2e4-b8231b9a0b16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 22:58:17.708585 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:17.708493    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c707f08-4328-42c3-a2e4-b8231b9a0b16-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6c707f08-4328-42c3-a2e4-b8231b9a0b16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 22:58:17.708585 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:17.708569    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5lfw\" (UniqueName: \"kubernetes.io/projected/6c707f08-4328-42c3-a2e4-b8231b9a0b16-kube-api-access-v5lfw\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6c707f08-4328-42c3-a2e4-b8231b9a0b16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 22:58:17.708816 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:17.708607    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c707f08-4328-42c3-a2e4-b8231b9a0b16-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6c707f08-4328-42c3-a2e4-b8231b9a0b16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 22:58:17.708816 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:17.708630    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c707f08-4328-42c3-a2e4-b8231b9a0b16-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6c707f08-4328-42c3-a2e4-b8231b9a0b16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 22:58:17.708962 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:17.708802    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c707f08-4328-42c3-a2e4-b8231b9a0b16-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6c707f08-4328-42c3-a2e4-b8231b9a0b16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 22:58:17.708962 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:17.708860    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c707f08-4328-42c3-a2e4-b8231b9a0b16-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6c707f08-4328-42c3-a2e4-b8231b9a0b16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 22:58:17.708962 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:17.708887    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c707f08-4328-42c3-a2e4-b8231b9a0b16-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6c707f08-4328-42c3-a2e4-b8231b9a0b16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 22:58:17.708962 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:17.708956    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c707f08-4328-42c3-a2e4-b8231b9a0b16-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6c707f08-4328-42c3-a2e4-b8231b9a0b16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 22:58:17.709169 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:17.708980    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c707f08-4328-42c3-a2e4-b8231b9a0b16-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6c707f08-4328-42c3-a2e4-b8231b9a0b16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 22:58:17.709169 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:17.709058    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c707f08-4328-42c3-a2e4-b8231b9a0b16-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6c707f08-4328-42c3-a2e4-b8231b9a0b16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 22:58:17.709321 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:17.709286    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c707f08-4328-42c3-a2e4-b8231b9a0b16-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6c707f08-4328-42c3-a2e4-b8231b9a0b16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 22:58:17.709727 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:17.709702    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c707f08-4328-42c3-a2e4-b8231b9a0b16-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6c707f08-4328-42c3-a2e4-b8231b9a0b16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 22:58:17.709872 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:17.709761    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c707f08-4328-42c3-a2e4-b8231b9a0b16-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6c707f08-4328-42c3-a2e4-b8231b9a0b16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 22:58:17.711123 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:17.711092    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c707f08-4328-42c3-a2e4-b8231b9a0b16-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6c707f08-4328-42c3-a2e4-b8231b9a0b16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 22:58:17.711395 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:17.711379    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c707f08-4328-42c3-a2e4-b8231b9a0b16-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6c707f08-4328-42c3-a2e4-b8231b9a0b16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 22:58:17.716207 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:17.716169    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5lfw\" (UniqueName: \"kubernetes.io/projected/6c707f08-4328-42c3-a2e4-b8231b9a0b16-kube-api-access-v5lfw\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6c707f08-4328-42c3-a2e4-b8231b9a0b16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 22:58:17.761293 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:17.761265    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 22:58:17.867212 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:17.867156    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q" podUID="a882a1c3-a875-4eb0-a09f-c32cee1b4fbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": dial tcp 10.132.0.90:8000: connect: connection refused"
Sep 17 22:58:18.042061 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:18.041924    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 22:58:18.042253 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:58:18.042071    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c707f08_4328_42c3_a2e4_b8231b9a0b16.slice/crio-cc82ea148e532c44f1903fd8b6854880eb6e63b776497b42acadfc58861c33e0 WatchSource:0}: Error finding container cc82ea148e532c44f1903fd8b6854880eb6e63b776497b42acadfc58861c33e0: Status 404 returned error can't find the container with id cc82ea148e532c44f1903fd8b6854880eb6e63b776497b42acadfc58861c33e0
Sep 17 22:58:18.909706 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:18.909662    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"6c707f08-4328-42c3-a2e4-b8231b9a0b16","Type":"ContainerStarted","Data":"779174cea4d043d70aa37e272a2da65daacd08c1bff92b66b516541fd25bdbf3"}
Sep 17 22:58:18.910312 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:18.910276    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"6c707f08-4328-42c3-a2e4-b8231b9a0b16","Type":"ContainerStarted","Data":"cc82ea148e532c44f1903fd8b6854880eb6e63b776497b42acadfc58861c33e0"}
Sep 17 22:58:20.926159 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:20.926121    2555 generic.go:356] "Generic (PLEG): container finished" podID="6c707f08-4328-42c3-a2e4-b8231b9a0b16" containerID="779174cea4d043d70aa37e272a2da65daacd08c1bff92b66b516541fd25bdbf3" exitCode=0
Sep 17 22:58:20.926591 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:20.926217    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"6c707f08-4328-42c3-a2e4-b8231b9a0b16","Type":"ContainerDied","Data":"779174cea4d043d70aa37e272a2da65daacd08c1bff92b66b516541fd25bdbf3"}
Sep 17 22:58:21.358517 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:21.358409    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:58:21.934362 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:21.934314    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"6c707f08-4328-42c3-a2e4-b8231b9a0b16","Type":"ContainerStarted","Data":"4092b0a58a2542a180d98b1b868675b6b8d48eb1f40c0b052718b115fe98d2e6"}
Sep 17 22:58:22.762151 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:22.762117    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 22:58:23.001809 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:23.001782    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-7njvj_a324c9ab-5d42-4d31-8ff7-449f619be774/console-operator/1.log"
Sep 17 22:58:23.006678 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:23.006656    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s7f5_4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf/ovn-acl-logging/0.log"
Sep 17 22:58:23.018410 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:23.018386    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-7njvj_a324c9ab-5d42-4d31-8ff7-449f619be774/console-operator/1.log"
Sep 17 22:58:23.022914 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:23.022896    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s7f5_4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf/ovn-acl-logging/0.log"
Sep 17 22:58:26.880344 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:26.880313    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:58:27.761709 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:27.761660    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 22:58:27.763368 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:27.763332    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="6c707f08-4328-42c3-a2e4-b8231b9a0b16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": dial tcp 10.132.0.91:8000: connect: connection refused"
Sep 17 22:58:27.867090 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:27.867033    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q" podUID="a882a1c3-a875-4eb0-a09f-c32cee1b4fbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": dial tcp 10.132.0.90:8000: connect: connection refused"
Sep 17 22:58:28.274697 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:28.274650    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:58:37.762482 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:37.762422    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="6c707f08-4328-42c3-a2e4-b8231b9a0b16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": dial tcp 10.132.0.91:8000: connect: connection refused"
Sep 17 22:58:37.867110 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:37.867063    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q" podUID="a882a1c3-a875-4eb0-a09f-c32cee1b4fbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": dial tcp 10.132.0.90:8000: connect: connection refused"
Sep 17 22:58:47.762304 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:47.762258    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="6c707f08-4328-42c3-a2e4-b8231b9a0b16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": dial tcp 10.132.0.91:8000: connect: connection refused"
Sep 17 22:58:47.877397 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:47.877351    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q"
Sep 17 22:58:47.885306 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:47.885279    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q"
Sep 17 22:58:47.900530 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:47.900484    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=28.52091023 podStartE2EDuration="30.900464846s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.379554616s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:58:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:58:21.953373376 +0000 UTC m=+3603.399418649" watchObservedRunningTime="2026-09-17 22:58:47.900464846 +0000 UTC m=+3629.346510119"
Sep 17 22:58:52.677117 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:52.677091    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:58:52.685208 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:52.685145    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q"]
Sep 17 22:58:52.735233 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:52.735197    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:58:57.762735 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:58:57.762690    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="6c707f08-4328-42c3-a2e4-b8231b9a0b16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": dial tcp 10.132.0.91:8000: connect: connection refused"
Sep 17 22:59:03.698552 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:03.698520    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:59:07.709354 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:07.709309    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q" podUID="a882a1c3-a875-4eb0-a09f-c32cee1b4fbb" containerName="main" containerID="cri-o://09cd0f092ccf75ba8f42a5fcf349bed08c8ddd173d230bbeffcd8c3934f29efd" gracePeriod=45
Sep 17 22:59:07.762472 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:07.762415    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="6c707f08-4328-42c3-a2e4-b8231b9a0b16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": dial tcp 10.132.0.91:8000: connect: connection refused"
Sep 17 22:59:07.879038 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:07.878994    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q" podUID="a882a1c3-a875-4eb0-a09f-c32cee1b4fbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": dial tcp 10.132.0.90:8000: connect: connection refused"
Sep 17 22:59:08.471715 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:08.471688    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q"
Sep 17 22:59:08.491468 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:08.491440    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-cabundle-cert\") pod \"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb\" (UID: \"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb\") "
Sep 17 22:59:08.491587 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:08.491480    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-tmp-dir\") pod \"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb\" (UID: \"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb\") "
Sep 17 22:59:08.491587 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:08.491514    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pb69x\" (UniqueName: \"kubernetes.io/projected/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-kube-api-access-pb69x\") pod \"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb\" (UID: \"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb\") "
Sep 17 22:59:08.491587 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:08.491539    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-dshm\") pod \"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb\" (UID: \"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb\") "
Sep 17 22:59:08.491587 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:08.491567    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-model-cache\") pod \"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb\" (UID: \"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb\") "
Sep 17 22:59:08.491797 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:08.491599    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-home\") pod \"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb\" (UID: \"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb\") "
Sep 17 22:59:08.491797 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:08.491633    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-kserve-provision-location\") pod \"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb\" (UID: \"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb\") "
Sep 17 22:59:08.491797 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:08.491683    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-tls-certs\") pod \"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb\" (UID: \"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb\") "
Sep 17 22:59:08.491972 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:08.491940    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-model-cache" (OuterVolumeSpecName: "model-cache") pod "a882a1c3-a875-4eb0-a09f-c32cee1b4fbb" (UID: "a882a1c3-a875-4eb0-a09f-c32cee1b4fbb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:59:08.494046 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:08.492072    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-model-cache\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:59:08.494046 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:08.492507    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-home" (OuterVolumeSpecName: "home") pod "a882a1c3-a875-4eb0-a09f-c32cee1b4fbb" (UID: "a882a1c3-a875-4eb0-a09f-c32cee1b4fbb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:59:08.500146 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:08.496358    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-kube-api-access-pb69x" (OuterVolumeSpecName: "kube-api-access-pb69x") pod "a882a1c3-a875-4eb0-a09f-c32cee1b4fbb" (UID: "a882a1c3-a875-4eb0-a09f-c32cee1b4fbb"). InnerVolumeSpecName "kube-api-access-pb69x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:59:08.500146 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:08.497365    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a882a1c3-a875-4eb0-a09f-c32cee1b4fbb" (UID: "a882a1c3-a875-4eb0-a09f-c32cee1b4fbb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:59:08.500146 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:08.498055    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a882a1c3-a875-4eb0-a09f-c32cee1b4fbb" (UID: "a882a1c3-a875-4eb0-a09f-c32cee1b4fbb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:59:08.504572 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:08.500526    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-dshm" (OuterVolumeSpecName: "dshm") pod "a882a1c3-a875-4eb0-a09f-c32cee1b4fbb" (UID: "a882a1c3-a875-4eb0-a09f-c32cee1b4fbb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:59:08.510420 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:08.510364    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a882a1c3-a875-4eb0-a09f-c32cee1b4fbb" (UID: "a882a1c3-a875-4eb0-a09f-c32cee1b4fbb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:59:08.533151 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:08.533119    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a882a1c3-a875-4eb0-a09f-c32cee1b4fbb" (UID: "a882a1c3-a875-4eb0-a09f-c32cee1b4fbb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:59:08.592809 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:08.592765    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-dshm\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:59:08.592809 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:08.592809    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-home\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:59:08.592992 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:08.592825    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:59:08.592992 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:08.592841    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-tls-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:59:08.592992 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:08.592856    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:59:08.592992 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:08.592870    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-tmp-dir\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:59:08.592992 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:08.592883    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb69x\" (UniqueName: \"kubernetes.io/projected/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb-kube-api-access-pb69x\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 22:59:09.241852 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:09.241812    2555 generic.go:356] "Generic (PLEG): container finished" podID="a882a1c3-a875-4eb0-a09f-c32cee1b4fbb" containerID="09cd0f092ccf75ba8f42a5fcf349bed08c8ddd173d230bbeffcd8c3934f29efd" exitCode=0
Sep 17 22:59:09.242467 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:09.241885    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q"
Sep 17 22:59:09.242467 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:09.241898    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q" event={"ID":"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb","Type":"ContainerDied","Data":"09cd0f092ccf75ba8f42a5fcf349bed08c8ddd173d230bbeffcd8c3934f29efd"}
Sep 17 22:59:09.242467 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:09.241966    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q" event={"ID":"a882a1c3-a875-4eb0-a09f-c32cee1b4fbb","Type":"ContainerDied","Data":"3f5615e76e1e6f0b009a351af994427ed2198f7487263e6bfb2af6cec3577df4"}
Sep 17 22:59:09.242467 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:09.242003    2555 scope.go:117] "RemoveContainer" containerID="09cd0f092ccf75ba8f42a5fcf349bed08c8ddd173d230bbeffcd8c3934f29efd"
Sep 17 22:59:09.251448 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:09.251428    2555 scope.go:117] "RemoveContainer" containerID="c4805edc5d2bb9d24be81acb8bcad67a0b324b206388a0501017294b56036d4c"
Sep 17 22:59:09.260672 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:09.260648    2555 scope.go:117] "RemoveContainer" containerID="09cd0f092ccf75ba8f42a5fcf349bed08c8ddd173d230bbeffcd8c3934f29efd"
Sep 17 22:59:09.260951 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:59:09.260929    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09cd0f092ccf75ba8f42a5fcf349bed08c8ddd173d230bbeffcd8c3934f29efd\": container with ID starting with 09cd0f092ccf75ba8f42a5fcf349bed08c8ddd173d230bbeffcd8c3934f29efd not found: ID does not exist" containerID="09cd0f092ccf75ba8f42a5fcf349bed08c8ddd173d230bbeffcd8c3934f29efd"
Sep 17 22:59:09.261038 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:09.260963    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09cd0f092ccf75ba8f42a5fcf349bed08c8ddd173d230bbeffcd8c3934f29efd"} err="failed to get container status \"09cd0f092ccf75ba8f42a5fcf349bed08c8ddd173d230bbeffcd8c3934f29efd\": rpc error: code = NotFound desc = could not find container \"09cd0f092ccf75ba8f42a5fcf349bed08c8ddd173d230bbeffcd8c3934f29efd\": container with ID starting with 09cd0f092ccf75ba8f42a5fcf349bed08c8ddd173d230bbeffcd8c3934f29efd not found: ID does not exist"
Sep 17 22:59:09.261038 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:09.260986    2555 scope.go:117] "RemoveContainer" containerID="c4805edc5d2bb9d24be81acb8bcad67a0b324b206388a0501017294b56036d4c"
Sep 17 22:59:09.261361 ip-10-0-140-157 kubenswrapper[2555]: E0917 22:59:09.261335    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4805edc5d2bb9d24be81acb8bcad67a0b324b206388a0501017294b56036d4c\": container with ID starting with c4805edc5d2bb9d24be81acb8bcad67a0b324b206388a0501017294b56036d4c not found: ID does not exist" containerID="c4805edc5d2bb9d24be81acb8bcad67a0b324b206388a0501017294b56036d4c"
Sep 17 22:59:09.261428 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:09.261369    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4805edc5d2bb9d24be81acb8bcad67a0b324b206388a0501017294b56036d4c"} err="failed to get container status \"c4805edc5d2bb9d24be81acb8bcad67a0b324b206388a0501017294b56036d4c\": rpc error: code = NotFound desc = could not find container \"c4805edc5d2bb9d24be81acb8bcad67a0b324b206388a0501017294b56036d4c\": container with ID starting with c4805edc5d2bb9d24be81acb8bcad67a0b324b206388a0501017294b56036d4c not found: ID does not exist"
Sep 17 22:59:09.262359 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:09.262336    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q"]
Sep 17 22:59:09.267741 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:09.267710    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5c798c6c78-q6v7q"]
Sep 17 22:59:10.680229 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:10.680175    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a882a1c3-a875-4eb0-a09f-c32cee1b4fbb" path="/var/lib/kubelet/pods/a882a1c3-a875-4eb0-a09f-c32cee1b4fbb/volumes"
Sep 17 22:59:17.762022 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:17.761973    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="6c707f08-4328-42c3-a2e4-b8231b9a0b16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": dial tcp 10.132.0.91:8000: connect: connection refused"
Sep 17 22:59:27.762483 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:27.762429    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="6c707f08-4328-42c3-a2e4-b8231b9a0b16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": dial tcp 10.132.0.91:8000: connect: connection refused"
Sep 17 22:59:37.761658 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:37.761575    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="6c707f08-4328-42c3-a2e4-b8231b9a0b16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": dial tcp 10.132.0.91:8000: connect: connection refused"
Sep 17 22:59:42.819740 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:42.819700    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:59:42.904867 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:42.904835    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:59:47.762001 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:47.761958    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="6c707f08-4328-42c3-a2e4-b8231b9a0b16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": dial tcp 10.132.0.91:8000: connect: connection refused"
Sep 17 22:59:49.206823 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:49.206788    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:59:52.959949 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:52.959909    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6"]
Sep 17 22:59:52.960472 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:52.960314    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a882a1c3-a875-4eb0-a09f-c32cee1b4fbb" containerName="storage-initializer"
Sep 17 22:59:52.960472 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:52.960329    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a882a1c3-a875-4eb0-a09f-c32cee1b4fbb" containerName="storage-initializer"
Sep 17 22:59:52.960472 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:52.960344    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a882a1c3-a875-4eb0-a09f-c32cee1b4fbb" containerName="main"
Sep 17 22:59:52.960472 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:52.960349    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a882a1c3-a875-4eb0-a09f-c32cee1b4fbb" containerName="main"
Sep 17 22:59:52.960472 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:52.960419    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a882a1c3-a875-4eb0-a09f-c32cee1b4fbb" containerName="main"
Sep 17 22:59:52.966609 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:52.966585    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6"
Sep 17 22:59:52.969113 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:52.969082    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-fg582\""
Sep 17 22:59:52.969113 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:52.969098    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:59:52.969332 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:52.969129    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 17 22:59:52.969332 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:52.969082    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 17 22:59:52.969332 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:52.969295    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 17 22:59:52.969458 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:52.969403    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 17 22:59:52.974705 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:52.974680    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6"]
Sep 17 22:59:53.001643 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:53.001608    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-kserve-provision-location\") pod \"prestop-hook-test-kserve-5bcc844858-6bzf6\" (UID: \"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6"
Sep 17 22:59:53.001821 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:53.001658    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-dshm\") pod \"prestop-hook-test-kserve-5bcc844858-6bzf6\" (UID: \"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6"
Sep 17 22:59:53.001821 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:53.001695    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-model-cache\") pod \"prestop-hook-test-kserve-5bcc844858-6bzf6\" (UID: \"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6"
Sep 17 22:59:53.001821 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:53.001745    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-cabundle-cert\") pod \"prestop-hook-test-kserve-5bcc844858-6bzf6\" (UID: \"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6"
Sep 17 22:59:53.001821 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:53.001770    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf5lh\" (UniqueName: \"kubernetes.io/projected/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-kube-api-access-qf5lh\") pod \"prestop-hook-test-kserve-5bcc844858-6bzf6\" (UID: \"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6"
Sep 17 22:59:53.002045 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:53.001845    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-tmp-dir\") pod \"prestop-hook-test-kserve-5bcc844858-6bzf6\" (UID: \"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6"
Sep 17 22:59:53.002045 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:53.001904    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-home\") pod \"prestop-hook-test-kserve-5bcc844858-6bzf6\" (UID: \"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6"
Sep 17 22:59:53.002045 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:53.001951    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-tls-certs\") pod \"prestop-hook-test-kserve-5bcc844858-6bzf6\" (UID: \"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6"
Sep 17 22:59:53.038523 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:53.038482    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:59:53.102662 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:53.102631    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-kserve-provision-location\") pod \"prestop-hook-test-kserve-5bcc844858-6bzf6\" (UID: \"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6"
Sep 17 22:59:53.102847 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:53.102671    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-dshm\") pod \"prestop-hook-test-kserve-5bcc844858-6bzf6\" (UID: \"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6"
Sep 17 22:59:53.102847 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:53.102795    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-model-cache\") pod \"prestop-hook-test-kserve-5bcc844858-6bzf6\" (UID: \"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6"
Sep 17 22:59:53.102847 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:53.102838    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-cabundle-cert\") pod \"prestop-hook-test-kserve-5bcc844858-6bzf6\" (UID: \"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6"
Sep 17 22:59:53.103041 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:53.102857    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf5lh\" (UniqueName: \"kubernetes.io/projected/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-kube-api-access-qf5lh\") pod \"prestop-hook-test-kserve-5bcc844858-6bzf6\" (UID: \"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6"
Sep 17 22:59:53.103102 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:53.103088    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-kserve-provision-location\") pod \"prestop-hook-test-kserve-5bcc844858-6bzf6\" (UID: \"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6"
Sep 17 22:59:53.103238 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:53.103175    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-model-cache\") pod \"prestop-hook-test-kserve-5bcc844858-6bzf6\" (UID: \"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6"
Sep 17 22:59:53.103238 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:53.103204    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-tmp-dir\") pod \"prestop-hook-test-kserve-5bcc844858-6bzf6\" (UID: \"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6"
Sep 17 22:59:53.103437 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:53.103295    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-home\") pod \"prestop-hook-test-kserve-5bcc844858-6bzf6\" (UID: \"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6"
Sep 17 22:59:53.103437 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:53.103330    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-tls-certs\") pod \"prestop-hook-test-kserve-5bcc844858-6bzf6\" (UID: \"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6"
Sep 17 22:59:53.103555 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:53.103505    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-tmp-dir\") pod \"prestop-hook-test-kserve-5bcc844858-6bzf6\" (UID: \"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6"
Sep 17 22:59:53.103672 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:53.103646    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-cabundle-cert\") pod \"prestop-hook-test-kserve-5bcc844858-6bzf6\" (UID: \"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6"
Sep 17 22:59:53.103837 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:53.103761    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-home\") pod \"prestop-hook-test-kserve-5bcc844858-6bzf6\" (UID: \"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6"
Sep 17 22:59:53.105193 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:53.105156    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-dshm\") pod \"prestop-hook-test-kserve-5bcc844858-6bzf6\" (UID: \"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6"
Sep 17 22:59:53.105863 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:53.105847    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-tls-certs\") pod \"prestop-hook-test-kserve-5bcc844858-6bzf6\" (UID: \"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6"
Sep 17 22:59:53.112098 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:53.112072    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf5lh\" (UniqueName: \"kubernetes.io/projected/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-kube-api-access-qf5lh\") pod \"prestop-hook-test-kserve-5bcc844858-6bzf6\" (UID: \"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6"
Sep 17 22:59:53.241078 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:53.240990    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:59:53.280930 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:53.280903    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6"
Sep 17 22:59:53.569068 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:53.569035    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6"]
Sep 17 22:59:53.570296 ip-10-0-140-157 kubenswrapper[2555]: W0917 22:59:53.570266    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f2efb8_8606_4bcd_9ac4_bea6ad87b402.slice/crio-a161837a0495d117783d6cc4d0c45cf55d52464dadb8d9b2639f0743322bff57 WatchSource:0}: Error finding container a161837a0495d117783d6cc4d0c45cf55d52464dadb8d9b2639f0743322bff57: Status 404 returned error can't find the container with id a161837a0495d117783d6cc4d0c45cf55d52464dadb8d9b2639f0743322bff57
Sep 17 22:59:54.515674 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:54.515634    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6" event={"ID":"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402","Type":"ContainerStarted","Data":"7e755485cc29fa338edf8bece4f8c04e15997849c39cad8ea3646c816472050f"}
Sep 17 22:59:54.516197 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:54.515702    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6" event={"ID":"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402","Type":"ContainerStarted","Data":"a161837a0495d117783d6cc4d0c45cf55d52464dadb8d9b2639f0743322bff57"}
Sep 17 22:59:55.169024 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:55.168993    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:59:55.269022 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:55.268987    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 22:59:56.530502 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:56.530470    2555 generic.go:356] "Generic (PLEG): container finished" podID="c6f2efb8-8606-4bcd-9ac4-bea6ad87b402" containerID="7e755485cc29fa338edf8bece4f8c04e15997849c39cad8ea3646c816472050f" exitCode=0
Sep 17 22:59:56.530993 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:56.530552    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6" event={"ID":"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402","Type":"ContainerDied","Data":"7e755485cc29fa338edf8bece4f8c04e15997849c39cad8ea3646c816472050f"}
Sep 17 22:59:57.538385 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:57.538350    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6" event={"ID":"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402","Type":"ContainerStarted","Data":"11d086999c07d21d0386a7c2a6d4b4c8fc9116abe923974ef412f44a8ae60c6f"}
Sep 17 22:59:57.572514 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:57.572461    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6" podStartSLOduration=3.248738733 podStartE2EDuration="5.57244549s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.323706757s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:59:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:59:57.569570305 +0000 UTC m=+3699.015615578" watchObservedRunningTime="2026-09-17 22:59:57.57244549 +0000 UTC m=+3699.018490763"
Sep 17 22:59:57.772436 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:57.772394    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 22:59:57.780839 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:57.780795    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 22:59:58.282007 ip-10-0-140-157 kubenswrapper[2555]: I0917 22:59:58.281960    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6"
Sep 17 23:00:03.281589 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:03.281548    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6"
Sep 17 23:00:03.283097 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:03.283063    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6" podUID="c6f2efb8-8606-4bcd-9ac4-bea6ad87b402" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": dial tcp 10.132.0.92:8000: connect: connection refused"
Sep 17 23:00:08.420766 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:08.420733    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:00:13.281599 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:13.281544    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6" podUID="c6f2efb8-8606-4bcd-9ac4-bea6ad87b402" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": dial tcp 10.132.0.92:8000: connect: connection refused"
Sep 17 23:00:16.159400 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:16.159357    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:00:16.185572 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:16.185539    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 23:00:16.328485 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:16.328452    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:00:23.164787 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:23.164749    2555 scope.go:117] "RemoveContainer" containerID="4a9e7bbbed0675af6d36a549fabcacce695f1beee5aa9b1e8443ac1bc9db7427"
Sep 17 23:00:23.281658 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:23.281617    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6" podUID="c6f2efb8-8606-4bcd-9ac4-bea6ad87b402" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": dial tcp 10.132.0.92:8000: connect: connection refused"
Sep 17 23:00:31.208057 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:31.208006    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="6c707f08-4328-42c3-a2e4-b8231b9a0b16" containerName="main" containerID="cri-o://4092b0a58a2542a180d98b1b868675b6b8d48eb1f40c0b052718b115fe98d2e6" gracePeriod=45
Sep 17 23:00:31.773505 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:31.773466    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="6c707f08-4328-42c3-a2e4-b8231b9a0b16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": dial tcp 10.132.0.91:8000: connect: connection refused"
Sep 17 23:00:31.972427 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:31.972402    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 23:00:32.068474 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:32.068380    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c707f08-4328-42c3-a2e4-b8231b9a0b16-tls-certs\") pod \"6c707f08-4328-42c3-a2e4-b8231b9a0b16\" (UID: \"6c707f08-4328-42c3-a2e4-b8231b9a0b16\") "
Sep 17 23:00:32.068474 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:32.068436    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c707f08-4328-42c3-a2e4-b8231b9a0b16-cabundle-cert\") pod \"6c707f08-4328-42c3-a2e4-b8231b9a0b16\" (UID: \"6c707f08-4328-42c3-a2e4-b8231b9a0b16\") "
Sep 17 23:00:32.068715 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:32.068498    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c707f08-4328-42c3-a2e4-b8231b9a0b16-kserve-provision-location\") pod \"6c707f08-4328-42c3-a2e4-b8231b9a0b16\" (UID: \"6c707f08-4328-42c3-a2e4-b8231b9a0b16\") "
Sep 17 23:00:32.068715 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:32.068581    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c707f08-4328-42c3-a2e4-b8231b9a0b16-model-cache\") pod \"6c707f08-4328-42c3-a2e4-b8231b9a0b16\" (UID: \"6c707f08-4328-42c3-a2e4-b8231b9a0b16\") "
Sep 17 23:00:32.068715 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:32.068611    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5lfw\" (UniqueName: \"kubernetes.io/projected/6c707f08-4328-42c3-a2e4-b8231b9a0b16-kube-api-access-v5lfw\") pod \"6c707f08-4328-42c3-a2e4-b8231b9a0b16\" (UID: \"6c707f08-4328-42c3-a2e4-b8231b9a0b16\") "
Sep 17 23:00:32.068715 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:32.068650    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c707f08-4328-42c3-a2e4-b8231b9a0b16-dshm\") pod \"6c707f08-4328-42c3-a2e4-b8231b9a0b16\" (UID: \"6c707f08-4328-42c3-a2e4-b8231b9a0b16\") "
Sep 17 23:00:32.068715 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:32.068684    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c707f08-4328-42c3-a2e4-b8231b9a0b16-home\") pod \"6c707f08-4328-42c3-a2e4-b8231b9a0b16\" (UID: \"6c707f08-4328-42c3-a2e4-b8231b9a0b16\") "
Sep 17 23:00:32.069013 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:32.068744    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c707f08-4328-42c3-a2e4-b8231b9a0b16-tmp-dir\") pod \"6c707f08-4328-42c3-a2e4-b8231b9a0b16\" (UID: \"6c707f08-4328-42c3-a2e4-b8231b9a0b16\") "
Sep 17 23:00:32.069013 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:32.068825    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c707f08-4328-42c3-a2e4-b8231b9a0b16-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6c707f08-4328-42c3-a2e4-b8231b9a0b16" (UID: "6c707f08-4328-42c3-a2e4-b8231b9a0b16"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:00:32.069013 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:32.068834    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c707f08-4328-42c3-a2e4-b8231b9a0b16-model-cache" (OuterVolumeSpecName: "model-cache") pod "6c707f08-4328-42c3-a2e4-b8231b9a0b16" (UID: "6c707f08-4328-42c3-a2e4-b8231b9a0b16"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:00:32.069194 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:32.069087    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c707f08-4328-42c3-a2e4-b8231b9a0b16-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 23:00:32.069194 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:32.069107    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c707f08-4328-42c3-a2e4-b8231b9a0b16-model-cache\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 23:00:32.069569 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:32.069527    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c707f08-4328-42c3-a2e4-b8231b9a0b16-home" (OuterVolumeSpecName: "home") pod "6c707f08-4328-42c3-a2e4-b8231b9a0b16" (UID: "6c707f08-4328-42c3-a2e4-b8231b9a0b16"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:00:32.071229 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:32.071200    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c707f08-4328-42c3-a2e4-b8231b9a0b16-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6c707f08-4328-42c3-a2e4-b8231b9a0b16" (UID: "6c707f08-4328-42c3-a2e4-b8231b9a0b16"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:00:32.071371 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:32.071342    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c707f08-4328-42c3-a2e4-b8231b9a0b16-dshm" (OuterVolumeSpecName: "dshm") pod "6c707f08-4328-42c3-a2e4-b8231b9a0b16" (UID: "6c707f08-4328-42c3-a2e4-b8231b9a0b16"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:00:32.071740 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:32.071716    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c707f08-4328-42c3-a2e4-b8231b9a0b16-kube-api-access-v5lfw" (OuterVolumeSpecName: "kube-api-access-v5lfw") pod "6c707f08-4328-42c3-a2e4-b8231b9a0b16" (UID: "6c707f08-4328-42c3-a2e4-b8231b9a0b16"). InnerVolumeSpecName "kube-api-access-v5lfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:00:32.083026 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:32.082997    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c707f08-4328-42c3-a2e4-b8231b9a0b16-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6c707f08-4328-42c3-a2e4-b8231b9a0b16" (UID: "6c707f08-4328-42c3-a2e4-b8231b9a0b16"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:00:32.090562 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:32.090535    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c707f08-4328-42c3-a2e4-b8231b9a0b16-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6c707f08-4328-42c3-a2e4-b8231b9a0b16" (UID: "6c707f08-4328-42c3-a2e4-b8231b9a0b16"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:00:32.170580 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:32.170528    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c707f08-4328-42c3-a2e4-b8231b9a0b16-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 23:00:32.170580 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:32.170582    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5lfw\" (UniqueName: \"kubernetes.io/projected/6c707f08-4328-42c3-a2e4-b8231b9a0b16-kube-api-access-v5lfw\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 23:00:32.170859 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:32.170593    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c707f08-4328-42c3-a2e4-b8231b9a0b16-dshm\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 23:00:32.170859 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:32.170604    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c707f08-4328-42c3-a2e4-b8231b9a0b16-home\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 23:00:32.170859 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:32.170612    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c707f08-4328-42c3-a2e4-b8231b9a0b16-tmp-dir\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 23:00:32.170859 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:32.170621    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c707f08-4328-42c3-a2e4-b8231b9a0b16-tls-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 23:00:32.758436 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:32.758399    2555 generic.go:356] "Generic (PLEG): container finished" podID="6c707f08-4328-42c3-a2e4-b8231b9a0b16" containerID="4092b0a58a2542a180d98b1b868675b6b8d48eb1f40c0b052718b115fe98d2e6" exitCode=0
Sep 17 23:00:32.758911 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:32.758440    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 23:00:32.758911 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:32.758477    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"6c707f08-4328-42c3-a2e4-b8231b9a0b16","Type":"ContainerDied","Data":"4092b0a58a2542a180d98b1b868675b6b8d48eb1f40c0b052718b115fe98d2e6"}
Sep 17 23:00:32.758911 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:32.758529    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"6c707f08-4328-42c3-a2e4-b8231b9a0b16","Type":"ContainerDied","Data":"cc82ea148e532c44f1903fd8b6854880eb6e63b776497b42acadfc58861c33e0"}
Sep 17 23:00:32.758911 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:32.758553    2555 scope.go:117] "RemoveContainer" containerID="4092b0a58a2542a180d98b1b868675b6b8d48eb1f40c0b052718b115fe98d2e6"
Sep 17 23:00:32.770577 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:32.770551    2555 scope.go:117] "RemoveContainer" containerID="779174cea4d043d70aa37e272a2da65daacd08c1bff92b66b516541fd25bdbf3"
Sep 17 23:00:32.777213 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:32.777171    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 23:00:32.780952 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:32.780931    2555 scope.go:117] "RemoveContainer" containerID="4092b0a58a2542a180d98b1b868675b6b8d48eb1f40c0b052718b115fe98d2e6"
Sep 17 23:00:32.781292 ip-10-0-140-157 kubenswrapper[2555]: E0917 23:00:32.781270    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4092b0a58a2542a180d98b1b868675b6b8d48eb1f40c0b052718b115fe98d2e6\": container with ID starting with 4092b0a58a2542a180d98b1b868675b6b8d48eb1f40c0b052718b115fe98d2e6 not found: ID does not exist" containerID="4092b0a58a2542a180d98b1b868675b6b8d48eb1f40c0b052718b115fe98d2e6"
Sep 17 23:00:32.781407 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:32.781304    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4092b0a58a2542a180d98b1b868675b6b8d48eb1f40c0b052718b115fe98d2e6"} err="failed to get container status \"4092b0a58a2542a180d98b1b868675b6b8d48eb1f40c0b052718b115fe98d2e6\": rpc error: code = NotFound desc = could not find container \"4092b0a58a2542a180d98b1b868675b6b8d48eb1f40c0b052718b115fe98d2e6\": container with ID starting with 4092b0a58a2542a180d98b1b868675b6b8d48eb1f40c0b052718b115fe98d2e6 not found: ID does not exist"
Sep 17 23:00:32.781407 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:32.781325    2555 scope.go:117] "RemoveContainer" containerID="779174cea4d043d70aa37e272a2da65daacd08c1bff92b66b516541fd25bdbf3"
Sep 17 23:00:32.781643 ip-10-0-140-157 kubenswrapper[2555]: E0917 23:00:32.781619    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"779174cea4d043d70aa37e272a2da65daacd08c1bff92b66b516541fd25bdbf3\": container with ID starting with 779174cea4d043d70aa37e272a2da65daacd08c1bff92b66b516541fd25bdbf3 not found: ID does not exist" containerID="779174cea4d043d70aa37e272a2da65daacd08c1bff92b66b516541fd25bdbf3"
Sep 17 23:00:32.781763 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:32.781643    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"779174cea4d043d70aa37e272a2da65daacd08c1bff92b66b516541fd25bdbf3"} err="failed to get container status \"779174cea4d043d70aa37e272a2da65daacd08c1bff92b66b516541fd25bdbf3\": rpc error: code = NotFound desc = could not find container \"779174cea4d043d70aa37e272a2da65daacd08c1bff92b66b516541fd25bdbf3\": container with ID starting with 779174cea4d043d70aa37e272a2da65daacd08c1bff92b66b516541fd25bdbf3 not found: ID does not exist"
Sep 17 23:00:32.783093 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:32.783068    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 23:00:33.282003 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:33.281957    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6" podUID="c6f2efb8-8606-4bcd-9ac4-bea6ad87b402" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": dial tcp 10.132.0.92:8000: connect: connection refused"
Sep 17 23:00:34.679462 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:34.679429    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c707f08-4328-42c3-a2e4-b8231b9a0b16" path="/var/lib/kubelet/pods/6c707f08-4328-42c3-a2e4-b8231b9a0b16/volumes"
Sep 17 23:00:43.281879 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:43.281831    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6" podUID="c6f2efb8-8606-4bcd-9ac4-bea6ad87b402" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": dial tcp 10.132.0.92:8000: connect: connection refused"
Sep 17 23:00:53.281459 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:00:53.281399    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6" podUID="c6f2efb8-8606-4bcd-9ac4-bea6ad87b402" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": dial tcp 10.132.0.92:8000: connect: connection refused"
Sep 17 23:01:03.282434 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:01:03.282331    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6" podUID="c6f2efb8-8606-4bcd-9ac4-bea6ad87b402" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": dial tcp 10.132.0.92:8000: connect: connection refused"
Sep 17 23:01:06.180686 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:01:06.180653    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:01:06.280347 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:01:06.280314    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:01:12.060727 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:01:12.060689    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:01:13.281628 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:01:13.281570    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6" podUID="c6f2efb8-8606-4bcd-9ac4-bea6ad87b402" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": dial tcp 10.132.0.92:8000: connect: connection refused"
Sep 17 23:01:17.661676 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:01:17.661640    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:01:17.728584 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:01:17.728546    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:01:18.826460 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:01:18.826427    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:01:19.008379 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:01:19.008343    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:01:23.281502 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:01:23.281446    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6" podUID="c6f2efb8-8606-4bcd-9ac4-bea6ad87b402" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": dial tcp 10.132.0.92:8000: connect: connection refused"
Sep 17 23:01:26.271512 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:01:26.271480    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:01:33.282316 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:01:33.282271    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6" podUID="c6f2efb8-8606-4bcd-9ac4-bea6ad87b402" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": dial tcp 10.132.0.92:8000: connect: connection refused"
Sep 17 23:01:43.282289 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:01:43.282239    2555 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6" podUID="c6f2efb8-8606-4bcd-9ac4-bea6ad87b402" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": dial tcp 10.132.0.92:8000: connect: connection refused"
Sep 17 23:01:52.415877 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:01:52.415834    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:01:52.556442 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:01:52.556374    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:01:53.292260 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:01:53.292227    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6"
Sep 17 23:01:53.300330 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:01:53.300304    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6"
Sep 17 23:01:53.714036 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:01:53.713986    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6"]
Sep 17 23:02:10.284296 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:10.284233    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6" podUID="c6f2efb8-8606-4bcd-9ac4-bea6ad87b402" containerName="main" containerID="cri-o://11d086999c07d21d0386a7c2a6d4b4c8fc9116abe923974ef412f44a8ae60c6f" gracePeriod=45
Sep 17 23:02:11.024899 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:11.024875    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6"
Sep 17 23:02:11.070290 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:11.070198    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-tmp-dir\") pod \"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402\" (UID: \"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402\") "
Sep 17 23:02:11.070290 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:11.070249    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qf5lh\" (UniqueName: \"kubernetes.io/projected/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-kube-api-access-qf5lh\") pod \"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402\" (UID: \"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402\") "
Sep 17 23:02:11.070460 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:11.070333    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-model-cache\") pod \"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402\" (UID: \"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402\") "
Sep 17 23:02:11.070460 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:11.070443    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-tls-certs\") pod \"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402\" (UID: \"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402\") "
Sep 17 23:02:11.070532 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:11.070470    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-cabundle-cert\") pod \"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402\" (UID: \"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402\") "
Sep 17 23:02:11.070532 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:11.070501    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-dshm\") pod \"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402\" (UID: \"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402\") "
Sep 17 23:02:11.070603 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:11.070538    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-home\") pod \"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402\" (UID: \"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402\") "
Sep 17 23:02:11.070603 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:11.070596    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-kserve-provision-location\") pod \"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402\" (UID: \"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402\") "
Sep 17 23:02:11.071160 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:11.070900    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c6f2efb8-8606-4bcd-9ac4-bea6ad87b402" (UID: "c6f2efb8-8606-4bcd-9ac4-bea6ad87b402"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:02:11.071160 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:11.071108    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-model-cache" (OuterVolumeSpecName: "model-cache") pod "c6f2efb8-8606-4bcd-9ac4-bea6ad87b402" (UID: "c6f2efb8-8606-4bcd-9ac4-bea6ad87b402"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:02:11.072041 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:11.071969    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-home" (OuterVolumeSpecName: "home") pod "c6f2efb8-8606-4bcd-9ac4-bea6ad87b402" (UID: "c6f2efb8-8606-4bcd-9ac4-bea6ad87b402"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:02:11.074206 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:11.074144    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-dshm" (OuterVolumeSpecName: "dshm") pod "c6f2efb8-8606-4bcd-9ac4-bea6ad87b402" (UID: "c6f2efb8-8606-4bcd-9ac4-bea6ad87b402"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:02:11.074206 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:11.074153    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c6f2efb8-8606-4bcd-9ac4-bea6ad87b402" (UID: "c6f2efb8-8606-4bcd-9ac4-bea6ad87b402"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:02:11.074416 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:11.074253    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-kube-api-access-qf5lh" (OuterVolumeSpecName: "kube-api-access-qf5lh") pod "c6f2efb8-8606-4bcd-9ac4-bea6ad87b402" (UID: "c6f2efb8-8606-4bcd-9ac4-bea6ad87b402"). InnerVolumeSpecName "kube-api-access-qf5lh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:02:11.090946 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:11.090913    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c6f2efb8-8606-4bcd-9ac4-bea6ad87b402" (UID: "c6f2efb8-8606-4bcd-9ac4-bea6ad87b402"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:02:11.098126 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:11.098101    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c6f2efb8-8606-4bcd-9ac4-bea6ad87b402" (UID: "c6f2efb8-8606-4bcd-9ac4-bea6ad87b402"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:02:11.171805 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:11.171774    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf5lh\" (UniqueName: \"kubernetes.io/projected/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-kube-api-access-qf5lh\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 23:02:11.171805 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:11.171802    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-model-cache\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 23:02:11.171805 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:11.171814    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-tls-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 23:02:11.172023 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:11.171822    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 23:02:11.172023 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:11.171831    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-dshm\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 23:02:11.172023 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:11.171838    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-home\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 23:02:11.172023 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:11.171845    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 23:02:11.172023 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:11.171855    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402-tmp-dir\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 23:02:11.350413 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:11.350315    2555 generic.go:356] "Generic (PLEG): container finished" podID="c6f2efb8-8606-4bcd-9ac4-bea6ad87b402" containerID="11d086999c07d21d0386a7c2a6d4b4c8fc9116abe923974ef412f44a8ae60c6f" exitCode=0
Sep 17 23:02:11.350413 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:11.350382    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6"
Sep 17 23:02:11.350413 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:11.350390    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6" event={"ID":"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402","Type":"ContainerDied","Data":"11d086999c07d21d0386a7c2a6d4b4c8fc9116abe923974ef412f44a8ae60c6f"}
Sep 17 23:02:11.351012 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:11.350458    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6" event={"ID":"c6f2efb8-8606-4bcd-9ac4-bea6ad87b402","Type":"ContainerDied","Data":"a161837a0495d117783d6cc4d0c45cf55d52464dadb8d9b2639f0743322bff57"}
Sep 17 23:02:11.351012 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:11.350493    2555 scope.go:117] "RemoveContainer" containerID="11d086999c07d21d0386a7c2a6d4b4c8fc9116abe923974ef412f44a8ae60c6f"
Sep 17 23:02:11.362284 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:11.362224    2555 scope.go:117] "RemoveContainer" containerID="7e755485cc29fa338edf8bece4f8c04e15997849c39cad8ea3646c816472050f"
Sep 17 23:02:11.371579 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:11.371554    2555 scope.go:117] "RemoveContainer" containerID="11d086999c07d21d0386a7c2a6d4b4c8fc9116abe923974ef412f44a8ae60c6f"
Sep 17 23:02:11.371857 ip-10-0-140-157 kubenswrapper[2555]: E0917 23:02:11.371839    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11d086999c07d21d0386a7c2a6d4b4c8fc9116abe923974ef412f44a8ae60c6f\": container with ID starting with 11d086999c07d21d0386a7c2a6d4b4c8fc9116abe923974ef412f44a8ae60c6f not found: ID does not exist" containerID="11d086999c07d21d0386a7c2a6d4b4c8fc9116abe923974ef412f44a8ae60c6f"
Sep 17 23:02:11.371905 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:11.371864    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11d086999c07d21d0386a7c2a6d4b4c8fc9116abe923974ef412f44a8ae60c6f"} err="failed to get container status \"11d086999c07d21d0386a7c2a6d4b4c8fc9116abe923974ef412f44a8ae60c6f\": rpc error: code = NotFound desc = could not find container \"11d086999c07d21d0386a7c2a6d4b4c8fc9116abe923974ef412f44a8ae60c6f\": container with ID starting with 11d086999c07d21d0386a7c2a6d4b4c8fc9116abe923974ef412f44a8ae60c6f not found: ID does not exist"
Sep 17 23:02:11.371905 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:11.371878    2555 scope.go:117] "RemoveContainer" containerID="7e755485cc29fa338edf8bece4f8c04e15997849c39cad8ea3646c816472050f"
Sep 17 23:02:11.372152 ip-10-0-140-157 kubenswrapper[2555]: E0917 23:02:11.372133    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e755485cc29fa338edf8bece4f8c04e15997849c39cad8ea3646c816472050f\": container with ID starting with 7e755485cc29fa338edf8bece4f8c04e15997849c39cad8ea3646c816472050f not found: ID does not exist" containerID="7e755485cc29fa338edf8bece4f8c04e15997849c39cad8ea3646c816472050f"
Sep 17 23:02:11.372242 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:11.372157    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e755485cc29fa338edf8bece4f8c04e15997849c39cad8ea3646c816472050f"} err="failed to get container status \"7e755485cc29fa338edf8bece4f8c04e15997849c39cad8ea3646c816472050f\": rpc error: code = NotFound desc = could not find container \"7e755485cc29fa338edf8bece4f8c04e15997849c39cad8ea3646c816472050f\": container with ID starting with 7e755485cc29fa338edf8bece4f8c04e15997849c39cad8ea3646c816472050f not found: ID does not exist"
Sep 17 23:02:11.378846 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:11.378821    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6"]
Sep 17 23:02:11.381737 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:11.381715    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-6bzf6"]
Sep 17 23:02:12.084299 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:12.084266    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:02:12.177458 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:12.177421    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:02:12.679068 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:12.679022    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6f2efb8-8606-4bcd-9ac4-bea6ad87b402" path="/var/lib/kubelet/pods/c6f2efb8-8606-4bcd-9ac4-bea6ad87b402/volumes"
Sep 17 23:02:42.310439 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:42.310398    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:02:42.403844 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:42.403811    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:02:47.239766 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:47.239733    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q"]
Sep 17 23:02:47.240295 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:47.240115    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c707f08-4328-42c3-a2e4-b8231b9a0b16" containerName="main"
Sep 17 23:02:47.240295 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:47.240127    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c707f08-4328-42c3-a2e4-b8231b9a0b16" containerName="main"
Sep 17 23:02:47.240295 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:47.240141    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6f2efb8-8606-4bcd-9ac4-bea6ad87b402" containerName="main"
Sep 17 23:02:47.240295 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:47.240147    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6f2efb8-8606-4bcd-9ac4-bea6ad87b402" containerName="main"
Sep 17 23:02:47.240295 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:47.240160    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c707f08-4328-42c3-a2e4-b8231b9a0b16" containerName="storage-initializer"
Sep 17 23:02:47.240295 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:47.240166    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c707f08-4328-42c3-a2e4-b8231b9a0b16" containerName="storage-initializer"
Sep 17 23:02:47.240295 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:47.240198    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6f2efb8-8606-4bcd-9ac4-bea6ad87b402" containerName="storage-initializer"
Sep 17 23:02:47.240295 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:47.240207    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6f2efb8-8606-4bcd-9ac4-bea6ad87b402" containerName="storage-initializer"
Sep 17 23:02:47.240295 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:47.240274    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6f2efb8-8606-4bcd-9ac4-bea6ad87b402" containerName="main"
Sep 17 23:02:47.240295 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:47.240284    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c707f08-4328-42c3-a2e4-b8231b9a0b16" containerName="main"
Sep 17 23:02:47.243278 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:47.243259    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q"
Sep 17 23:02:47.246458 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:47.246434    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 17 23:02:47.246581 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:47.246464    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-nchkf\""
Sep 17 23:02:47.246581 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:47.246468    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 17 23:02:47.246581 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:47.246437    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 17 23:02:47.252319 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:47.252298    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q"]
Sep 17 23:02:47.327544 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:47.327514    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:02:47.377757 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:47.377726    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwgpj\" (UniqueName: \"kubernetes.io/projected/8e97b523-29fb-474c-b711-8cd75b974f28-kube-api-access-wwgpj\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q\" (UID: \"8e97b523-29fb-474c-b711-8cd75b974f28\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q"
Sep 17 23:02:47.377896 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:47.377762    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e97b523-29fb-474c-b711-8cd75b974f28-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q\" (UID: \"8e97b523-29fb-474c-b711-8cd75b974f28\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q"
Sep 17 23:02:47.377896 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:47.377793    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e97b523-29fb-474c-b711-8cd75b974f28-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q\" (UID: \"8e97b523-29fb-474c-b711-8cd75b974f28\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q"
Sep 17 23:02:47.377896 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:47.377847    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e97b523-29fb-474c-b711-8cd75b974f28-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q\" (UID: \"8e97b523-29fb-474c-b711-8cd75b974f28\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q"
Sep 17 23:02:47.378028 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:47.377916    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e97b523-29fb-474c-b711-8cd75b974f28-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q\" (UID: \"8e97b523-29fb-474c-b711-8cd75b974f28\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q"
Sep 17 23:02:47.378028 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:47.377987    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e97b523-29fb-474c-b711-8cd75b974f28-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q\" (UID: \"8e97b523-29fb-474c-b711-8cd75b974f28\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q"
Sep 17 23:02:47.479076 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:47.479036    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e97b523-29fb-474c-b711-8cd75b974f28-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q\" (UID: \"8e97b523-29fb-474c-b711-8cd75b974f28\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q"
Sep 17 23:02:47.479269 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:47.479101    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwgpj\" (UniqueName: \"kubernetes.io/projected/8e97b523-29fb-474c-b711-8cd75b974f28-kube-api-access-wwgpj\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q\" (UID: \"8e97b523-29fb-474c-b711-8cd75b974f28\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q"
Sep 17 23:02:47.479269 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:47.479133    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e97b523-29fb-474c-b711-8cd75b974f28-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q\" (UID: \"8e97b523-29fb-474c-b711-8cd75b974f28\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q"
Sep 17 23:02:47.479269 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:47.479160    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e97b523-29fb-474c-b711-8cd75b974f28-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q\" (UID: \"8e97b523-29fb-474c-b711-8cd75b974f28\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q"
Sep 17 23:02:47.479269 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:47.479222    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e97b523-29fb-474c-b711-8cd75b974f28-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q\" (UID: \"8e97b523-29fb-474c-b711-8cd75b974f28\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q"
Sep 17 23:02:47.479498 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:47.479279    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e97b523-29fb-474c-b711-8cd75b974f28-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q\" (UID: \"8e97b523-29fb-474c-b711-8cd75b974f28\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q"
Sep 17 23:02:47.479668 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:47.479641    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e97b523-29fb-474c-b711-8cd75b974f28-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q\" (UID: \"8e97b523-29fb-474c-b711-8cd75b974f28\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q"
Sep 17 23:02:47.479790 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:47.479672    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e97b523-29fb-474c-b711-8cd75b974f28-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q\" (UID: \"8e97b523-29fb-474c-b711-8cd75b974f28\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q"
Sep 17 23:02:47.479790 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:47.479722    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e97b523-29fb-474c-b711-8cd75b974f28-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q\" (UID: \"8e97b523-29fb-474c-b711-8cd75b974f28\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q"
Sep 17 23:02:47.481546 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:47.481527    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e97b523-29fb-474c-b711-8cd75b974f28-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q\" (UID: \"8e97b523-29fb-474c-b711-8cd75b974f28\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q"
Sep 17 23:02:47.481734 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:47.481717    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e97b523-29fb-474c-b711-8cd75b974f28-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q\" (UID: \"8e97b523-29fb-474c-b711-8cd75b974f28\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q"
Sep 17 23:02:47.488486 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:47.488464    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwgpj\" (UniqueName: \"kubernetes.io/projected/8e97b523-29fb-474c-b711-8cd75b974f28-kube-api-access-wwgpj\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q\" (UID: \"8e97b523-29fb-474c-b711-8cd75b974f28\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q"
Sep 17 23:02:47.530642 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:47.530575    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:02:47.554234 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:47.554209    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q"
Sep 17 23:02:47.830817 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:47.830788    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q"]
Sep 17 23:02:47.832420 ip-10-0-140-157 kubenswrapper[2555]: W0917 23:02:47.832392    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e97b523_29fb_474c_b711_8cd75b974f28.slice/crio-6b370a15fd409b5530b3a3551b819a93d084e80e39e6125021eca745d6aac0ad WatchSource:0}: Error finding container 6b370a15fd409b5530b3a3551b819a93d084e80e39e6125021eca745d6aac0ad: Status 404 returned error can't find the container with id 6b370a15fd409b5530b3a3551b819a93d084e80e39e6125021eca745d6aac0ad
Sep 17 23:02:48.560889 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:48.560851    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q" event={"ID":"8e97b523-29fb-474c-b711-8cd75b974f28","Type":"ContainerStarted","Data":"6b5bc8c4c06110e93bf82a6150e608edd2f6d6e115b02d6e9025fd5573757a6b"}
Sep 17 23:02:48.561312 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:48.560914    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q" event={"ID":"8e97b523-29fb-474c-b711-8cd75b974f28","Type":"ContainerStarted","Data":"6b370a15fd409b5530b3a3551b819a93d084e80e39e6125021eca745d6aac0ad"}
Sep 17 23:02:48.579193 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:48.579130    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q" podStartSLOduration=1.579116172 podStartE2EDuration="1.579116172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:02:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:02:48.577428284 +0000 UTC m=+3870.023473557" watchObservedRunningTime="2026-09-17 23:02:48.579116172 +0000 UTC m=+3870.025161443"
Sep 17 23:02:49.113365 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:49.113335    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:02:49.554805 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:49.554769    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q"
Sep 17 23:02:54.654968 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:54.654932    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:02:54.720063 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:54.720033    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:02:57.555251 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:57.555217    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q"
Sep 17 23:02:57.567536 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:57.567508    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q"
Sep 17 23:02:57.621426 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:57.621391    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q"
Sep 17 23:02:59.533878 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:02:59.533828    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:03:10.834168 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:10.834134    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:03:10.900843 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:10.900782    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:03:17.292420 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:17.292345    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:03:17.383050 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:17.383004    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:03:23.042349 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:23.042322    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-7njvj_a324c9ab-5d42-4d31-8ff7-449f619be774/console-operator/1.log"
Sep 17 23:03:23.053852 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:23.053827    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s7f5_4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf/ovn-acl-logging/0.log"
Sep 17 23:03:23.068907 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:23.068880    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-7njvj_a324c9ab-5d42-4d31-8ff7-449f619be774/console-operator/1.log"
Sep 17 23:03:23.072415 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:23.072398    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s7f5_4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf/ovn-acl-logging/0.log"
Sep 17 23:03:23.611907 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:23.611870    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:03:26.131647 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:26.131611    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tddtj"]
Sep 17 23:03:26.135472 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:26.135452    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tddtj"
Sep 17 23:03:26.138631 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:26.138600    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 17 23:03:26.138631 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:26.138612    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 17 23:03:26.138834 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:26.138646    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-d9rk5\""
Sep 17 23:03:26.138834 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:26.138764    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 17 23:03:26.145397 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:26.145373    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tddtj"]
Sep 17 23:03:26.213017 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:26.212988    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz94b\" (UniqueName: \"kubernetes.io/projected/e0161425-76ba-4d41-987b-b19991e1bf69-kube-api-access-wz94b\") pod \"rolling-upgrade-test-kserve-74b79d7f77-tddtj\" (UID: \"e0161425-76ba-4d41-987b-b19991e1bf69\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tddtj"
Sep 17 23:03:26.213216 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:26.213069    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0161425-76ba-4d41-987b-b19991e1bf69-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-tddtj\" (UID: \"e0161425-76ba-4d41-987b-b19991e1bf69\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tddtj"
Sep 17 23:03:26.213216 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:26.213143    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0161425-76ba-4d41-987b-b19991e1bf69-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-tddtj\" (UID: \"e0161425-76ba-4d41-987b-b19991e1bf69\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tddtj"
Sep 17 23:03:26.213346 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:26.213231    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0161425-76ba-4d41-987b-b19991e1bf69-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-tddtj\" (UID: \"e0161425-76ba-4d41-987b-b19991e1bf69\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tddtj"
Sep 17 23:03:26.213346 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:26.213284    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0161425-76ba-4d41-987b-b19991e1bf69-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-tddtj\" (UID: \"e0161425-76ba-4d41-987b-b19991e1bf69\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tddtj"
Sep 17 23:03:26.213346 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:26.213309    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0161425-76ba-4d41-987b-b19991e1bf69-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-tddtj\" (UID: \"e0161425-76ba-4d41-987b-b19991e1bf69\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tddtj"
Sep 17 23:03:26.225420 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:26.225380    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:03:26.314522 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:26.314487    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0161425-76ba-4d41-987b-b19991e1bf69-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-tddtj\" (UID: \"e0161425-76ba-4d41-987b-b19991e1bf69\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tddtj"
Sep 17 23:03:26.314675 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:26.314545    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0161425-76ba-4d41-987b-b19991e1bf69-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-tddtj\" (UID: \"e0161425-76ba-4d41-987b-b19991e1bf69\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tddtj"
Sep 17 23:03:26.314675 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:26.314580    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0161425-76ba-4d41-987b-b19991e1bf69-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-tddtj\" (UID: \"e0161425-76ba-4d41-987b-b19991e1bf69\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tddtj"
Sep 17 23:03:26.314675 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:26.314619    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0161425-76ba-4d41-987b-b19991e1bf69-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-tddtj\" (UID: \"e0161425-76ba-4d41-987b-b19991e1bf69\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tddtj"
Sep 17 23:03:26.314675 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:26.314640    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0161425-76ba-4d41-987b-b19991e1bf69-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-tddtj\" (UID: \"e0161425-76ba-4d41-987b-b19991e1bf69\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tddtj"
Sep 17 23:03:26.314912 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:26.314771    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz94b\" (UniqueName: \"kubernetes.io/projected/e0161425-76ba-4d41-987b-b19991e1bf69-kube-api-access-wz94b\") pod \"rolling-upgrade-test-kserve-74b79d7f77-tddtj\" (UID: \"e0161425-76ba-4d41-987b-b19991e1bf69\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tddtj"
Sep 17 23:03:26.314978 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:26.314926    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0161425-76ba-4d41-987b-b19991e1bf69-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-tddtj\" (UID: \"e0161425-76ba-4d41-987b-b19991e1bf69\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tddtj"
Sep 17 23:03:26.315045 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:26.314983    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0161425-76ba-4d41-987b-b19991e1bf69-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-tddtj\" (UID: \"e0161425-76ba-4d41-987b-b19991e1bf69\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tddtj"
Sep 17 23:03:26.315156 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:26.315139    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0161425-76ba-4d41-987b-b19991e1bf69-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-tddtj\" (UID: \"e0161425-76ba-4d41-987b-b19991e1bf69\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tddtj"
Sep 17 23:03:26.317036 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:26.317009    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0161425-76ba-4d41-987b-b19991e1bf69-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-tddtj\" (UID: \"e0161425-76ba-4d41-987b-b19991e1bf69\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tddtj"
Sep 17 23:03:26.317618 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:26.317598    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0161425-76ba-4d41-987b-b19991e1bf69-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-tddtj\" (UID: \"e0161425-76ba-4d41-987b-b19991e1bf69\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tddtj"
Sep 17 23:03:26.322456 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:26.322433    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz94b\" (UniqueName: \"kubernetes.io/projected/e0161425-76ba-4d41-987b-b19991e1bf69-kube-api-access-wz94b\") pod \"rolling-upgrade-test-kserve-74b79d7f77-tddtj\" (UID: \"e0161425-76ba-4d41-987b-b19991e1bf69\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tddtj"
Sep 17 23:03:26.421513 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:26.421430    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:03:26.446901 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:26.446870    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tddtj"
Sep 17 23:03:26.727733 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:26.727644    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tddtj"]
Sep 17 23:03:26.730723 ip-10-0-140-157 kubenswrapper[2555]: W0917 23:03:26.730695    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0161425_76ba_4d41_987b_b19991e1bf69.slice/crio-56b0f88ccfb61a890400d823a22109d7a1cb820f42ce859de498dc1427fc7f89 WatchSource:0}: Error finding container 56b0f88ccfb61a890400d823a22109d7a1cb820f42ce859de498dc1427fc7f89: Status 404 returned error can't find the container with id 56b0f88ccfb61a890400d823a22109d7a1cb820f42ce859de498dc1427fc7f89
Sep 17 23:03:26.768304 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:26.768276    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tddtj" event={"ID":"e0161425-76ba-4d41-987b-b19991e1bf69","Type":"ContainerStarted","Data":"56b0f88ccfb61a890400d823a22109d7a1cb820f42ce859de498dc1427fc7f89"}
Sep 17 23:03:27.776160 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:27.776125    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tddtj" event={"ID":"e0161425-76ba-4d41-987b-b19991e1bf69","Type":"ContainerStarted","Data":"83e0bb5defa40e770eb8f1cf6ca97b01fa1807b5839b857db9a13b682b67dd99"}
Sep 17 23:03:27.792041 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:27.791974    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tddtj" podStartSLOduration=1.7919604759999999 podStartE2EDuration="1.791960476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:03:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:03:27.791231986 +0000 UTC m=+3909.237277258" watchObservedRunningTime="2026-09-17 23:03:27.791960476 +0000 UTC m=+3909.238005748"
Sep 17 23:03:28.447000 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:28.446959    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tddtj"
Sep 17 23:03:29.248548 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:29.248502    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:03:30.805036 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:30.805000    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:03:34.557138 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:34.557104    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:03:36.447691 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:36.447657    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tddtj"
Sep 17 23:03:36.460657 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:36.460631    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tddtj"
Sep 17 23:03:36.836194 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:36.836095    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tddtj"
Sep 17 23:03:37.372160 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:37.372108    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:03:37.372715 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:37.372688    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q"]
Sep 17 23:03:37.382476 ip-10-0-140-157 kubenswrapper[2555]: E0917 23:03:37.382440    2555 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 23:03:37.382476 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:03:37.382476 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 23:03:37.382476 ip-10-0-140-157 kubenswrapper[2555]:  > containerID="6b5bc8c4c06110e93bf82a6150e608edd2f6d6e115b02d6e9025fd5573757a6b" cmd=["/bin/sleep","15"]
Sep 17 23:03:37.382694 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:37.382498    2555 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q" message="" err=<
Sep 17 23:03:37.382694 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:03:37.382694 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 23:03:37.382694 ip-10-0-140-157 kubenswrapper[2555]:  >
Sep 17 23:03:37.382694 ip-10-0-140-157 kubenswrapper[2555]: E0917 23:03:37.382515    2555 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 23:03:37.382694 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:03:37.382694 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 23:03:37.382694 ip-10-0-140-157 kubenswrapper[2555]:  > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q" podUID="8e97b523-29fb-474c-b711-8cd75b974f28" containerName="main" containerID="cri-o://6b5bc8c4c06110e93bf82a6150e608edd2f6d6e115b02d6e9025fd5573757a6b"
Sep 17 23:03:37.382694 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:37.382559    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q" podUID="8e97b523-29fb-474c-b711-8cd75b974f28" containerName="main" containerID="cri-o://6b5bc8c4c06110e93bf82a6150e608edd2f6d6e115b02d6e9025fd5573757a6b" gracePeriod=60
Sep 17 23:03:37.431825 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:37.431784    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:03:37.611103 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:37.611060    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q" podUID="8e97b523-29fb-474c-b711-8cd75b974f28" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": dial tcp 10.132.0.93:8000: connect: connection refused"
Sep 17 23:03:37.835639 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:37.835600    2555 generic.go:356] "Generic (PLEG): container finished" podID="8e97b523-29fb-474c-b711-8cd75b974f28" containerID="6b5bc8c4c06110e93bf82a6150e608edd2f6d6e115b02d6e9025fd5573757a6b" exitCode=0
Sep 17 23:03:37.835846 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:37.835651    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q" event={"ID":"8e97b523-29fb-474c-b711-8cd75b974f28","Type":"ContainerDied","Data":"6b5bc8c4c06110e93bf82a6150e608edd2f6d6e115b02d6e9025fd5573757a6b"}
Sep 17 23:03:38.154206 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:38.154157    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q"
Sep 17 23:03:38.320456 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:38.320423    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwgpj\" (UniqueName: \"kubernetes.io/projected/8e97b523-29fb-474c-b711-8cd75b974f28-kube-api-access-wwgpj\") pod \"8e97b523-29fb-474c-b711-8cd75b974f28\" (UID: \"8e97b523-29fb-474c-b711-8cd75b974f28\") "
Sep 17 23:03:38.320631 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:38.320487    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e97b523-29fb-474c-b711-8cd75b974f28-model-cache\") pod \"8e97b523-29fb-474c-b711-8cd75b974f28\" (UID: \"8e97b523-29fb-474c-b711-8cd75b974f28\") "
Sep 17 23:03:38.320631 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:38.320508    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e97b523-29fb-474c-b711-8cd75b974f28-home\") pod \"8e97b523-29fb-474c-b711-8cd75b974f28\" (UID: \"8e97b523-29fb-474c-b711-8cd75b974f28\") "
Sep 17 23:03:38.320631 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:38.320573    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e97b523-29fb-474c-b711-8cd75b974f28-tls-certs\") pod \"8e97b523-29fb-474c-b711-8cd75b974f28\" (UID: \"8e97b523-29fb-474c-b711-8cd75b974f28\") "
Sep 17 23:03:38.320631 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:38.320592    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e97b523-29fb-474c-b711-8cd75b974f28-tmp-dir\") pod \"8e97b523-29fb-474c-b711-8cd75b974f28\" (UID: \"8e97b523-29fb-474c-b711-8cd75b974f28\") "
Sep 17 23:03:38.320631 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:38.320620    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e97b523-29fb-474c-b711-8cd75b974f28-dshm\") pod \"8e97b523-29fb-474c-b711-8cd75b974f28\" (UID: \"8e97b523-29fb-474c-b711-8cd75b974f28\") "
Sep 17 23:03:38.320954 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:38.320775    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e97b523-29fb-474c-b711-8cd75b974f28-model-cache" (OuterVolumeSpecName: "model-cache") pod "8e97b523-29fb-474c-b711-8cd75b974f28" (UID: "8e97b523-29fb-474c-b711-8cd75b974f28"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:03:38.320954 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:38.320794    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e97b523-29fb-474c-b711-8cd75b974f28-home" (OuterVolumeSpecName: "home") pod "8e97b523-29fb-474c-b711-8cd75b974f28" (UID: "8e97b523-29fb-474c-b711-8cd75b974f28"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:03:38.320954 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:38.320900    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e97b523-29fb-474c-b711-8cd75b974f28-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8e97b523-29fb-474c-b711-8cd75b974f28" (UID: "8e97b523-29fb-474c-b711-8cd75b974f28"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:03:38.320954 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:38.320927    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e97b523-29fb-474c-b711-8cd75b974f28-model-cache\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 23:03:38.320954 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:38.320940    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e97b523-29fb-474c-b711-8cd75b974f28-home\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 23:03:38.322808 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:38.322776    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e97b523-29fb-474c-b711-8cd75b974f28-dshm" (OuterVolumeSpecName: "dshm") pod "8e97b523-29fb-474c-b711-8cd75b974f28" (UID: "8e97b523-29fb-474c-b711-8cd75b974f28"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:03:38.322917 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:38.322827    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e97b523-29fb-474c-b711-8cd75b974f28-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8e97b523-29fb-474c-b711-8cd75b974f28" (UID: "8e97b523-29fb-474c-b711-8cd75b974f28"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:03:38.322917 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:38.322851    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e97b523-29fb-474c-b711-8cd75b974f28-kube-api-access-wwgpj" (OuterVolumeSpecName: "kube-api-access-wwgpj") pod "8e97b523-29fb-474c-b711-8cd75b974f28" (UID: "8e97b523-29fb-474c-b711-8cd75b974f28"). InnerVolumeSpecName "kube-api-access-wwgpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:03:38.421920 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:38.421825    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e97b523-29fb-474c-b711-8cd75b974f28-dshm\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 23:03:38.421920 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:38.421866    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwgpj\" (UniqueName: \"kubernetes.io/projected/8e97b523-29fb-474c-b711-8cd75b974f28-kube-api-access-wwgpj\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 23:03:38.421920 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:38.421881    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e97b523-29fb-474c-b711-8cd75b974f28-tls-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 23:03:38.421920 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:38.421899    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e97b523-29fb-474c-b711-8cd75b974f28-tmp-dir\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 23:03:38.843077 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:38.843040    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q" event={"ID":"8e97b523-29fb-474c-b711-8cd75b974f28","Type":"ContainerDied","Data":"6b370a15fd409b5530b3a3551b819a93d084e80e39e6125021eca745d6aac0ad"}
Sep 17 23:03:38.843505 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:38.843105    2555 scope.go:117] "RemoveContainer" containerID="6b5bc8c4c06110e93bf82a6150e608edd2f6d6e115b02d6e9025fd5573757a6b"
Sep 17 23:03:38.843505 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:38.843061    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q"
Sep 17 23:03:38.859364 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:38.859338    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q"]
Sep 17 23:03:38.865243 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:38.865220    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjw29q"]
Sep 17 23:03:40.679238 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:40.679203    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e97b523-29fb-474c-b711-8cd75b974f28" path="/var/lib/kubelet/pods/8e97b523-29fb-474c-b711-8cd75b974f28/volumes"
Sep 17 23:03:49.073700 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:49.073662    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:03:49.172739 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:49.172702    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:03:59.771671 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:59.771631    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-qc7z2"]
Sep 17 23:03:59.772101 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:59.772044    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e97b523-29fb-474c-b711-8cd75b974f28" containerName="main"
Sep 17 23:03:59.772101 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:59.772055    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e97b523-29fb-474c-b711-8cd75b974f28" containerName="main"
Sep 17 23:03:59.772209 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:59.772109    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e97b523-29fb-474c-b711-8cd75b974f28" containerName="main"
Sep 17 23:03:59.775193 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:59.775167    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-qc7z2"
Sep 17 23:03:59.783317 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:59.783293    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-qc7z2"]
Sep 17 23:03:59.796572 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:59.796542    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f30e19ad-4b2b-4cde-ab05-60566e7cb631-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-qc7z2\" (UID: \"f30e19ad-4b2b-4cde-ab05-60566e7cb631\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-qc7z2"
Sep 17 23:03:59.796701 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:59.796587    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f30e19ad-4b2b-4cde-ab05-60566e7cb631-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-qc7z2\" (UID: \"f30e19ad-4b2b-4cde-ab05-60566e7cb631\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-qc7z2"
Sep 17 23:03:59.796701 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:59.796673    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44lx6\" (UniqueName: \"kubernetes.io/projected/f30e19ad-4b2b-4cde-ab05-60566e7cb631-kube-api-access-44lx6\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-qc7z2\" (UID: \"f30e19ad-4b2b-4cde-ab05-60566e7cb631\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-qc7z2"
Sep 17 23:03:59.796815 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:59.796703    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f30e19ad-4b2b-4cde-ab05-60566e7cb631-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-qc7z2\" (UID: \"f30e19ad-4b2b-4cde-ab05-60566e7cb631\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-qc7z2"
Sep 17 23:03:59.796815 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:59.796752    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f30e19ad-4b2b-4cde-ab05-60566e7cb631-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-qc7z2\" (UID: \"f30e19ad-4b2b-4cde-ab05-60566e7cb631\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-qc7z2"
Sep 17 23:03:59.796815 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:59.796777    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f30e19ad-4b2b-4cde-ab05-60566e7cb631-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-qc7z2\" (UID: \"f30e19ad-4b2b-4cde-ab05-60566e7cb631\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-qc7z2"
Sep 17 23:03:59.897494 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:59.897458    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f30e19ad-4b2b-4cde-ab05-60566e7cb631-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-qc7z2\" (UID: \"f30e19ad-4b2b-4cde-ab05-60566e7cb631\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-qc7z2"
Sep 17 23:03:59.897494 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:59.897496    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f30e19ad-4b2b-4cde-ab05-60566e7cb631-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-qc7z2\" (UID: \"f30e19ad-4b2b-4cde-ab05-60566e7cb631\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-qc7z2"
Sep 17 23:03:59.897727 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:59.897527    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f30e19ad-4b2b-4cde-ab05-60566e7cb631-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-qc7z2\" (UID: \"f30e19ad-4b2b-4cde-ab05-60566e7cb631\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-qc7z2"
Sep 17 23:03:59.897727 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:59.897566    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f30e19ad-4b2b-4cde-ab05-60566e7cb631-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-qc7z2\" (UID: \"f30e19ad-4b2b-4cde-ab05-60566e7cb631\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-qc7z2"
Sep 17 23:03:59.897727 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:59.897648    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44lx6\" (UniqueName: \"kubernetes.io/projected/f30e19ad-4b2b-4cde-ab05-60566e7cb631-kube-api-access-44lx6\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-qc7z2\" (UID: \"f30e19ad-4b2b-4cde-ab05-60566e7cb631\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-qc7z2"
Sep 17 23:03:59.897727 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:59.897677    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f30e19ad-4b2b-4cde-ab05-60566e7cb631-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-qc7z2\" (UID: \"f30e19ad-4b2b-4cde-ab05-60566e7cb631\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-qc7z2"
Sep 17 23:03:59.897949 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:59.897858    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f30e19ad-4b2b-4cde-ab05-60566e7cb631-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-qc7z2\" (UID: \"f30e19ad-4b2b-4cde-ab05-60566e7cb631\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-qc7z2"
Sep 17 23:03:59.897949 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:59.897932    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f30e19ad-4b2b-4cde-ab05-60566e7cb631-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-qc7z2\" (UID: \"f30e19ad-4b2b-4cde-ab05-60566e7cb631\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-qc7z2"
Sep 17 23:03:59.898065 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:59.898032    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f30e19ad-4b2b-4cde-ab05-60566e7cb631-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-qc7z2\" (UID: \"f30e19ad-4b2b-4cde-ab05-60566e7cb631\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-qc7z2"
Sep 17 23:03:59.899885 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:59.899860    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f30e19ad-4b2b-4cde-ab05-60566e7cb631-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-qc7z2\" (UID: \"f30e19ad-4b2b-4cde-ab05-60566e7cb631\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-qc7z2"
Sep 17 23:03:59.900201 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:59.900169    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f30e19ad-4b2b-4cde-ab05-60566e7cb631-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-qc7z2\" (UID: \"f30e19ad-4b2b-4cde-ab05-60566e7cb631\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-qc7z2"
Sep 17 23:03:59.904769 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:03:59.904747    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44lx6\" (UniqueName: \"kubernetes.io/projected/f30e19ad-4b2b-4cde-ab05-60566e7cb631-kube-api-access-44lx6\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-qc7z2\" (UID: \"f30e19ad-4b2b-4cde-ab05-60566e7cb631\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-qc7z2"
Sep 17 23:04:00.087799 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:00.087713    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-qc7z2"
Sep 17 23:04:00.366101 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:00.366074    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-qc7z2"]
Sep 17 23:04:00.367380 ip-10-0-140-157 kubenswrapper[2555]: W0917 23:04:00.367347    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf30e19ad_4b2b_4cde_ab05_60566e7cb631.slice/crio-e8a8d4ad998bda9250a6319ed3a5b6bab4dc15a06d95c65b55215859ee630fc8 WatchSource:0}: Error finding container e8a8d4ad998bda9250a6319ed3a5b6bab4dc15a06d95c65b55215859ee630fc8: Status 404 returned error can't find the container with id e8a8d4ad998bda9250a6319ed3a5b6bab4dc15a06d95c65b55215859ee630fc8
Sep 17 23:04:00.974133 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:00.974091    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-qc7z2" event={"ID":"f30e19ad-4b2b-4cde-ab05-60566e7cb631","Type":"ContainerStarted","Data":"6070d9900b1a472116421e6f47596bc15040d960b2c6b31100b0549fa28647e5"}
Sep 17 23:04:00.974615 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:00.974153    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-qc7z2" event={"ID":"f30e19ad-4b2b-4cde-ab05-60566e7cb631","Type":"ContainerStarted","Data":"e8a8d4ad998bda9250a6319ed3a5b6bab4dc15a06d95c65b55215859ee630fc8"}
Sep 17 23:04:00.991571 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:00.991523    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-qc7z2" podStartSLOduration=1.991508841 podStartE2EDuration="1.991508841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:03:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:04:00.990217381 +0000 UTC m=+3942.436262654" watchObservedRunningTime="2026-09-17 23:04:00.991508841 +0000 UTC m=+3942.437554113"
Sep 17 23:04:01.088360 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:01.088322    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-qc7z2"
Sep 17 23:04:10.088567 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:10.088488    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-qc7z2"
Sep 17 23:04:10.101319 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:10.101291    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-qc7z2"
Sep 17 23:04:11.041451 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:11.041420    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-qc7z2"
Sep 17 23:04:11.154533 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:11.154501    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tddtj"]
Sep 17 23:04:11.163549 ip-10-0-140-157 kubenswrapper[2555]: E0917 23:04:11.163508    2555 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 23:04:11.163549 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:04:11.163549 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 23:04:11.163549 ip-10-0-140-157 kubenswrapper[2555]:  > containerID="83e0bb5defa40e770eb8f1cf6ca97b01fa1807b5839b857db9a13b682b67dd99" cmd=["/bin/sleep","15"]
Sep 17 23:04:11.163804 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:11.163568    2555 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tddtj" message="" err=<
Sep 17 23:04:11.163804 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:04:11.163804 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 23:04:11.163804 ip-10-0-140-157 kubenswrapper[2555]:  >
Sep 17 23:04:11.163804 ip-10-0-140-157 kubenswrapper[2555]: E0917 23:04:11.163580    2555 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 23:04:11.163804 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:04:11.163804 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 23:04:11.163804 ip-10-0-140-157 kubenswrapper[2555]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tddtj" podUID="e0161425-76ba-4d41-987b-b19991e1bf69" containerName="main" containerID="cri-o://83e0bb5defa40e770eb8f1cf6ca97b01fa1807b5839b857db9a13b682b67dd99"
Sep 17 23:04:11.163804 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:11.163632    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tddtj" podUID="e0161425-76ba-4d41-987b-b19991e1bf69" containerName="main" containerID="cri-o://83e0bb5defa40e770eb8f1cf6ca97b01fa1807b5839b857db9a13b682b67dd99" gracePeriod=60
Sep 17 23:04:11.516664 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:11.516634    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tddtj"
Sep 17 23:04:11.603537 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:11.603506    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0161425-76ba-4d41-987b-b19991e1bf69-dshm\") pod \"e0161425-76ba-4d41-987b-b19991e1bf69\" (UID: \"e0161425-76ba-4d41-987b-b19991e1bf69\") "
Sep 17 23:04:11.603713 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:11.603547    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0161425-76ba-4d41-987b-b19991e1bf69-tls-certs\") pod \"e0161425-76ba-4d41-987b-b19991e1bf69\" (UID: \"e0161425-76ba-4d41-987b-b19991e1bf69\") "
Sep 17 23:04:11.603713 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:11.603567    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0161425-76ba-4d41-987b-b19991e1bf69-tmp-dir\") pod \"e0161425-76ba-4d41-987b-b19991e1bf69\" (UID: \"e0161425-76ba-4d41-987b-b19991e1bf69\") "
Sep 17 23:04:11.603713 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:11.603589    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0161425-76ba-4d41-987b-b19991e1bf69-home\") pod \"e0161425-76ba-4d41-987b-b19991e1bf69\" (UID: \"e0161425-76ba-4d41-987b-b19991e1bf69\") "
Sep 17 23:04:11.603713 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:11.603674    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wz94b\" (UniqueName: \"kubernetes.io/projected/e0161425-76ba-4d41-987b-b19991e1bf69-kube-api-access-wz94b\") pod \"e0161425-76ba-4d41-987b-b19991e1bf69\" (UID: \"e0161425-76ba-4d41-987b-b19991e1bf69\") "
Sep 17 23:04:11.603713 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:11.603702    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0161425-76ba-4d41-987b-b19991e1bf69-model-cache\") pod \"e0161425-76ba-4d41-987b-b19991e1bf69\" (UID: \"e0161425-76ba-4d41-987b-b19991e1bf69\") "
Sep 17 23:04:11.604022 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:11.603840    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0161425-76ba-4d41-987b-b19991e1bf69-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e0161425-76ba-4d41-987b-b19991e1bf69" (UID: "e0161425-76ba-4d41-987b-b19991e1bf69"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:04:11.604022 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:11.603953    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0161425-76ba-4d41-987b-b19991e1bf69-home" (OuterVolumeSpecName: "home") pod "e0161425-76ba-4d41-987b-b19991e1bf69" (UID: "e0161425-76ba-4d41-987b-b19991e1bf69"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:04:11.604113 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:11.604043    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0161425-76ba-4d41-987b-b19991e1bf69-model-cache" (OuterVolumeSpecName: "model-cache") pod "e0161425-76ba-4d41-987b-b19991e1bf69" (UID: "e0161425-76ba-4d41-987b-b19991e1bf69"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:04:11.604113 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:11.604084    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0161425-76ba-4d41-987b-b19991e1bf69-tmp-dir\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 23:04:11.604113 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:11.604103    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0161425-76ba-4d41-987b-b19991e1bf69-home\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 23:04:11.605899 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:11.605872    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0161425-76ba-4d41-987b-b19991e1bf69-dshm" (OuterVolumeSpecName: "dshm") pod "e0161425-76ba-4d41-987b-b19991e1bf69" (UID: "e0161425-76ba-4d41-987b-b19991e1bf69"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:04:11.605993 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:11.605944    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0161425-76ba-4d41-987b-b19991e1bf69-kube-api-access-wz94b" (OuterVolumeSpecName: "kube-api-access-wz94b") pod "e0161425-76ba-4d41-987b-b19991e1bf69" (UID: "e0161425-76ba-4d41-987b-b19991e1bf69"). InnerVolumeSpecName "kube-api-access-wz94b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:04:11.605993 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:11.605953    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0161425-76ba-4d41-987b-b19991e1bf69-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e0161425-76ba-4d41-987b-b19991e1bf69" (UID: "e0161425-76ba-4d41-987b-b19991e1bf69"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:04:11.705098 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:11.705027    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0161425-76ba-4d41-987b-b19991e1bf69-dshm\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 23:04:11.705098 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:11.705053    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0161425-76ba-4d41-987b-b19991e1bf69-tls-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 23:04:11.705098 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:11.705064    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz94b\" (UniqueName: \"kubernetes.io/projected/e0161425-76ba-4d41-987b-b19991e1bf69-kube-api-access-wz94b\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 23:04:11.705098 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:11.705073    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0161425-76ba-4d41-987b-b19991e1bf69-model-cache\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 23:04:12.037489 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:12.037383    2555 generic.go:356] "Generic (PLEG): container finished" podID="e0161425-76ba-4d41-987b-b19991e1bf69" containerID="83e0bb5defa40e770eb8f1cf6ca97b01fa1807b5839b857db9a13b682b67dd99" exitCode=0
Sep 17 23:04:12.037489 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:12.037427    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tddtj"
Sep 17 23:04:12.037489 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:12.037462    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tddtj" event={"ID":"e0161425-76ba-4d41-987b-b19991e1bf69","Type":"ContainerDied","Data":"83e0bb5defa40e770eb8f1cf6ca97b01fa1807b5839b857db9a13b682b67dd99"}
Sep 17 23:04:12.037802 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:12.037529    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tddtj" event={"ID":"e0161425-76ba-4d41-987b-b19991e1bf69","Type":"ContainerDied","Data":"56b0f88ccfb61a890400d823a22109d7a1cb820f42ce859de498dc1427fc7f89"}
Sep 17 23:04:12.037802 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:12.037557    2555 scope.go:117] "RemoveContainer" containerID="83e0bb5defa40e770eb8f1cf6ca97b01fa1807b5839b857db9a13b682b67dd99"
Sep 17 23:04:12.047793 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:12.047776    2555 scope.go:117] "RemoveContainer" containerID="83e0bb5defa40e770eb8f1cf6ca97b01fa1807b5839b857db9a13b682b67dd99"
Sep 17 23:04:12.048063 ip-10-0-140-157 kubenswrapper[2555]: E0917 23:04:12.048043    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83e0bb5defa40e770eb8f1cf6ca97b01fa1807b5839b857db9a13b682b67dd99\": container with ID starting with 83e0bb5defa40e770eb8f1cf6ca97b01fa1807b5839b857db9a13b682b67dd99 not found: ID does not exist" containerID="83e0bb5defa40e770eb8f1cf6ca97b01fa1807b5839b857db9a13b682b67dd99"
Sep 17 23:04:12.048108 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:12.048071    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83e0bb5defa40e770eb8f1cf6ca97b01fa1807b5839b857db9a13b682b67dd99"} err="failed to get container status \"83e0bb5defa40e770eb8f1cf6ca97b01fa1807b5839b857db9a13b682b67dd99\": rpc error: code = NotFound desc = could not find container \"83e0bb5defa40e770eb8f1cf6ca97b01fa1807b5839b857db9a13b682b67dd99\": container with ID starting with 83e0bb5defa40e770eb8f1cf6ca97b01fa1807b5839b857db9a13b682b67dd99 not found: ID does not exist"
Sep 17 23:04:12.067463 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:12.067434    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tddtj"]
Sep 17 23:04:12.072920 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:12.072896    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tddtj"]
Sep 17 23:04:12.679988 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:12.679947    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0161425-76ba-4d41-987b-b19991e1bf69" path="/var/lib/kubelet/pods/e0161425-76ba-4d41-987b-b19991e1bf69/volumes"
Sep 17 23:04:13.063649 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:13.063534    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:04:13.064165 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:13.064143    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-qc7z2"]
Sep 17 23:04:13.073803 ip-10-0-140-157 kubenswrapper[2555]: E0917 23:04:13.073744    2555 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 23:04:13.073803 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:04:13.073803 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 23:04:13.073803 ip-10-0-140-157 kubenswrapper[2555]:  > containerID="6070d9900b1a472116421e6f47596bc15040d960b2c6b31100b0549fa28647e5" cmd=["/bin/sleep","15"]
Sep 17 23:04:13.074140 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:13.073814    2555 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-qc7z2" message="" err=<
Sep 17 23:04:13.074140 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:04:13.074140 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 23:04:13.074140 ip-10-0-140-157 kubenswrapper[2555]:  >
Sep 17 23:04:13.074140 ip-10-0-140-157 kubenswrapper[2555]: E0917 23:04:13.073828    2555 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 23:04:13.074140 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:04:13.074140 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 23:04:13.074140 ip-10-0-140-157 kubenswrapper[2555]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-qc7z2" podUID="f30e19ad-4b2b-4cde-ab05-60566e7cb631" containerName="main" containerID="cri-o://6070d9900b1a472116421e6f47596bc15040d960b2c6b31100b0549fa28647e5"
Sep 17 23:04:13.074140 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:13.073881    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-qc7z2" podUID="f30e19ad-4b2b-4cde-ab05-60566e7cb631" containerName="main" containerID="cri-o://6070d9900b1a472116421e6f47596bc15040d960b2c6b31100b0549fa28647e5" gracePeriod=60
Sep 17 23:04:13.161626 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:13.161596    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:04:13.823576 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:13.823549    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-qc7z2"
Sep 17 23:04:13.927418 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:13.927387    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f30e19ad-4b2b-4cde-ab05-60566e7cb631-home\") pod \"f30e19ad-4b2b-4cde-ab05-60566e7cb631\" (UID: \"f30e19ad-4b2b-4cde-ab05-60566e7cb631\") "
Sep 17 23:04:13.927622 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:13.927464    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f30e19ad-4b2b-4cde-ab05-60566e7cb631-tmp-dir\") pod \"f30e19ad-4b2b-4cde-ab05-60566e7cb631\" (UID: \"f30e19ad-4b2b-4cde-ab05-60566e7cb631\") "
Sep 17 23:04:13.927622 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:13.927519    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44lx6\" (UniqueName: \"kubernetes.io/projected/f30e19ad-4b2b-4cde-ab05-60566e7cb631-kube-api-access-44lx6\") pod \"f30e19ad-4b2b-4cde-ab05-60566e7cb631\" (UID: \"f30e19ad-4b2b-4cde-ab05-60566e7cb631\") "
Sep 17 23:04:13.927622 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:13.927591    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f30e19ad-4b2b-4cde-ab05-60566e7cb631-dshm\") pod \"f30e19ad-4b2b-4cde-ab05-60566e7cb631\" (UID: \"f30e19ad-4b2b-4cde-ab05-60566e7cb631\") "
Sep 17 23:04:13.927622 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:13.927618    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f30e19ad-4b2b-4cde-ab05-60566e7cb631-model-cache\") pod \"f30e19ad-4b2b-4cde-ab05-60566e7cb631\" (UID: \"f30e19ad-4b2b-4cde-ab05-60566e7cb631\") "
Sep 17 23:04:13.927834 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:13.927664    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f30e19ad-4b2b-4cde-ab05-60566e7cb631-tls-certs\") pod \"f30e19ad-4b2b-4cde-ab05-60566e7cb631\" (UID: \"f30e19ad-4b2b-4cde-ab05-60566e7cb631\") "
Sep 17 23:04:13.927834 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:13.927698    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f30e19ad-4b2b-4cde-ab05-60566e7cb631-home" (OuterVolumeSpecName: "home") pod "f30e19ad-4b2b-4cde-ab05-60566e7cb631" (UID: "f30e19ad-4b2b-4cde-ab05-60566e7cb631"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:04:13.927834 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:13.927711    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f30e19ad-4b2b-4cde-ab05-60566e7cb631-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f30e19ad-4b2b-4cde-ab05-60566e7cb631" (UID: "f30e19ad-4b2b-4cde-ab05-60566e7cb631"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:04:13.927974 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:13.927880    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f30e19ad-4b2b-4cde-ab05-60566e7cb631-model-cache" (OuterVolumeSpecName: "model-cache") pod "f30e19ad-4b2b-4cde-ab05-60566e7cb631" (UID: "f30e19ad-4b2b-4cde-ab05-60566e7cb631"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:04:13.927974 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:13.927922    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f30e19ad-4b2b-4cde-ab05-60566e7cb631-home\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 23:04:13.927974 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:13.927939    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f30e19ad-4b2b-4cde-ab05-60566e7cb631-tmp-dir\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 23:04:13.929756 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:13.929725    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f30e19ad-4b2b-4cde-ab05-60566e7cb631-dshm" (OuterVolumeSpecName: "dshm") pod "f30e19ad-4b2b-4cde-ab05-60566e7cb631" (UID: "f30e19ad-4b2b-4cde-ab05-60566e7cb631"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:04:13.929756 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:13.929732    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f30e19ad-4b2b-4cde-ab05-60566e7cb631-kube-api-access-44lx6" (OuterVolumeSpecName: "kube-api-access-44lx6") pod "f30e19ad-4b2b-4cde-ab05-60566e7cb631" (UID: "f30e19ad-4b2b-4cde-ab05-60566e7cb631"). InnerVolumeSpecName "kube-api-access-44lx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:04:13.929907 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:13.929766    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f30e19ad-4b2b-4cde-ab05-60566e7cb631-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f30e19ad-4b2b-4cde-ab05-60566e7cb631" (UID: "f30e19ad-4b2b-4cde-ab05-60566e7cb631"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:04:14.029208 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:14.029072    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44lx6\" (UniqueName: \"kubernetes.io/projected/f30e19ad-4b2b-4cde-ab05-60566e7cb631-kube-api-access-44lx6\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 23:04:14.029208 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:14.029121    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f30e19ad-4b2b-4cde-ab05-60566e7cb631-dshm\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 23:04:14.029208 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:14.029133    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f30e19ad-4b2b-4cde-ab05-60566e7cb631-model-cache\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 23:04:14.029208 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:14.029141    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f30e19ad-4b2b-4cde-ab05-60566e7cb631-tls-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 23:04:14.051965 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:14.051924    2555 generic.go:356] "Generic (PLEG): container finished" podID="f30e19ad-4b2b-4cde-ab05-60566e7cb631" containerID="6070d9900b1a472116421e6f47596bc15040d960b2c6b31100b0549fa28647e5" exitCode=0
Sep 17 23:04:14.052172 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:14.051987    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-qc7z2"
Sep 17 23:04:14.052172 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:14.052006    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-qc7z2" event={"ID":"f30e19ad-4b2b-4cde-ab05-60566e7cb631","Type":"ContainerDied","Data":"6070d9900b1a472116421e6f47596bc15040d960b2c6b31100b0549fa28647e5"}
Sep 17 23:04:14.052172 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:14.052062    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-qc7z2" event={"ID":"f30e19ad-4b2b-4cde-ab05-60566e7cb631","Type":"ContainerDied","Data":"e8a8d4ad998bda9250a6319ed3a5b6bab4dc15a06d95c65b55215859ee630fc8"}
Sep 17 23:04:14.052172 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:14.052086    2555 scope.go:117] "RemoveContainer" containerID="6070d9900b1a472116421e6f47596bc15040d960b2c6b31100b0549fa28647e5"
Sep 17 23:04:14.062070 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:14.062049    2555 scope.go:117] "RemoveContainer" containerID="6070d9900b1a472116421e6f47596bc15040d960b2c6b31100b0549fa28647e5"
Sep 17 23:04:14.062445 ip-10-0-140-157 kubenswrapper[2555]: E0917 23:04:14.062423    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6070d9900b1a472116421e6f47596bc15040d960b2c6b31100b0549fa28647e5\": container with ID starting with 6070d9900b1a472116421e6f47596bc15040d960b2c6b31100b0549fa28647e5 not found: ID does not exist" containerID="6070d9900b1a472116421e6f47596bc15040d960b2c6b31100b0549fa28647e5"
Sep 17 23:04:14.062505 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:14.062454    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6070d9900b1a472116421e6f47596bc15040d960b2c6b31100b0549fa28647e5"} err="failed to get container status \"6070d9900b1a472116421e6f47596bc15040d960b2c6b31100b0549fa28647e5\": rpc error: code = NotFound desc = could not find container \"6070d9900b1a472116421e6f47596bc15040d960b2c6b31100b0549fa28647e5\": container with ID starting with 6070d9900b1a472116421e6f47596bc15040d960b2c6b31100b0549fa28647e5 not found: ID does not exist"
Sep 17 23:04:14.076169 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:14.076134    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-qc7z2"]
Sep 17 23:04:14.078565 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:14.078539    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-qc7z2"]
Sep 17 23:04:14.679234 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:14.679198    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f30e19ad-4b2b-4cde-ab05-60566e7cb631" path="/var/lib/kubelet/pods/f30e19ad-4b2b-4cde-ab05-60566e7cb631/volumes"
Sep 17 23:04:26.842887 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:26.842850    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:04:26.935519 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:26.935484    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:04:33.022319 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:33.022283    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:04:38.637479 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:38.637441    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:04:38.706194 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:38.706148    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:04:38.835714 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:38.835675    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8"]
Sep 17 23:04:38.836134 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:38.836114    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0161425-76ba-4d41-987b-b19991e1bf69" containerName="main"
Sep 17 23:04:38.836134 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:38.836133    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0161425-76ba-4d41-987b-b19991e1bf69" containerName="main"
Sep 17 23:04:38.836339 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:38.836141    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f30e19ad-4b2b-4cde-ab05-60566e7cb631" containerName="main"
Sep 17 23:04:38.836339 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:38.836148    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f30e19ad-4b2b-4cde-ab05-60566e7cb631" containerName="main"
Sep 17 23:04:38.836339 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:38.836243    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f30e19ad-4b2b-4cde-ab05-60566e7cb631" containerName="main"
Sep 17 23:04:38.836339 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:38.836256    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0161425-76ba-4d41-987b-b19991e1bf69" containerName="main"
Sep 17 23:04:38.839429 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:38.839409    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8"
Sep 17 23:04:38.843015 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:38.842988    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 17 23:04:38.843134 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:38.843105    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 17 23:04:38.843212 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:38.843129    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 17 23:04:38.843275 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:38.843218    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-s9hm7\""
Sep 17 23:04:38.849288 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:38.849266    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8"]
Sep 17 23:04:38.869007 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:38.868980    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:04:38.941348 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:38.941278    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ab8107d-193d-4f3b-8071-c0450f7f97d7-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8\" (UID: \"5ab8107d-193d-4f3b-8071-c0450f7f97d7\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8"
Sep 17 23:04:38.941348 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:38.941319    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ab8107d-193d-4f3b-8071-c0450f7f97d7-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8\" (UID: \"5ab8107d-193d-4f3b-8071-c0450f7f97d7\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8"
Sep 17 23:04:38.941348 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:38.941336    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ab8107d-193d-4f3b-8071-c0450f7f97d7-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8\" (UID: \"5ab8107d-193d-4f3b-8071-c0450f7f97d7\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8"
Sep 17 23:04:38.941550 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:38.941430    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ab8107d-193d-4f3b-8071-c0450f7f97d7-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8\" (UID: \"5ab8107d-193d-4f3b-8071-c0450f7f97d7\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8"
Sep 17 23:04:38.941550 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:38.941465    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ab8107d-193d-4f3b-8071-c0450f7f97d7-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8\" (UID: \"5ab8107d-193d-4f3b-8071-c0450f7f97d7\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8"
Sep 17 23:04:38.941550 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:38.941506    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n2dd\" (UniqueName: \"kubernetes.io/projected/5ab8107d-193d-4f3b-8071-c0450f7f97d7-kube-api-access-5n2dd\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8\" (UID: \"5ab8107d-193d-4f3b-8071-c0450f7f97d7\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8"
Sep 17 23:04:39.042505 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:39.042465    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ab8107d-193d-4f3b-8071-c0450f7f97d7-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8\" (UID: \"5ab8107d-193d-4f3b-8071-c0450f7f97d7\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8"
Sep 17 23:04:39.042692 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:39.042530    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ab8107d-193d-4f3b-8071-c0450f7f97d7-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8\" (UID: \"5ab8107d-193d-4f3b-8071-c0450f7f97d7\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8"
Sep 17 23:04:39.042692 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:39.042550    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ab8107d-193d-4f3b-8071-c0450f7f97d7-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8\" (UID: \"5ab8107d-193d-4f3b-8071-c0450f7f97d7\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8"
Sep 17 23:04:39.042692 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:39.042606    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ab8107d-193d-4f3b-8071-c0450f7f97d7-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8\" (UID: \"5ab8107d-193d-4f3b-8071-c0450f7f97d7\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8"
Sep 17 23:04:39.042692 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:39.042632    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ab8107d-193d-4f3b-8071-c0450f7f97d7-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8\" (UID: \"5ab8107d-193d-4f3b-8071-c0450f7f97d7\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8"
Sep 17 23:04:39.042692 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:39.042661    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n2dd\" (UniqueName: \"kubernetes.io/projected/5ab8107d-193d-4f3b-8071-c0450f7f97d7-kube-api-access-5n2dd\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8\" (UID: \"5ab8107d-193d-4f3b-8071-c0450f7f97d7\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8"
Sep 17 23:04:39.042981 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:39.042918    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ab8107d-193d-4f3b-8071-c0450f7f97d7-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8\" (UID: \"5ab8107d-193d-4f3b-8071-c0450f7f97d7\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8"
Sep 17 23:04:39.043224 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:39.043204    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ab8107d-193d-4f3b-8071-c0450f7f97d7-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8\" (UID: \"5ab8107d-193d-4f3b-8071-c0450f7f97d7\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8"
Sep 17 23:04:39.043328 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:39.043304    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ab8107d-193d-4f3b-8071-c0450f7f97d7-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8\" (UID: \"5ab8107d-193d-4f3b-8071-c0450f7f97d7\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8"
Sep 17 23:04:39.045229 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:39.045173    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ab8107d-193d-4f3b-8071-c0450f7f97d7-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8\" (UID: \"5ab8107d-193d-4f3b-8071-c0450f7f97d7\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8"
Sep 17 23:04:39.045346 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:39.045207    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ab8107d-193d-4f3b-8071-c0450f7f97d7-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8\" (UID: \"5ab8107d-193d-4f3b-8071-c0450f7f97d7\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8"
Sep 17 23:04:39.054080 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:39.054053    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n2dd\" (UniqueName: \"kubernetes.io/projected/5ab8107d-193d-4f3b-8071-c0450f7f97d7-kube-api-access-5n2dd\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8\" (UID: \"5ab8107d-193d-4f3b-8071-c0450f7f97d7\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8"
Sep 17 23:04:39.089434 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:39.089392    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:04:39.152344 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:39.152310    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8"
Sep 17 23:04:39.433611 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:39.433560    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8"]
Sep 17 23:04:39.434646 ip-10-0-140-157 kubenswrapper[2555]: W0917 23:04:39.434613    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ab8107d_193d_4f3b_8071_c0450f7f97d7.slice/crio-1e75fbec71c784b198431930cc113dfbb3222444e1794b2337dea3241bbdf574 WatchSource:0}: Error finding container 1e75fbec71c784b198431930cc113dfbb3222444e1794b2337dea3241bbdf574: Status 404 returned error can't find the container with id 1e75fbec71c784b198431930cc113dfbb3222444e1794b2337dea3241bbdf574
Sep 17 23:04:40.199748 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:40.199702    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8" event={"ID":"5ab8107d-193d-4f3b-8071-c0450f7f97d7","Type":"ContainerStarted","Data":"0a728d8a4c7ed40ae7a37d20fb056a03a6791b7c03eec250a8c48dc96d5922a0"}
Sep 17 23:04:40.200236 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:40.199804    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8" event={"ID":"5ab8107d-193d-4f3b-8071-c0450f7f97d7","Type":"ContainerStarted","Data":"1e75fbec71c784b198431930cc113dfbb3222444e1794b2337dea3241bbdf574"}
Sep 17 23:04:40.218049 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:40.218000    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8" podStartSLOduration=2.217984913 podStartE2EDuration="2.217984913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:04:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:04:40.215749554 +0000 UTC m=+3981.661794852" watchObservedRunningTime="2026-09-17 23:04:40.217984913 +0000 UTC m=+3981.664030184"
Sep 17 23:04:40.680927 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:40.680894    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:04:40.781905 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:40.781866    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:04:41.153341 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:41.153291    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8"
Sep 17 23:04:49.153291 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:49.153251    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8"
Sep 17 23:04:49.165706 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:49.165678    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8"
Sep 17 23:04:49.260364 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:49.260336    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8"
Sep 17 23:04:57.255534 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:04:57.255499    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:05:01.662284 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:05:01.662251    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:05:04.015002 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:05:04.014964    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:05:07.456757 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:05:07.456720    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:05:12.871692 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:05:12.871649    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:05:12.881168 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:05:12.881123    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8"]
Sep 17 23:05:12.890937 ip-10-0-140-157 kubenswrapper[2555]: E0917 23:05:12.890900    2555 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 23:05:12.890937 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:05:12.890937 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 23:05:12.890937 ip-10-0-140-157 kubenswrapper[2555]:  > containerID="0a728d8a4c7ed40ae7a37d20fb056a03a6791b7c03eec250a8c48dc96d5922a0" cmd=["/bin/sleep","15"]
Sep 17 23:05:12.891384 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:05:12.890954    2555 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8" message="" err=<
Sep 17 23:05:12.891384 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:05:12.891384 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 23:05:12.891384 ip-10-0-140-157 kubenswrapper[2555]:  >
Sep 17 23:05:12.891384 ip-10-0-140-157 kubenswrapper[2555]: E0917 23:05:12.890966    2555 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 23:05:12.891384 ip-10-0-140-157 kubenswrapper[2555]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:05:12.891384 ip-10-0-140-157 kubenswrapper[2555]:         , stdout: , stderr: , exit code -1
Sep 17 23:05:12.891384 ip-10-0-140-157 kubenswrapper[2555]:  > pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8" podUID="5ab8107d-193d-4f3b-8071-c0450f7f97d7" containerName="main" containerID="cri-o://0a728d8a4c7ed40ae7a37d20fb056a03a6791b7c03eec250a8c48dc96d5922a0"
Sep 17 23:05:12.891384 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:05:12.891002    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8" podUID="5ab8107d-193d-4f3b-8071-c0450f7f97d7" containerName="main" containerID="cri-o://0a728d8a4c7ed40ae7a37d20fb056a03a6791b7c03eec250a8c48dc96d5922a0" gracePeriod=60
Sep 17 23:05:12.969707 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:05:12.969653    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:05:13.253038 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:05:13.253011    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8"
Sep 17 23:05:13.341171 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:05:13.341141    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ab8107d-193d-4f3b-8071-c0450f7f97d7-model-cache\") pod \"5ab8107d-193d-4f3b-8071-c0450f7f97d7\" (UID: \"5ab8107d-193d-4f3b-8071-c0450f7f97d7\") "
Sep 17 23:05:13.341171 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:05:13.341176    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ab8107d-193d-4f3b-8071-c0450f7f97d7-tls-certs\") pod \"5ab8107d-193d-4f3b-8071-c0450f7f97d7\" (UID: \"5ab8107d-193d-4f3b-8071-c0450f7f97d7\") "
Sep 17 23:05:13.341412 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:05:13.341268    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5n2dd\" (UniqueName: \"kubernetes.io/projected/5ab8107d-193d-4f3b-8071-c0450f7f97d7-kube-api-access-5n2dd\") pod \"5ab8107d-193d-4f3b-8071-c0450f7f97d7\" (UID: \"5ab8107d-193d-4f3b-8071-c0450f7f97d7\") "
Sep 17 23:05:13.341412 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:05:13.341338    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ab8107d-193d-4f3b-8071-c0450f7f97d7-tmp-dir\") pod \"5ab8107d-193d-4f3b-8071-c0450f7f97d7\" (UID: \"5ab8107d-193d-4f3b-8071-c0450f7f97d7\") "
Sep 17 23:05:13.341412 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:05:13.341385    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ab8107d-193d-4f3b-8071-c0450f7f97d7-dshm\") pod \"5ab8107d-193d-4f3b-8071-c0450f7f97d7\" (UID: \"5ab8107d-193d-4f3b-8071-c0450f7f97d7\") "
Sep 17 23:05:13.341547 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:05:13.341424    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ab8107d-193d-4f3b-8071-c0450f7f97d7-home\") pod \"5ab8107d-193d-4f3b-8071-c0450f7f97d7\" (UID: \"5ab8107d-193d-4f3b-8071-c0450f7f97d7\") "
Sep 17 23:05:13.341547 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:05:13.341382    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ab8107d-193d-4f3b-8071-c0450f7f97d7-model-cache" (OuterVolumeSpecName: "model-cache") pod "5ab8107d-193d-4f3b-8071-c0450f7f97d7" (UID: "5ab8107d-193d-4f3b-8071-c0450f7f97d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:05:13.341547 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:05:13.341526    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ab8107d-193d-4f3b-8071-c0450f7f97d7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5ab8107d-193d-4f3b-8071-c0450f7f97d7" (UID: "5ab8107d-193d-4f3b-8071-c0450f7f97d7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:05:13.341731 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:05:13.341709    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ab8107d-193d-4f3b-8071-c0450f7f97d7-home" (OuterVolumeSpecName: "home") pod "5ab8107d-193d-4f3b-8071-c0450f7f97d7" (UID: "5ab8107d-193d-4f3b-8071-c0450f7f97d7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:05:13.341731 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:05:13.341718    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ab8107d-193d-4f3b-8071-c0450f7f97d7-tmp-dir\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 23:05:13.341894 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:05:13.341740    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ab8107d-193d-4f3b-8071-c0450f7f97d7-model-cache\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 23:05:13.343625 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:05:13.343602    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ab8107d-193d-4f3b-8071-c0450f7f97d7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5ab8107d-193d-4f3b-8071-c0450f7f97d7" (UID: "5ab8107d-193d-4f3b-8071-c0450f7f97d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:05:13.343625 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:05:13.343614    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ab8107d-193d-4f3b-8071-c0450f7f97d7-dshm" (OuterVolumeSpecName: "dshm") pod "5ab8107d-193d-4f3b-8071-c0450f7f97d7" (UID: "5ab8107d-193d-4f3b-8071-c0450f7f97d7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:05:13.343756 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:05:13.343610    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ab8107d-193d-4f3b-8071-c0450f7f97d7-kube-api-access-5n2dd" (OuterVolumeSpecName: "kube-api-access-5n2dd") pod "5ab8107d-193d-4f3b-8071-c0450f7f97d7" (UID: "5ab8107d-193d-4f3b-8071-c0450f7f97d7"). InnerVolumeSpecName "kube-api-access-5n2dd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:05:13.386041 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:05:13.386010    2555 generic.go:356] "Generic (PLEG): container finished" podID="5ab8107d-193d-4f3b-8071-c0450f7f97d7" containerID="0a728d8a4c7ed40ae7a37d20fb056a03a6791b7c03eec250a8c48dc96d5922a0" exitCode=0
Sep 17 23:05:13.386158 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:05:13.386072    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8"
Sep 17 23:05:13.386158 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:05:13.386110    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8" event={"ID":"5ab8107d-193d-4f3b-8071-c0450f7f97d7","Type":"ContainerDied","Data":"0a728d8a4c7ed40ae7a37d20fb056a03a6791b7c03eec250a8c48dc96d5922a0"}
Sep 17 23:05:13.386295 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:05:13.386167    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8" event={"ID":"5ab8107d-193d-4f3b-8071-c0450f7f97d7","Type":"ContainerDied","Data":"1e75fbec71c784b198431930cc113dfbb3222444e1794b2337dea3241bbdf574"}
Sep 17 23:05:13.386295 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:05:13.386208    2555 scope.go:117] "RemoveContainer" containerID="0a728d8a4c7ed40ae7a37d20fb056a03a6791b7c03eec250a8c48dc96d5922a0"
Sep 17 23:05:13.395664 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:05:13.395648    2555 scope.go:117] "RemoveContainer" containerID="0a728d8a4c7ed40ae7a37d20fb056a03a6791b7c03eec250a8c48dc96d5922a0"
Sep 17 23:05:13.395964 ip-10-0-140-157 kubenswrapper[2555]: E0917 23:05:13.395941    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a728d8a4c7ed40ae7a37d20fb056a03a6791b7c03eec250a8c48dc96d5922a0\": container with ID starting with 0a728d8a4c7ed40ae7a37d20fb056a03a6791b7c03eec250a8c48dc96d5922a0 not found: ID does not exist" containerID="0a728d8a4c7ed40ae7a37d20fb056a03a6791b7c03eec250a8c48dc96d5922a0"
Sep 17 23:05:13.396008 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:05:13.395973    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a728d8a4c7ed40ae7a37d20fb056a03a6791b7c03eec250a8c48dc96d5922a0"} err="failed to get container status \"0a728d8a4c7ed40ae7a37d20fb056a03a6791b7c03eec250a8c48dc96d5922a0\": rpc error: code = NotFound desc = could not find container \"0a728d8a4c7ed40ae7a37d20fb056a03a6791b7c03eec250a8c48dc96d5922a0\": container with ID starting with 0a728d8a4c7ed40ae7a37d20fb056a03a6791b7c03eec250a8c48dc96d5922a0 not found: ID does not exist"
Sep 17 23:05:13.422979 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:05:13.422923    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8"]
Sep 17 23:05:13.435106 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:05:13.435081    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cmvz8"]
Sep 17 23:05:13.442652 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:05:13.442632    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ab8107d-193d-4f3b-8071-c0450f7f97d7-dshm\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 23:05:13.442725 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:05:13.442653    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ab8107d-193d-4f3b-8071-c0450f7f97d7-home\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 23:05:13.442725 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:05:13.442663    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ab8107d-193d-4f3b-8071-c0450f7f97d7-tls-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 23:05:13.442725 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:05:13.442672    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n2dd\" (UniqueName: \"kubernetes.io/projected/5ab8107d-193d-4f3b-8071-c0450f7f97d7-kube-api-access-5n2dd\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 23:05:14.062242 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:05:14.062199    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:05:14.162488 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:05:14.162448    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:05:14.679351 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:05:14.679318    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ab8107d-193d-4f3b-8071-c0450f7f97d7" path="/var/lib/kubelet/pods/5ab8107d-193d-4f3b-8071-c0450f7f97d7/volumes"
Sep 17 23:05:17.709472 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:05:17.709439    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:02.833577 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:02.833543    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:06.739682 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:06.739636    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6l9vt/must-gather-hmdkt"]
Sep 17 23:06:06.740276 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:06.740237    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ab8107d-193d-4f3b-8071-c0450f7f97d7" containerName="main"
Sep 17 23:06:06.740276 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:06.740261    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ab8107d-193d-4f3b-8071-c0450f7f97d7" containerName="main"
Sep 17 23:06:06.740433 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:06.740418    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ab8107d-193d-4f3b-8071-c0450f7f97d7" containerName="main"
Sep 17 23:06:06.743603 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:06.743581    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6l9vt/must-gather-hmdkt"
Sep 17 23:06:06.745994 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:06.745966    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6l9vt\"/\"kube-root-ca.crt\""
Sep 17 23:06:06.747486 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:06.747457    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6l9vt\"/\"default-dockercfg-krmfh\""
Sep 17 23:06:06.747595 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:06.747463    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6l9vt\"/\"openshift-service-ca.crt\""
Sep 17 23:06:06.752763 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:06.752734    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6l9vt/must-gather-hmdkt"]
Sep 17 23:06:06.753916 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:06.753894    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:06.789855 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:06.789809    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d17a0b3d-7745-44d8-8e53-8faf694ba74d-must-gather-output\") pod \"must-gather-hmdkt\" (UID: \"d17a0b3d-7745-44d8-8e53-8faf694ba74d\") " pod="openshift-must-gather-6l9vt/must-gather-hmdkt"
Sep 17 23:06:06.790059 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:06.789903    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zllfm\" (UniqueName: \"kubernetes.io/projected/d17a0b3d-7745-44d8-8e53-8faf694ba74d-kube-api-access-zllfm\") pod \"must-gather-hmdkt\" (UID: \"d17a0b3d-7745-44d8-8e53-8faf694ba74d\") " pod="openshift-must-gather-6l9vt/must-gather-hmdkt"
Sep 17 23:06:06.820323 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:06.820290    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:06.891196 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:06.891147    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zllfm\" (UniqueName: \"kubernetes.io/projected/d17a0b3d-7745-44d8-8e53-8faf694ba74d-kube-api-access-zllfm\") pod \"must-gather-hmdkt\" (UID: \"d17a0b3d-7745-44d8-8e53-8faf694ba74d\") " pod="openshift-must-gather-6l9vt/must-gather-hmdkt"
Sep 17 23:06:06.891388 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:06.891273    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d17a0b3d-7745-44d8-8e53-8faf694ba74d-must-gather-output\") pod \"must-gather-hmdkt\" (UID: \"d17a0b3d-7745-44d8-8e53-8faf694ba74d\") " pod="openshift-must-gather-6l9vt/must-gather-hmdkt"
Sep 17 23:06:06.891645 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:06.891625    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d17a0b3d-7745-44d8-8e53-8faf694ba74d-must-gather-output\") pod \"must-gather-hmdkt\" (UID: \"d17a0b3d-7745-44d8-8e53-8faf694ba74d\") " pod="openshift-must-gather-6l9vt/must-gather-hmdkt"
Sep 17 23:06:06.898923 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:06.898885    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zllfm\" (UniqueName: \"kubernetes.io/projected/d17a0b3d-7745-44d8-8e53-8faf694ba74d-kube-api-access-zllfm\") pod \"must-gather-hmdkt\" (UID: \"d17a0b3d-7745-44d8-8e53-8faf694ba74d\") " pod="openshift-must-gather-6l9vt/must-gather-hmdkt"
Sep 17 23:06:07.054335 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:07.054243    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6l9vt/must-gather-hmdkt"
Sep 17 23:06:07.327489 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:07.327462    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6l9vt/must-gather-hmdkt"]
Sep 17 23:06:07.328853 ip-10-0-140-157 kubenswrapper[2555]: W0917 23:06:07.328805    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd17a0b3d_7745_44d8_8e53_8faf694ba74d.slice/crio-1cd00c8ffb5f2edf9ebbcf8a733a0ccbd12dfbe625b4fba78591d9783d65c2bb WatchSource:0}: Error finding container 1cd00c8ffb5f2edf9ebbcf8a733a0ccbd12dfbe625b4fba78591d9783d65c2bb: Status 404 returned error can't find the container with id 1cd00c8ffb5f2edf9ebbcf8a733a0ccbd12dfbe625b4fba78591d9783d65c2bb
Sep 17 23:06:07.330763 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:07.330747    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:06:07.685627 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:07.685583    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6l9vt/must-gather-hmdkt" event={"ID":"d17a0b3d-7745-44d8-8e53-8faf694ba74d","Type":"ContainerStarted","Data":"1cd00c8ffb5f2edf9ebbcf8a733a0ccbd12dfbe625b4fba78591d9783d65c2bb"}
Sep 17 23:06:08.608091 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:08.607990    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:13.729159 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:13.729111    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6l9vt/must-gather-hmdkt" event={"ID":"d17a0b3d-7745-44d8-8e53-8faf694ba74d","Type":"ContainerStarted","Data":"b2a588b97a3e37e6b6a7e976bc6f5db2b46e01a97e84f713684cd6100b44560e"}
Sep 17 23:06:14.148239 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:14.148198    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:14.248772 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:14.248734    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:14.738098 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:14.738045    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6l9vt/must-gather-hmdkt" event={"ID":"d17a0b3d-7745-44d8-8e53-8faf694ba74d","Type":"ContainerStarted","Data":"3f049f112544cc37f7e83526b97a943e8a3c0ae66c18f92fbed05d90bf8a6acd"}
Sep 17 23:06:14.754100 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:14.754046    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6l9vt/must-gather-hmdkt" podStartSLOduration=2.5501256530000003 podStartE2EDuration="8.754032407s" totalImagesPullingTime="6.203906754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:06:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:14.753922983 +0000 UTC m=+4076.199968257" watchObservedRunningTime="2026-09-17 23:06:14.754032407 +0000 UTC m=+4076.200077679"
Sep 17 23:06:29.790746 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:29.790707    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-2q5j8_296255f9-9ebf-49bb-8fcb-be7c8b392d12/istio-proxy/0.log"
Sep 17 23:06:29.794664 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:29.794639    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-2q5j8_296255f9-9ebf-49bb-8fcb-be7c8b392d12/istio-proxy/1.log"
Sep 17 23:06:29.807054 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:29.807031    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-775b7455c8-brwr9_845db971-2b15-4713-8f19-3e9e851b0d37/router/0.log"
Sep 17 23:06:30.519030 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:30.518993    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-2q5j8_296255f9-9ebf-49bb-8fcb-be7c8b392d12/istio-proxy/0.log"
Sep 17 23:06:30.522961 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:30.522938    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-2q5j8_296255f9-9ebf-49bb-8fcb-be7c8b392d12/istio-proxy/1.log"
Sep 17 23:06:30.536970 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:30.536941    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-775b7455c8-brwr9_845db971-2b15-4713-8f19-3e9e851b0d37/router/0.log"
Sep 17 23:06:31.354526 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:31.354496    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-ndwgf_20a39873-82d6-4f52-8b6e-a2bbf66ae0e2/manager/0.log"
Sep 17 23:06:31.463137 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:31.463096    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-88zj2_7a22ea5b-8d83-4dee-9a49-9183acd54d19/manager/0.log"
Sep 17 23:06:31.472527 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:31.472506    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-t2dgh_5463fd22-38ab-45a8-862b-6d143f816cd6/limitador/0.log"
Sep 17 23:06:36.878462 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:36.878429    2555 generic.go:356] "Generic (PLEG): container finished" podID="d17a0b3d-7745-44d8-8e53-8faf694ba74d" containerID="b2a588b97a3e37e6b6a7e976bc6f5db2b46e01a97e84f713684cd6100b44560e" exitCode=0
Sep 17 23:06:36.878915 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:36.878509    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6l9vt/must-gather-hmdkt" event={"ID":"d17a0b3d-7745-44d8-8e53-8faf694ba74d","Type":"ContainerDied","Data":"b2a588b97a3e37e6b6a7e976bc6f5db2b46e01a97e84f713684cd6100b44560e"}
Sep 17 23:06:36.878915 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:36.878815    2555 scope.go:117] "RemoveContainer" containerID="b2a588b97a3e37e6b6a7e976bc6f5db2b46e01a97e84f713684cd6100b44560e"
Sep 17 23:06:36.910661 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:36.910636    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-6l9vt_must-gather-hmdkt_d17a0b3d-7745-44d8-8e53-8faf694ba74d/gather/0.log"
Sep 17 23:06:37.335727 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:37.335643    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:37.503461 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:37.503425    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:37.606123 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:37.606045    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:40.359426 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:40.359395    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vl469_84426995-45a3-420d-81b6-2676aba8686b/global-pull-secret-syncer/0.log"
Sep 17 23:06:40.415292 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:40.415256    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log"
Sep 17 23:06:40.617523 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:40.617437    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-157.ec2.internal_745ac86812f97f62360d667f4353a619/haproxy/0.log"
Sep 17 23:06:42.317588 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:42.317550    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-6l9vt/must-gather-hmdkt"]
Sep 17 23:06:42.318141 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:42.317779    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-6l9vt/must-gather-hmdkt" podUID="d17a0b3d-7745-44d8-8e53-8faf694ba74d" containerName="copy" containerID="cri-o://3f049f112544cc37f7e83526b97a943e8a3c0ae66c18f92fbed05d90bf8a6acd" gracePeriod=2
Sep 17 23:06:42.320012 ip-10-0-140-157 kubenswrapper[2555]: E0917 23:06:42.319976    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-hmdkt\" is forbidden: User \"system:node:ip-10-0-140-157.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-6l9vt\": no relationship found between node 'ip-10-0-140-157.ec2.internal' and this object" podUID="d17a0b3d-7745-44d8-8e53-8faf694ba74d" pod="openshift-must-gather-6l9vt/must-gather-hmdkt"
Sep 17 23:06:42.320238 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:42.320218    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-6l9vt/must-gather-hmdkt"]
Sep 17 23:06:42.557700 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:42.557678    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-6l9vt_must-gather-hmdkt_d17a0b3d-7745-44d8-8e53-8faf694ba74d/copy/0.log"
Sep 17 23:06:42.560173 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:42.560154    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6l9vt/must-gather-hmdkt"
Sep 17 23:06:42.562173 ip-10-0-140-157 kubenswrapper[2555]: E0917 23:06:42.562143    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-hmdkt\" is forbidden: User \"system:node:ip-10-0-140-157.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-6l9vt\": no relationship found between node 'ip-10-0-140-157.ec2.internal' and this object" podUID="d17a0b3d-7745-44d8-8e53-8faf694ba74d" pod="openshift-must-gather-6l9vt/must-gather-hmdkt"
Sep 17 23:06:42.621247 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:42.621138    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d17a0b3d-7745-44d8-8e53-8faf694ba74d-must-gather-output\") pod \"d17a0b3d-7745-44d8-8e53-8faf694ba74d\" (UID: \"d17a0b3d-7745-44d8-8e53-8faf694ba74d\") "
Sep 17 23:06:42.621388 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:42.621255    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zllfm\" (UniqueName: \"kubernetes.io/projected/d17a0b3d-7745-44d8-8e53-8faf694ba74d-kube-api-access-zllfm\") pod \"d17a0b3d-7745-44d8-8e53-8faf694ba74d\" (UID: \"d17a0b3d-7745-44d8-8e53-8faf694ba74d\") "
Sep 17 23:06:42.623511 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:42.623486    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d17a0b3d-7745-44d8-8e53-8faf694ba74d-kube-api-access-zllfm" (OuterVolumeSpecName: "kube-api-access-zllfm") pod "d17a0b3d-7745-44d8-8e53-8faf694ba74d" (UID: "d17a0b3d-7745-44d8-8e53-8faf694ba74d"). InnerVolumeSpecName "kube-api-access-zllfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:06:42.626829 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:42.626801    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d17a0b3d-7745-44d8-8e53-8faf694ba74d-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "d17a0b3d-7745-44d8-8e53-8faf694ba74d" (UID: "d17a0b3d-7745-44d8-8e53-8faf694ba74d"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:06:42.680035 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:42.679999    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d17a0b3d-7745-44d8-8e53-8faf694ba74d" path="/var/lib/kubelet/pods/d17a0b3d-7745-44d8-8e53-8faf694ba74d/volumes"
Sep 17 23:06:42.722769 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:42.722723    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zllfm\" (UniqueName: \"kubernetes.io/projected/d17a0b3d-7745-44d8-8e53-8faf694ba74d-kube-api-access-zllfm\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 23:06:42.722769 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:42.722762    2555 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d17a0b3d-7745-44d8-8e53-8faf694ba74d-must-gather-output\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 23:06:42.913676 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:42.913646    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-6l9vt_must-gather-hmdkt_d17a0b3d-7745-44d8-8e53-8faf694ba74d/copy/0.log"
Sep 17 23:06:42.914022 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:42.913992    2555 generic.go:356] "Generic (PLEG): container finished" podID="d17a0b3d-7745-44d8-8e53-8faf694ba74d" containerID="3f049f112544cc37f7e83526b97a943e8a3c0ae66c18f92fbed05d90bf8a6acd" exitCode=143
Sep 17 23:06:42.914079 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:42.914051    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6l9vt/must-gather-hmdkt"
Sep 17 23:06:42.914116 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:42.914077    2555 scope.go:117] "RemoveContainer" containerID="3f049f112544cc37f7e83526b97a943e8a3c0ae66c18f92fbed05d90bf8a6acd"
Sep 17 23:06:42.922680 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:42.922660    2555 scope.go:117] "RemoveContainer" containerID="b2a588b97a3e37e6b6a7e976bc6f5db2b46e01a97e84f713684cd6100b44560e"
Sep 17 23:06:42.936894 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:42.936876    2555 scope.go:117] "RemoveContainer" containerID="3f049f112544cc37f7e83526b97a943e8a3c0ae66c18f92fbed05d90bf8a6acd"
Sep 17 23:06:42.937157 ip-10-0-140-157 kubenswrapper[2555]: E0917 23:06:42.937130    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f049f112544cc37f7e83526b97a943e8a3c0ae66c18f92fbed05d90bf8a6acd\": container with ID starting with 3f049f112544cc37f7e83526b97a943e8a3c0ae66c18f92fbed05d90bf8a6acd not found: ID does not exist" containerID="3f049f112544cc37f7e83526b97a943e8a3c0ae66c18f92fbed05d90bf8a6acd"
Sep 17 23:06:42.937242 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:42.937163    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f049f112544cc37f7e83526b97a943e8a3c0ae66c18f92fbed05d90bf8a6acd"} err="failed to get container status \"3f049f112544cc37f7e83526b97a943e8a3c0ae66c18f92fbed05d90bf8a6acd\": rpc error: code = NotFound desc = could not find container \"3f049f112544cc37f7e83526b97a943e8a3c0ae66c18f92fbed05d90bf8a6acd\": container with ID starting with 3f049f112544cc37f7e83526b97a943e8a3c0ae66c18f92fbed05d90bf8a6acd not found: ID does not exist"
Sep 17 23:06:42.937242 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:42.937207    2555 scope.go:117] "RemoveContainer" containerID="b2a588b97a3e37e6b6a7e976bc6f5db2b46e01a97e84f713684cd6100b44560e"
Sep 17 23:06:42.937487 ip-10-0-140-157 kubenswrapper[2555]: E0917 23:06:42.937472    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2a588b97a3e37e6b6a7e976bc6f5db2b46e01a97e84f713684cd6100b44560e\": container with ID starting with b2a588b97a3e37e6b6a7e976bc6f5db2b46e01a97e84f713684cd6100b44560e not found: ID does not exist" containerID="b2a588b97a3e37e6b6a7e976bc6f5db2b46e01a97e84f713684cd6100b44560e"
Sep 17 23:06:42.937541 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:42.937490    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2a588b97a3e37e6b6a7e976bc6f5db2b46e01a97e84f713684cd6100b44560e"} err="failed to get container status \"b2a588b97a3e37e6b6a7e976bc6f5db2b46e01a97e84f713684cd6100b44560e\": rpc error: code = NotFound desc = could not find container \"b2a588b97a3e37e6b6a7e976bc6f5db2b46e01a97e84f713684cd6100b44560e\": container with ID starting with b2a588b97a3e37e6b6a7e976bc6f5db2b46e01a97e84f713684cd6100b44560e not found: ID does not exist"
Sep 17 23:06:43.129320 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:43.129285    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:43.230382 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:43.230291    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8sfbd_65875e8d-a9e5-4031-bedd-b304efbafde8/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:45.827954 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:45.827921    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-ndwgf_20a39873-82d6-4f52-8b6e-a2bbf66ae0e2/manager/0.log"
Sep 17 23:06:46.019695 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:46.019658    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-88zj2_7a22ea5b-8d83-4dee-9a49-9183acd54d19/manager/0.log"
Sep 17 23:06:46.043083 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:46.043058    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-t2dgh_5463fd22-38ab-45a8-862b-6d143f816cd6/limitador/0.log"
Sep 17 23:06:47.438673 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:47.438635    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-zddx8_44044918-464c-4ce3-9ace-e88ffeb8b926/cluster-monitoring-operator/0.log"
Sep 17 23:06:47.842390 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:47.842281    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jxgq6_1d57efce-da57-4e90-8063-9781be3cda5b/node-exporter/0.log"
Sep 17 23:06:47.869300 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:47.869264    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jxgq6_1d57efce-da57-4e90-8063-9781be3cda5b/kube-rbac-proxy/0.log"
Sep 17 23:06:47.891869 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:47.891843    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jxgq6_1d57efce-da57-4e90-8063-9781be3cda5b/init-textfile/0.log"
Sep 17 23:06:48.255427 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:48.255378    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-b64656858-jwsq5_86c7e467-08dd-4134-8557-3748db54ab9d/telemeter-client/0.log"
Sep 17 23:06:48.278648 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:48.278616    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-b64656858-jwsq5_86c7e467-08dd-4134-8557-3748db54ab9d/reload/0.log"
Sep 17 23:06:48.304406 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:48.304375    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-b64656858-jwsq5_86c7e467-08dd-4134-8557-3748db54ab9d/kube-rbac-proxy/0.log"
Sep 17 23:06:49.473393 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:49.473360    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-2tt9k_d6369f71-f42c-4c68-98b0-bb1dcb9e1e8e/networking-console-plugin/0.log"
Sep 17 23:06:49.916880 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:49.916845    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-r2jh9/perf-node-gather-daemonset-n4jpz"]
Sep 17 23:06:49.917238 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:49.917224    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d17a0b3d-7745-44d8-8e53-8faf694ba74d" containerName="copy"
Sep 17 23:06:49.917294 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:49.917240    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d17a0b3d-7745-44d8-8e53-8faf694ba74d" containerName="copy"
Sep 17 23:06:49.917294 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:49.917263    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d17a0b3d-7745-44d8-8e53-8faf694ba74d" containerName="gather"
Sep 17 23:06:49.917294 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:49.917268    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d17a0b3d-7745-44d8-8e53-8faf694ba74d" containerName="gather"
Sep 17 23:06:49.917401 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:49.917332    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d17a0b3d-7745-44d8-8e53-8faf694ba74d" containerName="copy"
Sep 17 23:06:49.917401 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:49.917341    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d17a0b3d-7745-44d8-8e53-8faf694ba74d" containerName="gather"
Sep 17 23:06:49.923394 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:49.923372    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r2jh9/perf-node-gather-daemonset-n4jpz"
Sep 17 23:06:49.925888 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:49.925857    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-r2jh9\"/\"default-dockercfg-gmh4m\""
Sep 17 23:06:49.926404 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:49.926377    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r2jh9\"/\"openshift-service-ca.crt\""
Sep 17 23:06:49.926404 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:49.926390    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r2jh9\"/\"kube-root-ca.crt\""
Sep 17 23:06:49.928962 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:49.928940    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r2jh9/perf-node-gather-daemonset-n4jpz"]
Sep 17 23:06:49.973731 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:49.973698    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-7njvj_a324c9ab-5d42-4d31-8ff7-449f619be774/console-operator/1.log"
Sep 17 23:06:49.980024 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:49.979994    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-7njvj_a324c9ab-5d42-4d31-8ff7-449f619be774/console-operator/2.log"
Sep 17 23:06:49.984619 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:49.984597    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0db5fbb2-aa9e-435f-92ff-5a6f13e88d1f-proc\") pod \"perf-node-gather-daemonset-n4jpz\" (UID: \"0db5fbb2-aa9e-435f-92ff-5a6f13e88d1f\") " pod="openshift-must-gather-r2jh9/perf-node-gather-daemonset-n4jpz"
Sep 17 23:06:49.984737 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:49.984640    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0db5fbb2-aa9e-435f-92ff-5a6f13e88d1f-lib-modules\") pod \"perf-node-gather-daemonset-n4jpz\" (UID: \"0db5fbb2-aa9e-435f-92ff-5a6f13e88d1f\") " pod="openshift-must-gather-r2jh9/perf-node-gather-daemonset-n4jpz"
Sep 17 23:06:49.984737 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:49.984657    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g82f\" (UniqueName: \"kubernetes.io/projected/0db5fbb2-aa9e-435f-92ff-5a6f13e88d1f-kube-api-access-7g82f\") pod \"perf-node-gather-daemonset-n4jpz\" (UID: \"0db5fbb2-aa9e-435f-92ff-5a6f13e88d1f\") " pod="openshift-must-gather-r2jh9/perf-node-gather-daemonset-n4jpz"
Sep 17 23:06:49.984737 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:49.984721    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0db5fbb2-aa9e-435f-92ff-5a6f13e88d1f-podres\") pod \"perf-node-gather-daemonset-n4jpz\" (UID: \"0db5fbb2-aa9e-435f-92ff-5a6f13e88d1f\") " pod="openshift-must-gather-r2jh9/perf-node-gather-daemonset-n4jpz"
Sep 17 23:06:49.984859 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:49.984758    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0db5fbb2-aa9e-435f-92ff-5a6f13e88d1f-sys\") pod \"perf-node-gather-daemonset-n4jpz\" (UID: \"0db5fbb2-aa9e-435f-92ff-5a6f13e88d1f\") " pod="openshift-must-gather-r2jh9/perf-node-gather-daemonset-n4jpz"
Sep 17 23:06:50.086172 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:50.086130    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0db5fbb2-aa9e-435f-92ff-5a6f13e88d1f-podres\") pod \"perf-node-gather-daemonset-n4jpz\" (UID: \"0db5fbb2-aa9e-435f-92ff-5a6f13e88d1f\") " pod="openshift-must-gather-r2jh9/perf-node-gather-daemonset-n4jpz"
Sep 17 23:06:50.086403 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:50.086208    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0db5fbb2-aa9e-435f-92ff-5a6f13e88d1f-sys\") pod \"perf-node-gather-daemonset-n4jpz\" (UID: \"0db5fbb2-aa9e-435f-92ff-5a6f13e88d1f\") " pod="openshift-must-gather-r2jh9/perf-node-gather-daemonset-n4jpz"
Sep 17 23:06:50.086403 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:50.086269    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0db5fbb2-aa9e-435f-92ff-5a6f13e88d1f-proc\") pod \"perf-node-gather-daemonset-n4jpz\" (UID: \"0db5fbb2-aa9e-435f-92ff-5a6f13e88d1f\") " pod="openshift-must-gather-r2jh9/perf-node-gather-daemonset-n4jpz"
Sep 17 23:06:50.086403 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:50.086305    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0db5fbb2-aa9e-435f-92ff-5a6f13e88d1f-lib-modules\") pod \"perf-node-gather-daemonset-n4jpz\" (UID: \"0db5fbb2-aa9e-435f-92ff-5a6f13e88d1f\") " pod="openshift-must-gather-r2jh9/perf-node-gather-daemonset-n4jpz"
Sep 17 23:06:50.086403 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:50.086321    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g82f\" (UniqueName: \"kubernetes.io/projected/0db5fbb2-aa9e-435f-92ff-5a6f13e88d1f-kube-api-access-7g82f\") pod \"perf-node-gather-daemonset-n4jpz\" (UID: \"0db5fbb2-aa9e-435f-92ff-5a6f13e88d1f\") " pod="openshift-must-gather-r2jh9/perf-node-gather-daemonset-n4jpz"
Sep 17 23:06:50.086403 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:50.086339    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0db5fbb2-aa9e-435f-92ff-5a6f13e88d1f-sys\") pod \"perf-node-gather-daemonset-n4jpz\" (UID: \"0db5fbb2-aa9e-435f-92ff-5a6f13e88d1f\") " pod="openshift-must-gather-r2jh9/perf-node-gather-daemonset-n4jpz"
Sep 17 23:06:50.086403 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:50.086339    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0db5fbb2-aa9e-435f-92ff-5a6f13e88d1f-podres\") pod \"perf-node-gather-daemonset-n4jpz\" (UID: \"0db5fbb2-aa9e-435f-92ff-5a6f13e88d1f\") " pod="openshift-must-gather-r2jh9/perf-node-gather-daemonset-n4jpz"
Sep 17 23:06:50.086628 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:50.086404    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0db5fbb2-aa9e-435f-92ff-5a6f13e88d1f-proc\") pod \"perf-node-gather-daemonset-n4jpz\" (UID: \"0db5fbb2-aa9e-435f-92ff-5a6f13e88d1f\") " pod="openshift-must-gather-r2jh9/perf-node-gather-daemonset-n4jpz"
Sep 17 23:06:50.086628 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:50.086530    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0db5fbb2-aa9e-435f-92ff-5a6f13e88d1f-lib-modules\") pod \"perf-node-gather-daemonset-n4jpz\" (UID: \"0db5fbb2-aa9e-435f-92ff-5a6f13e88d1f\") " pod="openshift-must-gather-r2jh9/perf-node-gather-daemonset-n4jpz"
Sep 17 23:06:50.094287 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:50.094254    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g82f\" (UniqueName: \"kubernetes.io/projected/0db5fbb2-aa9e-435f-92ff-5a6f13e88d1f-kube-api-access-7g82f\") pod \"perf-node-gather-daemonset-n4jpz\" (UID: \"0db5fbb2-aa9e-435f-92ff-5a6f13e88d1f\") " pod="openshift-must-gather-r2jh9/perf-node-gather-daemonset-n4jpz"
Sep 17 23:06:50.234714 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:50.234625    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r2jh9/perf-node-gather-daemonset-n4jpz"
Sep 17 23:06:50.428617 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:50.428588    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-84b584ccc8-th4zj_8ecc6ad2-d7e3-4b5f-8bc8-f1eacb39e42f/console/0.log"
Sep 17 23:06:50.467089 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:50.467058    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-n2h8c_1c48dd88-220b-45c2-a407-1240b4294030/download-server/0.log"
Sep 17 23:06:50.505708 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:50.505677    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r2jh9/perf-node-gather-daemonset-n4jpz"]
Sep 17 23:06:50.506923 ip-10-0-140-157 kubenswrapper[2555]: W0917 23:06:50.506895    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0db5fbb2_aa9e_435f_92ff_5a6f13e88d1f.slice/crio-a7bcbab1121df76831e5b7ac18f0a65817889653a82f80624c1367b82a934eeb WatchSource:0}: Error finding container a7bcbab1121df76831e5b7ac18f0a65817889653a82f80624c1367b82a934eeb: Status 404 returned error can't find the container with id a7bcbab1121df76831e5b7ac18f0a65817889653a82f80624c1367b82a934eeb
Sep 17 23:06:50.959664 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:50.959624    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r2jh9/perf-node-gather-daemonset-n4jpz" event={"ID":"0db5fbb2-aa9e-435f-92ff-5a6f13e88d1f","Type":"ContainerStarted","Data":"d545261ec423f83b20a7c9d64448670e23b68429eb209111fb56e21346f5c0f1"}
Sep 17 23:06:50.959880 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:50.959786    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-r2jh9/perf-node-gather-daemonset-n4jpz"
Sep 17 23:06:50.959880 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:50.959826    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r2jh9/perf-node-gather-daemonset-n4jpz" event={"ID":"0db5fbb2-aa9e-435f-92ff-5a6f13e88d1f","Type":"ContainerStarted","Data":"a7bcbab1121df76831e5b7ac18f0a65817889653a82f80624c1367b82a934eeb"}
Sep 17 23:06:50.975848 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:50.975803    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-r2jh9/perf-node-gather-daemonset-n4jpz" podStartSLOduration=1.975787957 podStartE2EDuration="1.975787957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:50.975646856 +0000 UTC m=+4112.421692136" watchObservedRunningTime="2026-09-17 23:06:50.975787957 +0000 UTC m=+4112.421833228"
Sep 17 23:06:51.026978 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:51.026953    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-x9xmw_da853081-8619-4926-8610-89b3cb4d9557/volume-data-source-validator/0.log"
Sep 17 23:06:52.002173 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:52.002139    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7jzhx_8c8d2e30-748c-4f7c-a848-f5f0a327726a/dns/0.log"
Sep 17 23:06:52.025862 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:52.025837    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7jzhx_8c8d2e30-748c-4f7c-a848-f5f0a327726a/kube-rbac-proxy/0.log"
Sep 17 23:06:52.205033 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:52.204996    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sggv9_cc6119ef-2509-4a37-9edb-1c6da2fdef14/dns-node-resolver/0.log"
Sep 17 23:06:52.786558 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:52.786528    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rdqkk_838fe5eb-ddc1-4cea-8a3c-0c5f0e203174/node-ca/0.log"
Sep 17 23:06:53.786097 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:53.786058    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-2q5j8_296255f9-9ebf-49bb-8fcb-be7c8b392d12/istio-proxy/0.log"
Sep 17 23:06:53.789316 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:53.789294    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-2q5j8_296255f9-9ebf-49bb-8fcb-be7c8b392d12/istio-proxy/1.log"
Sep 17 23:06:53.811122 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:53.811093    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-775b7455c8-brwr9_845db971-2b15-4713-8f19-3e9e851b0d37/router/0.log"
Sep 17 23:06:54.325066 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:54.325039    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-f87z5_a53be88f-7ad0-4542-9c47-4ec5728fed7b/serve-healthcheck-canary/0.log"
Sep 17 23:06:54.797576 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:54.797544    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-2kws6_b90aafb6-01ac-4c30-a30a-530c91d1c944/insights-operator/0.log"
Sep 17 23:06:54.798558 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:54.798536    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-2kws6_b90aafb6-01ac-4c30-a30a-530c91d1c944/insights-operator/1.log"
Sep 17 23:06:54.952672 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:54.952645    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pxjbl_ab4de47d-58d7-44d5-b1c8-884f9801c218/kube-rbac-proxy/0.log"
Sep 17 23:06:54.974095 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:54.974073    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pxjbl_ab4de47d-58d7-44d5-b1c8-884f9801c218/exporter/0.log"
Sep 17 23:06:55.001300 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:55.001279    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pxjbl_ab4de47d-58d7-44d5-b1c8-884f9801c218/extractor/0.log"
Sep 17 23:06:56.975584 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:56.975555    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-r2jh9/perf-node-gather-daemonset-n4jpz"
Sep 17 23:06:58.617463 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:58.617438    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-79b679679b-67wr2_c8e4a5c2-827d-4191-9e34-8ce6b6ffda58/manager/0.log"
Sep 17 23:06:58.885861 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:58.885822    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-7c5846b585-w4kcq_3ae29a8f-5039-461a-846e-6215646c155b/manager/0.log"
Sep 17 23:06:58.912710 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:58.912690    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-cwvkh_eb0ca825-2d2f-4bd7-a6eb-34c2d8b09f84/server/0.log"
Sep 17 23:06:59.259829 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:59.259790    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-2q5j8_296255f9-9ebf-49bb-8fcb-be7c8b392d12/istio-proxy/0.log"
Sep 17 23:06:59.264598 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:59.264569    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-2q5j8_296255f9-9ebf-49bb-8fcb-be7c8b392d12/istio-proxy/1.log"
Sep 17 23:06:59.277755 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:59.277733    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-775b7455c8-brwr9_845db971-2b15-4713-8f19-3e9e851b0d37/router/0.log"
Sep 17 23:06:59.501293 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:59.501258    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-vsj9h_bdfe5af3-14c1-48b0-957c-b8f88779fc27/manager/0.log"
Sep 17 23:06:59.544694 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:59.544611    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-2q2fq_a3815a4c-d508-4bda-bb25-9ed3166b9332/s3-init/0.log"
Sep 17 23:06:59.565322 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:59.565294    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-2q2fq_a3815a4c-d508-4bda-bb25-9ed3166b9332/download-hf-model/0.log"
Sep 17 23:06:59.594543 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:06:59.594515    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-4hswx_4c524947-6176-4f5b-88bc-2d971a15e0c1/seaweedfs/0.log"
Sep 17 23:07:00.409881 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:07:00.409851    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-2q5j8_296255f9-9ebf-49bb-8fcb-be7c8b392d12/istio-proxy/0.log"
Sep 17 23:07:00.413331 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:07:00.413310    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-2q5j8_296255f9-9ebf-49bb-8fcb-be7c8b392d12/istio-proxy/1.log"
Sep 17 23:07:00.428584 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:07:00.428564    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-775b7455c8-brwr9_845db971-2b15-4713-8f19-3e9e851b0d37/router/0.log"
Sep 17 23:07:04.670337 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:07:04.670246    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-pw29d_c62743be-af33-4043-abb5-a7afcc097a8b/kube-storage-version-migrator-operator/1.log"
Sep 17 23:07:04.671688 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:07:04.671666    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-pw29d_c62743be-af33-4043-abb5-a7afcc097a8b/kube-storage-version-migrator-operator/0.log"
Sep 17 23:07:05.547511 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:07:05.547484    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9sgr6_f2ab99ad-b85d-494c-967c-c4e75d08553e/kube-multus-additional-cni-plugins/0.log"
Sep 17 23:07:05.569743 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:07:05.569698    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9sgr6_f2ab99ad-b85d-494c-967c-c4e75d08553e/egress-router-binary-copy/0.log"
Sep 17 23:07:05.592117 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:07:05.592092    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9sgr6_f2ab99ad-b85d-494c-967c-c4e75d08553e/cni-plugins/0.log"
Sep 17 23:07:05.613073 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:07:05.613049    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9sgr6_f2ab99ad-b85d-494c-967c-c4e75d08553e/bond-cni-plugin/0.log"
Sep 17 23:07:05.633892 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:07:05.633869    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9sgr6_f2ab99ad-b85d-494c-967c-c4e75d08553e/routeoverride-cni/0.log"
Sep 17 23:07:05.654297 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:07:05.654279    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9sgr6_f2ab99ad-b85d-494c-967c-c4e75d08553e/whereabouts-cni-bincopy/0.log"
Sep 17 23:07:05.677047 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:07:05.677025    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9sgr6_f2ab99ad-b85d-494c-967c-c4e75d08553e/whereabouts-cni/0.log"
Sep 17 23:07:06.048380 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:07:06.048320    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-cn4mv_66effb77-e868-4e17-9bab-d9ed02f73237/kube-multus/0.log"
Sep 17 23:07:06.229166 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:07:06.229140    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-n7n4d_a236d0c6-0efa-47e2-94c1-f4ca507442c4/network-metrics-daemon/0.log"
Sep 17 23:07:06.249909 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:07:06.249884    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-n7n4d_a236d0c6-0efa-47e2-94c1-f4ca507442c4/kube-rbac-proxy/0.log"
Sep 17 23:07:07.247930 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:07:07.247900    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s7f5_4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf/ovn-controller/0.log"
Sep 17 23:07:07.265653 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:07:07.265624    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s7f5_4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf/ovn-acl-logging/1.log"
Sep 17 23:07:07.265653 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:07:07.265645    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s7f5_4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf/ovn-acl-logging/0.log"
Sep 17 23:07:07.284745 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:07:07.284722    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s7f5_4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf/kube-rbac-proxy-node/0.log"
Sep 17 23:07:07.307407 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:07:07.307387    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s7f5_4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 23:07:07.327960 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:07:07.327939    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s7f5_4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf/northd/0.log"
Sep 17 23:07:07.354952 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:07:07.354928    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s7f5_4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf/nbdb/0.log"
Sep 17 23:07:07.377537 ip-10-0-140-157 kubenswrapper[2555]: I0917 23:07:07.377517    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s7f5_4e6c570d-7932-49e8-b0dc-6cf8dd77d1cf/sbdb/0.log"