Sep 17 16:41:52.368889 ip-10-0-141-66 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 16:41:56.137263 ip-10-0-141-66 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 16:41:56.137263 ip-10-0-141-66 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 16:41:56.137263 ip-10-0-141-66 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 16:41:56.137263 ip-10-0-141-66 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 16:41:56.137263 ip-10-0-141-66 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 16:41:56.137968 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.137348    2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 16:41:56.141806 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141770    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 16:41:56.141806 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141792    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 16:41:56.141806 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141796    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 16:41:56.141806 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141800    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 16:41:56.141806 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141803    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 16:41:56.141806 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141808    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 16:41:56.141806 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141811    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 16:41:56.141806 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141814    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 16:41:56.141806 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141817    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 16:41:56.141806 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141819    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 16:41:56.141806 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141822    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 16:41:56.142188 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141825    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 16:41:56.142188 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141828    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 16:41:56.142188 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141830    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 16:41:56.142188 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141833    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 16:41:56.142188 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141835    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 16:41:56.142188 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141839    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 16:41:56.142188 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141843    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 16:41:56.142188 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141845    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 16:41:56.142188 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141848    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 16:41:56.142188 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141850    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 16:41:56.142188 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141857    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 16:41:56.142188 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141860    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 16:41:56.142188 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141862    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 16:41:56.142188 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141866    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 16:41:56.142188 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141868    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 16:41:56.142188 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141870    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 16:41:56.142188 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141873    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 16:41:56.142188 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141876    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 16:41:56.142188 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141878    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 16:41:56.142188 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141881    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 16:41:56.142697 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141883    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 16:41:56.142697 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141886    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 16:41:56.142697 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141888    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 16:41:56.142697 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141891    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 16:41:56.142697 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141893    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 16:41:56.142697 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141896    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 16:41:56.142697 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141898    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 16:41:56.142697 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141900    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 16:41:56.142697 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141902    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 16:41:56.142697 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141905    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 16:41:56.142697 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141907    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 16:41:56.142697 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141910    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 16:41:56.142697 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141912    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 16:41:56.142697 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141915    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 16:41:56.142697 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141918    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 16:41:56.142697 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141921    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 16:41:56.142697 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141923    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 16:41:56.142697 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141925    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 16:41:56.142697 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141928    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 16:41:56.142697 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141930    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 16:41:56.143199 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141933    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 16:41:56.143199 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141936    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 16:41:56.143199 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141938    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 16:41:56.143199 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141942    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 16:41:56.143199 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141947    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 16:41:56.143199 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141950    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 16:41:56.143199 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141953    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 16:41:56.143199 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141956    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 16:41:56.143199 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141958    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 16:41:56.143199 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141961    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 16:41:56.143199 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141963    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 16:41:56.143199 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141966    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 16:41:56.143199 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141968    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 16:41:56.143199 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141970    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 16:41:56.143199 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141973    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 16:41:56.143199 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141976    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 16:41:56.143199 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141978    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 16:41:56.143199 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141983    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 16:41:56.143199 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141986    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 16:41:56.143676 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141990    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 16:41:56.143676 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141993    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 16:41:56.143676 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141995    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 16:41:56.143676 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.141998    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 16:41:56.143676 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142000    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 16:41:56.143676 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142003    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 16:41:56.143676 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142006    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 16:41:56.143676 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142009    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 16:41:56.143676 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142011    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 16:41:56.143676 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142014    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 16:41:56.143676 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142016    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 16:41:56.143676 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142019    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 16:41:56.143676 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142022    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 16:41:56.143676 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142026    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 16:41:56.143676 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142028    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 16:41:56.143676 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142031    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 16:41:56.143676 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142033    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 16:41:56.143676 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142036    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 16:41:56.143676 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142038    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 16:41:56.143676 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142041    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 16:41:56.144232 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142043    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 16:41:56.144232 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142046    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 16:41:56.144232 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142048    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 16:41:56.144232 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142051    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 16:41:56.144232 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142053    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 16:41:56.144232 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142056    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 16:41:56.144232 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142058    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 16:41:56.144232 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142060    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 16:41:56.144232 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142063    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 16:41:56.144232 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142774    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 16:41:56.144232 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142782    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 16:41:56.144232 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142784    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 16:41:56.144232 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142787    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 16:41:56.144232 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142790    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 16:41:56.144232 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142792    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 16:41:56.144232 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142795    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 16:41:56.144232 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142797    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 16:41:56.144232 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142801    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 16:41:56.144232 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142804    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 16:41:56.144232 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142808    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 16:41:56.144750 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142811    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 16:41:56.144750 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142813    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 16:41:56.144750 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142816    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 16:41:56.144750 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142819    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 16:41:56.144750 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142822    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 16:41:56.144750 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142824    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 16:41:56.144750 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142828    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 16:41:56.144750 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142830    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 16:41:56.144750 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142833    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 16:41:56.144750 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142835    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 16:41:56.144750 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142838    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 16:41:56.144750 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142841    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 16:41:56.144750 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142843    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 16:41:56.144750 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142846    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 16:41:56.144750 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142848    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 16:41:56.144750 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142851    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 16:41:56.144750 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142853    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 16:41:56.144750 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142856    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 16:41:56.144750 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142858    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 16:41:56.145247 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142861    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 16:41:56.145247 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142864    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 16:41:56.145247 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142866    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 16:41:56.145247 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142871    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 16:41:56.145247 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142875    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 16:41:56.145247 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142878    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 16:41:56.145247 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142881    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 16:41:56.145247 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142884    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 16:41:56.145247 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142886    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 16:41:56.145247 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142888    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 16:41:56.145247 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142891    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 16:41:56.145247 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142894    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 16:41:56.145247 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142897    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 16:41:56.145247 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142900    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 16:41:56.145247 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142902    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 16:41:56.145247 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142905    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 16:41:56.145247 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142908    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 16:41:56.145247 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142911    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 16:41:56.145247 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142914    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 16:41:56.145730 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142916    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 16:41:56.145730 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142919    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 16:41:56.145730 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142922    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 16:41:56.145730 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142924    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 16:41:56.145730 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142927    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 16:41:56.145730 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142929    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 16:41:56.145730 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142931    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 16:41:56.145730 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142934    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 16:41:56.145730 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142936    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 16:41:56.145730 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142938    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 16:41:56.145730 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142941    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 16:41:56.145730 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142943    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 16:41:56.145730 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142946    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 16:41:56.145730 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142948    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 16:41:56.145730 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142952    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 16:41:56.145730 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142955    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 16:41:56.145730 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142958    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 16:41:56.145730 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142960    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 16:41:56.145730 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142963    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 16:41:56.146194 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142965    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 16:41:56.146194 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142967    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 16:41:56.146194 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142970    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 16:41:56.146194 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142972    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 16:41:56.146194 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142975    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 16:41:56.146194 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142978    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 16:41:56.146194 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142980    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 16:41:56.146194 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142983    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 16:41:56.146194 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142986    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 16:41:56.146194 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142988    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 16:41:56.146194 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142991    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 16:41:56.146194 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142993    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 16:41:56.146194 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142997    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 16:41:56.146194 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.142999    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 16:41:56.146194 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143002    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 16:41:56.146194 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143005    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 16:41:56.146194 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143007    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 16:41:56.146194 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143009    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 16:41:56.146194 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143012    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 16:41:56.146194 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143014    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 16:41:56.146689 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143016    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 16:41:56.146689 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143019    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 16:41:56.146689 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143021    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 16:41:56.146689 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143023    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 16:41:56.146689 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143026    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 16:41:56.146689 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143028    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 16:41:56.146689 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143031    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 16:41:56.146689 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143033    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 16:41:56.146689 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143035    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 16:41:56.146689 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143038    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 16:41:56.146689 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143040    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 16:41:56.146689 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143456    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 16:41:56.146689 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143463    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 16:41:56.146689 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143466    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 16:41:56.146689 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143470    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 16:41:56.146689 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143473    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 16:41:56.146689 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143476    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 16:41:56.146689 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143479    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 16:41:56.146689 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143482    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 16:41:56.146689 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143484    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 16:41:56.147170 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143487    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 16:41:56.147170 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143490    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 16:41:56.147170 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143492    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 16:41:56.147170 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143495    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 16:41:56.147170 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143497    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 16:41:56.147170 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143500    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 16:41:56.147170 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143502    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 16:41:56.147170 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143505    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 16:41:56.147170 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143508    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 16:41:56.147170 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143510    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 16:41:56.147170 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143513    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 16:41:56.147170 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143515    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 16:41:56.147170 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143518    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 16:41:56.147170 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143520    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 16:41:56.147170 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143522    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 16:41:56.147170 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143525    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 16:41:56.147170 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143527    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 16:41:56.147170 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143529    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 16:41:56.147170 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143532    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 16:41:56.147170 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143534    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 16:41:56.147673 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143536    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 16:41:56.147673 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143539    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 16:41:56.147673 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143541    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 16:41:56.147673 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143544    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 16:41:56.147673 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143546    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 16:41:56.147673 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143549    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 16:41:56.147673 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143551    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 16:41:56.147673 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143553    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 16:41:56.147673 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143556    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 16:41:56.147673 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143561    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 16:41:56.147673 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143564    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 16:41:56.147673 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143566    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 16:41:56.147673 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143569    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 16:41:56.147673 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143571    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 16:41:56.147673 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143574    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 16:41:56.147673 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143576    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 16:41:56.147673 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143579    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 16:41:56.147673 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143581    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 16:41:56.147673 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143583    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 16:41:56.147673 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143593    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 16:41:56.148192 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143596    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 16:41:56.148192 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143599    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 16:41:56.148192 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143602    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 16:41:56.148192 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143604    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 16:41:56.148192 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143607    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 16:41:56.148192 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143609    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 16:41:56.148192 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143611    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 16:41:56.148192 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143614    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 16:41:56.148192 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143616    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 16:41:56.148192 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143619    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 16:41:56.148192 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143621    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 16:41:56.148192 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143624    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 16:41:56.148192 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143626    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 16:41:56.148192 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143629    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 16:41:56.148192 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143631    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 16:41:56.148192 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143633    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 16:41:56.148192 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143636    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 16:41:56.148192 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143638    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 16:41:56.148192 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143640    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 16:41:56.148676 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143643    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 16:41:56.148676 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143647    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 16:41:56.148676 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143650    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 16:41:56.148676 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143653    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 16:41:56.148676 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143656    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 16:41:56.148676 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143658    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 16:41:56.148676 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143663    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 16:41:56.148676 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143667    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 16:41:56.148676 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143670    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 16:41:56.148676 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143673    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 16:41:56.148676 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143676    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 16:41:56.148676 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143678    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 16:41:56.148676 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143681    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 16:41:56.148676 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143683    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 16:41:56.148676 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143687    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 16:41:56.148676 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143689    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 16:41:56.148676 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143692    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 16:41:56.148676 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143694    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 16:41:56.148676 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143697    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 16:41:56.149136 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143699    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 16:41:56.149136 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143701    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 16:41:56.149136 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143704    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 16:41:56.149136 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143706    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 16:41:56.149136 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143710    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 16:41:56.149136 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143714    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 16:41:56.149136 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143717    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 16:41:56.149136 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143720    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 16:41:56.149136 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143722    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 16:41:56.149136 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143728    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 16:41:56.149136 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143731    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 16:41:56.149136 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.143733    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 16:41:56.149136 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143802    2566 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 16:41:56.149136 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143836    2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 16:41:56.149136 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143843    2566 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 16:41:56.149136 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143849    2566 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 16:41:56.149136 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143853    2566 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 16:41:56.149136 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143856    2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 16:41:56.149136 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143860    2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 16:41:56.149136 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143864    2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 16:41:56.149136 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143867    2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 16:41:56.149646 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143870    2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 16:41:56.149646 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143873    2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 16:41:56.149646 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143876    2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 16:41:56.149646 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143879    2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 16:41:56.149646 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143882    2566 flags.go:64] FLAG: --cgroup-root=""
Sep 17 16:41:56.149646 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143885    2566 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 16:41:56.149646 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143888    2566 flags.go:64] FLAG: --client-ca-file=""
Sep 17 16:41:56.149646 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143897    2566 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 16:41:56.149646 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143900    2566 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 16:41:56.149646 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143907    2566 flags.go:64] FLAG: --cluster-domain=""
Sep 17 16:41:56.149646 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143910    2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 16:41:56.149646 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143913    2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 16:41:56.149646 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143916    2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 16:41:56.149646 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143919    2566 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 16:41:56.149646 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143922    2566 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 16:41:56.149646 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143925    2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 16:41:56.149646 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143929    2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 16:41:56.149646 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143932    2566 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 16:41:56.149646 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143935    2566 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 16:41:56.149646 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143937    2566 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 16:41:56.149646 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143941    2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 16:41:56.149646 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143944    2566 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 16:41:56.149646 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143947    2566 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 16:41:56.149646 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143951    2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 16:41:56.150218 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143954    2566 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 16:41:56.150218 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143957    2566 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 16:41:56.150218 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143961    2566 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 16:41:56.150218 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143964    2566 flags.go:64] FLAG: --enable-server="true"
Sep 17 16:41:56.150218 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143967    2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 16:41:56.150218 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143971    2566 flags.go:64] FLAG: --event-burst="100"
Sep 17 16:41:56.150218 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143974    2566 flags.go:64] FLAG: --event-qps="50"
Sep 17 16:41:56.150218 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143977    2566 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 16:41:56.150218 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143980    2566 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 16:41:56.150218 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143983    2566 flags.go:64] FLAG: --eviction-hard=""
Sep 17 16:41:56.150218 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143986    2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 16:41:56.150218 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143989    2566 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 16:41:56.150218 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143992    2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 16:41:56.150218 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143994    2566 flags.go:64] FLAG: --eviction-soft=""
Sep 17 16:41:56.150218 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.143997    2566 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 16:41:56.150218 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144000    2566 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 16:41:56.150218 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144003    2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 16:41:56.150218 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144007    2566 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 16:41:56.150218 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144009    2566 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 16:41:56.150218 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144012    2566 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 16:41:56.150218 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144015    2566 flags.go:64] FLAG: --feature-gates=""
Sep 17 16:41:56.150218 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144019    2566 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 16:41:56.150218 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144022    2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 16:41:56.150218 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144024    2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 16:41:56.150218 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144027    2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 16:41:56.150845 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144030    2566 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 16:41:56.150845 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144033    2566 flags.go:64] FLAG: --help="false"
Sep 17 16:41:56.150845 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144035    2566 flags.go:64] FLAG: --hostname-override="ip-10-0-141-66.ec2.internal"
Sep 17 16:41:56.150845 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144040    2566 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 16:41:56.150845 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144043    2566 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 16:41:56.150845 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144046    2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 16:41:56.150845 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144049    2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 16:41:56.150845 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144052    2566 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 16:41:56.150845 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144055    2566 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 16:41:56.150845 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144058    2566 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 16:41:56.150845 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144061    2566 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 16:41:56.150845 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144064    2566 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 16:41:56.150845 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144066    2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 16:41:56.150845 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144069    2566 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 16:41:56.150845 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144072    2566 flags.go:64] FLAG: --kube-reserved=""
Sep 17 16:41:56.150845 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144075    2566 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 16:41:56.150845 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144078    2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 16:41:56.150845 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144080    2566 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 16:41:56.150845 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144083    2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 16:41:56.150845 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144086    2566 flags.go:64] FLAG: --lock-file=""
Sep 17 16:41:56.150845 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144088    2566 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 16:41:56.150845 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144091    2566 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 16:41:56.150845 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144094    2566 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 16:41:56.150845 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144099    2566 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 16:41:56.151478 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144102    2566 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 16:41:56.151478 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144105    2566 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 16:41:56.151478 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144108    2566 flags.go:64] FLAG: --logging-format="text"
Sep 17 16:41:56.151478 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144110    2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 16:41:56.151478 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144113    2566 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 16:41:56.151478 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144116    2566 flags.go:64] FLAG: --manifest-url=""
Sep 17 16:41:56.151478 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144119    2566 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 16:41:56.151478 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144122    2566 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 16:41:56.151478 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144125    2566 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 16:41:56.151478 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144129    2566 flags.go:64] FLAG: --max-pods="110"
Sep 17 16:41:56.151478 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144131    2566 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 16:41:56.151478 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144135    2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 16:41:56.151478 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144138    2566 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 16:41:56.151478 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144141    2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 16:41:56.151478 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144144    2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 16:41:56.151478 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144154    2566 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 16:41:56.151478 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144159    2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 16:41:56.151478 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144168    2566 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 16:41:56.151478 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144171    2566 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 16:41:56.151478 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144174    2566 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 16:41:56.151478 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144177    2566 flags.go:64] FLAG: --pod-cidr=""
Sep 17 16:41:56.151478 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144180    2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 16:41:56.151478 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144185    2566 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 16:41:56.152076 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144188    2566 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 16:41:56.152076 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144191    2566 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 16:41:56.152076 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144194    2566 flags.go:64] FLAG: --port="10250"
Sep 17 16:41:56.152076 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144197    2566 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 16:41:56.152076 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144200    2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-055cffce96189c678"
Sep 17 16:41:56.152076 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144203    2566 flags.go:64] FLAG: --qos-reserved=""
Sep 17 16:41:56.152076 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144206    2566 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 16:41:56.152076 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144209    2566 flags.go:64] FLAG: --register-node="true"
Sep 17 16:41:56.152076 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144212    2566 flags.go:64] FLAG: --register-with-taints=""
Sep 17 16:41:56.152076 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144215    2566 flags.go:64] FLAG: --registry-burst="10"
Sep 17 16:41:56.152076 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144219    2566 flags.go:64] FLAG: --registry-qps="5"
Sep 17 16:41:56.152076 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144221    2566 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 16:41:56.152076 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144224    2566 flags.go:64] FLAG: --reserved-memory=""
Sep 17 16:41:56.152076 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144228    2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 16:41:56.152076 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144231    2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 16:41:56.152076 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144234    2566 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 16:41:56.152076 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144237    2566 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 16:41:56.152076 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144240    2566 flags.go:64] FLAG: --runonce="false"
Sep 17 16:41:56.152076 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144243    2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 16:41:56.152076 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144246    2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 16:41:56.152076 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144249    2566 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 16:41:56.152076 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144253    2566 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 16:41:56.152076 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144256    2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 16:41:56.152076 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144259    2566 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 16:41:56.152076 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144262    2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 16:41:56.152076 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144265    2566 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 16:41:56.152741 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144268    2566 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 16:41:56.152741 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144271    2566 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 16:41:56.152741 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144274    2566 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 16:41:56.152741 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144277    2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 16:41:56.152741 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144280    2566 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 16:41:56.152741 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144282    2566 flags.go:64] FLAG: --system-cgroups=""
Sep 17 16:41:56.152741 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144285    2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 16:41:56.152741 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144290    2566 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 16:41:56.152741 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144293    2566 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 16:41:56.152741 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144296    2566 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 16:41:56.152741 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144300    2566 flags.go:64] FLAG: --tls-min-version=""
Sep 17 16:41:56.152741 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144302    2566 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 16:41:56.152741 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144305    2566 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 16:41:56.152741 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144308    2566 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 16:41:56.152741 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144311    2566 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 16:41:56.152741 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144314    2566 flags.go:64] FLAG: --v="2"
Sep 17 16:41:56.152741 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144317    2566 flags.go:64] FLAG: --version="false"
Sep 17 16:41:56.152741 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144322    2566 flags.go:64] FLAG: --vmodule=""
Sep 17 16:41:56.152741 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144325    2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 16:41:56.152741 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.144328    2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 16:41:56.152741 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144453    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 16:41:56.152741 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144456    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 16:41:56.152741 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144459    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 16:41:56.152741 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144463    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 16:41:56.153319 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144466    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 16:41:56.153319 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144469    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 16:41:56.153319 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144472    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 16:41:56.153319 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144476    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 16:41:56.153319 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144478    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 16:41:56.153319 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144481    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 16:41:56.153319 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144483    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 16:41:56.153319 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144486    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 16:41:56.153319 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144490    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 16:41:56.153319 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144492    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 16:41:56.153319 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144495    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 16:41:56.153319 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144498    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 16:41:56.153319 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144500    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 16:41:56.153319 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144503    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 16:41:56.153319 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144505    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 16:41:56.153319 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144508    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 16:41:56.153319 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144511    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 16:41:56.153319 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144513    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 16:41:56.153319 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144516    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 16:41:56.153319 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144518    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 16:41:56.153812 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144521    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 16:41:56.153812 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144523    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 16:41:56.153812 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144527    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 16:41:56.153812 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144529    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 16:41:56.153812 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144532    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 16:41:56.153812 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144534    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 16:41:56.153812 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144537    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 16:41:56.153812 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144540    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 16:41:56.153812 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144542    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 16:41:56.153812 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144545    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 16:41:56.153812 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144547    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 16:41:56.153812 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144550    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 16:41:56.153812 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144552    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 16:41:56.153812 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144555    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 16:41:56.153812 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144557    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 16:41:56.153812 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144561    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 16:41:56.153812 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144563    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 16:41:56.153812 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144566    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 16:41:56.153812 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144569    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 16:41:56.153812 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144571    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 16:41:56.154300 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144575    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 16:41:56.154300 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144577    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 16:41:56.154300 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144580    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 16:41:56.154300 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144582    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 16:41:56.154300 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144585    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 16:41:56.154300 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144588    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 16:41:56.154300 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144590    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 16:41:56.154300 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144592    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 16:41:56.154300 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144595    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 16:41:56.154300 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144597    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 16:41:56.154300 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144601    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 16:41:56.154300 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144605    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 16:41:56.154300 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144608    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 16:41:56.154300 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144610    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 16:41:56.154300 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144613    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 16:41:56.154300 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144616    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 16:41:56.154300 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144619    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 16:41:56.154300 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144621    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 16:41:56.154300 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144624    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 16:41:56.154773 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144627    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 16:41:56.154773 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144629    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 16:41:56.154773 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144632    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 16:41:56.154773 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144634    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 16:41:56.154773 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144637    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 16:41:56.154773 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144639    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 16:41:56.154773 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144643    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 16:41:56.154773 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144647    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 16:41:56.154773 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144651    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 16:41:56.154773 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144654    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 16:41:56.154773 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144656    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 16:41:56.154773 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144659    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 16:41:56.154773 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144661    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 16:41:56.154773 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144666    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 16:41:56.154773 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144668    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 16:41:56.154773 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144671    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 16:41:56.154773 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144673    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 16:41:56.154773 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144676    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 16:41:56.154773 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144678    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 16:41:56.154773 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144681    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 16:41:56.155262 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144683    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 16:41:56.155262 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144686    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 16:41:56.155262 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144688    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 16:41:56.155262 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144691    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 16:41:56.155262 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144693    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 16:41:56.155262 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144696    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 16:41:56.155262 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144698    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 16:41:56.155262 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144701    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 16:41:56.155262 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144703    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 16:41:56.155262 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144706    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 16:41:56.155262 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144708    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 16:41:56.155262 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144711    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 16:41:56.155262 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144713    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 16:41:56.155262 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144716    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 16:41:56.155262 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144719    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 16:41:56.155262 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.144721    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 16:41:56.155662 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.145341    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 16:41:56.155662 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.152802    2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 16:41:56.155662 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.152819    2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 16:41:56.155662 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.152842    2566 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 16:41:56.155662 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.152851    2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 16:41:56.155662 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.152922    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 16:41:56.155662 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.152927    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 16:41:56.155662 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.152931    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 16:41:56.155662 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.152935    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 16:41:56.155662 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.152939    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 16:41:56.155662 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.152942    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 16:41:56.155662 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.152946    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 16:41:56.155662 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.152949    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 16:41:56.155662 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.152951    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 16:41:56.155662 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.152954    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 16:41:56.156029 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.152957    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 16:41:56.156029 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.152960    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 16:41:56.156029 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.152963    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 16:41:56.156029 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.152965    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 16:41:56.156029 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.152968    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 16:41:56.156029 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.152971    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 16:41:56.156029 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.152973    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 16:41:56.156029 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.152976    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 16:41:56.156029 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.152980    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 16:41:56.156029 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.152984    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 16:41:56.156029 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.152987    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 16:41:56.156029 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.152990    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 16:41:56.156029 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.152992    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 16:41:56.156029 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.152995    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 16:41:56.156029 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.152998    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 16:41:56.156029 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153001    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 16:41:56.156029 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153004    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 16:41:56.156029 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153006    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 16:41:56.156029 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153009    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 16:41:56.156518 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153014    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 16:41:56.156518 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153017    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 16:41:56.156518 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153020    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 16:41:56.156518 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153022    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 16:41:56.156518 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153025    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 16:41:56.156518 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153027    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 16:41:56.156518 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153030    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 16:41:56.156518 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153032    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 16:41:56.156518 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153035    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 16:41:56.156518 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153037    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 16:41:56.156518 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153040    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 16:41:56.156518 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153042    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 16:41:56.156518 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153045    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 16:41:56.156518 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153047    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 16:41:56.156518 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153050    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 16:41:56.156518 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153052    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 16:41:56.156518 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153054    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 16:41:56.156518 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153057    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 16:41:56.156518 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153059    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 16:41:56.156518 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153062    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 16:41:56.156998 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153064    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 16:41:56.156998 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153067    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 16:41:56.156998 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153069    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 16:41:56.156998 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153072    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 16:41:56.156998 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153074    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 16:41:56.156998 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153077    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 16:41:56.156998 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153079    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 16:41:56.156998 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153082    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 16:41:56.156998 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153085    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 16:41:56.156998 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153087    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 16:41:56.156998 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153090    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 16:41:56.156998 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153092    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 16:41:56.156998 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153095    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 16:41:56.156998 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153099    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 16:41:56.156998 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153101    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 16:41:56.156998 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153105    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 16:41:56.156998 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153107    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 16:41:56.156998 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153110    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 16:41:56.156998 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153112    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 16:41:56.156998 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153115    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 16:41:56.157518 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153117    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 16:41:56.157518 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153120    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 16:41:56.157518 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153122    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 16:41:56.157518 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153125    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 16:41:56.157518 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153127    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 16:41:56.157518 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153130    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 16:41:56.157518 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153132    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 16:41:56.157518 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153135    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 16:41:56.157518 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153137    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 16:41:56.157518 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153140    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 16:41:56.157518 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153143    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 16:41:56.157518 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153145    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 16:41:56.157518 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153148    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 16:41:56.157518 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153150    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 16:41:56.157518 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153153    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 16:41:56.157518 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153155    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 16:41:56.157518 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153158    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 16:41:56.157518 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153161    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 16:41:56.157518 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153165    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 16:41:56.157980 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153167    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 16:41:56.157980 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153170    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 16:41:56.157980 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153173    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 16:41:56.157980 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153175    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 16:41:56.157980 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153178    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 16:41:56.157980 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153181    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 16:41:56.157980 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153185    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 16:41:56.157980 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153187    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 16:41:56.157980 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153190    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 16:41:56.157980 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153193    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 16:41:56.157980 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153195    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 16:41:56.157980 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.153200    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 16:41:56.157980 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153311    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 16:41:56.157980 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153316    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 16:41:56.157980 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153319    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 16:41:56.158361 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153322    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 16:41:56.158361 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153324    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 16:41:56.158361 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153327    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 16:41:56.158361 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153330    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 16:41:56.158361 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153333    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 16:41:56.158361 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153335    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 16:41:56.158361 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153338    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 16:41:56.158361 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153340    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 16:41:56.158361 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153343    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 16:41:56.158361 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153345    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 16:41:56.158361 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153348    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 16:41:56.158361 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153350    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 16:41:56.158361 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153352    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 16:41:56.158361 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153355    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 16:41:56.158361 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153357    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 16:41:56.158361 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153360    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 16:41:56.158361 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153362    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 16:41:56.158361 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153365    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 16:41:56.158361 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153367    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 16:41:56.158832 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153370    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 16:41:56.158832 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153372    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 16:41:56.158832 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153375    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 16:41:56.158832 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153377    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 16:41:56.158832 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153381    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 16:41:56.158832 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153383    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 16:41:56.158832 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153386    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 16:41:56.158832 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153389    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 16:41:56.158832 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153391    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 16:41:56.158832 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153394    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 16:41:56.158832 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153396    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 16:41:56.158832 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153399    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 16:41:56.158832 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153401    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 16:41:56.158832 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153404    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 16:41:56.158832 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153406    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 16:41:56.158832 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153409    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 16:41:56.158832 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153411    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 16:41:56.158832 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153414    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 16:41:56.158832 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153416    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 16:41:56.159283 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153419    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 16:41:56.159283 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153421    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 16:41:56.159283 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153425    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 16:41:56.159283 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153428    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 16:41:56.159283 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153431    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 16:41:56.159283 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153448    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 16:41:56.159283 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153451    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 16:41:56.159283 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153453    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 16:41:56.159283 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153456    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 16:41:56.159283 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153459    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 16:41:56.159283 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153461    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 16:41:56.159283 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153464    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 16:41:56.159283 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153466    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 16:41:56.159283 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153470    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 16:41:56.159283 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153473    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 16:41:56.159283 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153477    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 16:41:56.159283 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153481    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 16:41:56.159283 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153484    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 16:41:56.159283 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153487    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 16:41:56.159732 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153490    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 16:41:56.159732 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153493    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 16:41:56.159732 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153496    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 16:41:56.159732 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153499    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 16:41:56.159732 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153501    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 16:41:56.159732 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153504    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 16:41:56.159732 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153507    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 16:41:56.159732 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153509    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 16:41:56.159732 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153512    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 16:41:56.159732 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153515    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 16:41:56.159732 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153517    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 16:41:56.159732 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153520    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 16:41:56.159732 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153522    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 16:41:56.159732 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153525    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 16:41:56.159732 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153527    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 16:41:56.159732 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153530    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 16:41:56.159732 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153532    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 16:41:56.159732 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153535    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 16:41:56.159732 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153537    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 16:41:56.159732 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153540    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 16:41:56.160241 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153542    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 16:41:56.160241 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153545    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 16:41:56.160241 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153547    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 16:41:56.160241 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153550    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 16:41:56.160241 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153552    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 16:41:56.160241 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153555    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 16:41:56.160241 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153558    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 16:41:56.160241 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153560    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 16:41:56.160241 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153563    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 16:41:56.160241 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153566    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 16:41:56.160241 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153568    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 16:41:56.160241 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153571    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 16:41:56.160241 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153573    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 16:41:56.160241 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153576    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 16:41:56.160241 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153579    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 16:41:56.160241 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153581    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 16:41:56.160241 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153584    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 16:41:56.160241 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153586    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 16:41:56.160241 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:56.153588    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 16:41:56.160816 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.153593    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 16:41:56.160816 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.153699    2566 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 16:41:56.160816 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.155725    2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 16:41:56.160816 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.156914    2566 server.go:1019] "Starting client certificate rotation"
Sep 17 16:41:56.160816 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.157017    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 16:41:56.160816 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.157614    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 16:41:56.160816 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.159608    2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 16:41:56.161650 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.161632    2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 16:41:56.170572 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.170549    2566 log.go:25] "Validated CRI v1 runtime API"
Sep 17 16:41:56.172055 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.172040    2566 log.go:25] "Validated CRI v1 image API"
Sep 17 16:41:56.172836 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.172817    2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 16:41:56.175160 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.175140    2566 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 f0782958-7c30-4531-bf01-1fc6a0bb792a:/dev/nvme0n1p3 fb53b16f-14fb-4600-a334-f48e9f5c8feb:/dev/nvme0n1p4]
Sep 17 16:41:56.175232 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.175158    2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 16:41:56.181131 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.181026    2566 manager.go:216] Machine: {Timestamp:2026-09-17 16:41:56.180552678 +0000 UTC m=+2.075031808 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3083927 MemoryCapacity:32812150784 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27453cfaa123267444982cd46709f8 SystemUUID:ec27453c-faa1-2326-7444-982cd46709f8 BootID:66a15ada-b9ae-433d-bdb9-7d6dfc500bb6 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e6:c7:c3:83 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e6:c7:c3:83 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ba:11:23:cc:62:58 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812150784 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 16:41:56.181131 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.181129    2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 16:41:56.181226 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.181205    2566 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 16:41:56.182855 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.182829    2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 16:41:56.182983 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.182857    2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-66.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 16:41:56.184194 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.184184    2566 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 16:41:56.184228 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.184197    2566 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 16:41:56.184228 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.184205    2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 16:41:56.184228 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.184217    2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 16:41:56.184309 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.184288    2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 16:41:56.187821 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.187809    2566 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 16:41:56.187930 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.187921    2566 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 16:41:56.187959 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.187954    2566 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 16:41:56.187986 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.187968    2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 16:41:56.187986 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.187979    2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 16:41:56.188035 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.187987    2566 kubelet.go:398] "Adding apiserver pod source"
Sep 17 16:41:56.188035 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.187995    2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 16:41:56.188398 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.188385    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 16:41:56.188447 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.188410    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 16:41:56.191184 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.191160    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 16:41:56.191791 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.191777    2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 16:41:56.194521 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.194507    2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 16:41:56.194583 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.194537    2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 16:41:56.195662 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.195649    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 16:41:56.195708 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.195667    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 16:41:56.195708 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.195673    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 16:41:56.195708 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.195679    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 16:41:56.195708 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.195687    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 16:41:56.195708 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.195693    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 16:41:56.195708 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.195706    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 16:41:56.195863 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.195714    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 16:41:56.195863 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.195721    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 16:41:56.195863 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.195726    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 16:41:56.195863 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.195740    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 16:41:56.195863 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.195745    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 16:41:56.195863 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.195770    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 16:41:56.195863 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.195776    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 16:41:56.196202 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:56.196180    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-66.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 16:41:56.196256 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:56.196225    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 16:41:56.199409 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.199396    2566 server.go:1268] "Started kubelet"
Sep 17 16:41:56.199504 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.199475    2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 16:41:56.199547 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.199493    2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 16:41:56.199582 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.199554    2566 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 16:41:56.200294 ip-10-0-141-66 systemd[1]: Started Kubernetes Kubelet.
Sep 17 16:41:56.200807 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.200571    2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 16:41:56.230092 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:56.230042    2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 16:41:56.230362 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.230344    2566 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 16:41:56.231415 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.231384    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qbkxt"
Sep 17 16:41:56.232107 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.232092    2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-66.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 16:41:56.235292 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.235275    2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 16:41:56.235380 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.235288    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 16:41:56.235465 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:56.235426    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-66.ec2.internal\" not found"
Sep 17 16:41:56.236090 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.235961    2566 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 16:41:56.236090 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.235980    2566 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 16:41:56.236090 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.236055    2566 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 16:41:56.236090 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.236066    2566 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 16:41:56.236090 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.236077    2566 factory.go:55] Registering systemd factory
Sep 17 16:41:56.236090 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.236092    2566 factory.go:223] Registration of the systemd container factory successfully
Sep 17 16:41:56.236410 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.236121    2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 16:41:56.236410 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.236359    2566 factory.go:153] Registering CRI-O factory
Sep 17 16:41:56.236410 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.236376    2566 factory.go:223] Registration of the crio container factory successfully
Sep 17 16:41:56.236570 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.236428    2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 16:41:56.236570 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.236467    2566 factory.go:103] Registering Raw factory
Sep 17 16:41:56.236570 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.236482    2566 manager.go:1196] Started watching for new ooms in manager
Sep 17 16:41:56.237042 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.237030    2566 manager.go:349] Starting recovery of all containers
Sep 17 16:41:56.241054 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.241032    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qbkxt"
Sep 17 16:41:56.245987 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.245971    2566 manager.go:354] Recovery completed
Sep 17 16:41:56.248427 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:56.248402    2566 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-66.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 16:41:56.248427 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:56.248415    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 16:41:56.249582 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:56.248504    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-66.ec2.internal.18d6294dd59c62b5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-66.ec2.internal,UID:ip-10-0-141-66.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-66.ec2.internal,},FirstTimestamp:2026-09-17 16:41:56.199367349 +0000 UTC m=+2.093846478,LastTimestamp:2026-09-17 16:41:56.199367349 +0000 UTC m=+2.093846478,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-66.ec2.internal,}"
Sep 17 16:41:56.250111 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.250099    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 16:41:56.251845 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.251828    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 16:41:56.251917 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.251861    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 16:41:56.251917 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.251876    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasSufficientPID"
Sep 17 16:41:56.252361 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.252345    2566 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 16:41:56.252361 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.252354    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 16:41:56.252361 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.252360    2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 16:41:56.252521 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.252378    2566 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 16:41:56.253123 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.253109    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 16:41:56.253167 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.253140    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 16:41:56.253167 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.253155    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasSufficientPID"
Sep 17 16:41:56.254277 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.254266    2566 policy_none.go:49] "None policy: Start"
Sep 17 16:41:56.254316 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.254283    2566 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 16:41:56.254316 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.254292    2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 16:41:56.256180 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.256169    2566 policy_none.go:47] "Start"
Sep 17 16:41:56.291158 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.291014    2566 manager.go:337] "Starting Device Plugin manager"
Sep 17 16:41:56.309567 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:56.291199    2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 16:41:56.309567 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.291211    2566 server.go:87] "Starting device plugin registration server"
Sep 17 16:41:56.309567 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.291428    2566 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 16:41:56.309567 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.291483    2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 16:41:56.309567 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.291570    2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 16:41:56.309567 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.291642    2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 16:41:56.309567 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.291651    2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 16:41:56.309567 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:56.292139    2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 16:41:56.309567 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:56.292184    2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-66.ec2.internal\" not found"
Sep 17 16:41:56.338937 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.338902    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 16:41:56.340237 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.340220    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 16:41:56.340306 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.340265    2566 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 16:41:56.340306 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.340283    2566 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 16:41:56.340415 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:56.340318    2566 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 16:41:56.342514 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.342497    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 16:41:56.391619 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.391565    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 16:41:56.392387 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.392370    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 16:41:56.392492 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.392407    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 16:41:56.392492 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.392422    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasSufficientPID"
Sep 17 16:41:56.392492 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.392469    2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-66.ec2.internal"
Sep 17 16:41:56.404384 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.404366    2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-66.ec2.internal"
Sep 17 16:41:56.404487 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:56.404387    2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-66.ec2.internal\": node \"ip-10-0-141-66.ec2.internal\" not found"
Sep 17 16:41:56.422688 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:56.422672    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-66.ec2.internal\" not found"
Sep 17 16:41:56.440548 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.440507    2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-66.ec2.internal"]
Sep 17 16:41:56.440606 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.440594    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 16:41:56.441400 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.441386    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 16:41:56.441489 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.441411    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 16:41:56.441489 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.441420    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasSufficientPID"
Sep 17 16:41:56.443521 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.443509    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 16:41:56.443652 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.443638    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal"
Sep 17 16:41:56.443724 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.443670    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 16:41:56.444179 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.444159    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 16:41:56.444224 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.444188    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 16:41:56.444224 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.444200    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasSufficientPID"
Sep 17 16:41:56.445353 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.445331    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 16:41:56.445427 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.445357    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 16:41:56.445427 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.445367    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasSufficientPID"
Sep 17 16:41:56.446222 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.446203    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-66.ec2.internal"
Sep 17 16:41:56.446300 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.446230    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 16:41:56.446876 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.446861    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 16:41:56.446965 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.446889    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 16:41:56.446965 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.446903    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasSufficientPID"
Sep 17 16:41:56.463952 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:56.463933    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-66.ec2.internal\" not found" node="ip-10-0-141-66.ec2.internal"
Sep 17 16:41:56.468405 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:56.468385    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-66.ec2.internal\" not found" node="ip-10-0-141-66.ec2.internal"
Sep 17 16:41:56.523075 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:56.523054    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-66.ec2.internal\" not found"
Sep 17 16:41:56.537705 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.537686    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dcbcba84d85e3930495a76b3760ea833-config\") pod \"kube-apiserver-proxy-ip-10-0-141-66.ec2.internal\" (UID: \"dcbcba84d85e3930495a76b3760ea833\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-66.ec2.internal"
Sep 17 16:41:56.537768 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.537708    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/07f553a18377ccfd877b2a2f13bd91e5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal\" (UID: \"07f553a18377ccfd877b2a2f13bd91e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal"
Sep 17 16:41:56.537768 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.537728    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07f553a18377ccfd877b2a2f13bd91e5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal\" (UID: \"07f553a18377ccfd877b2a2f13bd91e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal"
Sep 17 16:41:56.624053 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:56.624031    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-66.ec2.internal\" not found"
Sep 17 16:41:56.638351 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.638333    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dcbcba84d85e3930495a76b3760ea833-config\") pod \"kube-apiserver-proxy-ip-10-0-141-66.ec2.internal\" (UID: \"dcbcba84d85e3930495a76b3760ea833\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-66.ec2.internal"
Sep 17 16:41:56.638402 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.638373    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/07f553a18377ccfd877b2a2f13bd91e5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal\" (UID: \"07f553a18377ccfd877b2a2f13bd91e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal"
Sep 17 16:41:56.638402 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.638382    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dcbcba84d85e3930495a76b3760ea833-config\") pod \"kube-apiserver-proxy-ip-10-0-141-66.ec2.internal\" (UID: \"dcbcba84d85e3930495a76b3760ea833\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-66.ec2.internal"
Sep 17 16:41:56.638402 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.638397    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07f553a18377ccfd877b2a2f13bd91e5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal\" (UID: \"07f553a18377ccfd877b2a2f13bd91e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal"
Sep 17 16:41:56.638513 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.638427    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07f553a18377ccfd877b2a2f13bd91e5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal\" (UID: \"07f553a18377ccfd877b2a2f13bd91e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal"
Sep 17 16:41:56.638513 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.638429    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/07f553a18377ccfd877b2a2f13bd91e5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal\" (UID: \"07f553a18377ccfd877b2a2f13bd91e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal"
Sep 17 16:41:56.724823 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:56.724763    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-66.ec2.internal\" not found"
Sep 17 16:41:56.769307 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.769279    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal"
Sep 17 16:41:56.773079 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:56.773063    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-66.ec2.internal"
Sep 17 16:41:56.825201 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:56.825176    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-66.ec2.internal\" not found"
Sep 17 16:41:56.925672 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:56.925645    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-66.ec2.internal\" not found"
Sep 17 16:41:57.026262 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:57.026187    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-66.ec2.internal\" not found"
Sep 17 16:41:57.097865 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:57.097842    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 16:41:57.113841 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:57.113818    2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 16:41:57.126715 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:57.126693    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-66.ec2.internal\" not found"
Sep 17 16:41:57.156211 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:57.156189    2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 16:41:57.156602 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:57.156290    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 16:41:57.156602 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:57.156335    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 16:41:57.156602 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:57.156335    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 16:41:57.227766 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:57.227730    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-66.ec2.internal\" not found"
Sep 17 16:41:57.246958 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:57.246922    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 16:36:56 +0000 UTC" deadline="2028-06-04 07:28:03.502854994 +0000 UTC"
Sep 17 16:41:57.246958 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:57.246953    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15014h46m6.255904272s"
Sep 17 16:41:57.327874 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:57.327814    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-66.ec2.internal\" not found"
Sep 17 16:41:57.327874 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:57.327835    2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 16:41:57.335965 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:57.335946    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-66.ec2.internal"
Sep 17 16:41:57.344398 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:57.344378    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 16:41:57.345643 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:57.345630    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal"
Sep 17 16:41:57.358320 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:57.358303    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 16:41:57.374968 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:57.374935    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcbcba84d85e3930495a76b3760ea833.slice/crio-aabec4d49f56a628aa9a790906d3979122699c1e980bc8d723f82c2a817328c9 WatchSource:0}: Error finding container aabec4d49f56a628aa9a790906d3979122699c1e980bc8d723f82c2a817328c9: Status 404 returned error can't find the container with id aabec4d49f56a628aa9a790906d3979122699c1e980bc8d723f82c2a817328c9
Sep 17 16:41:57.375359 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:57.375337    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f553a18377ccfd877b2a2f13bd91e5.slice/crio-621ea3851504c6c63f19b9201b27af39332eb03563835e00c2845be255dc3c79 WatchSource:0}: Error finding container 621ea3851504c6c63f19b9201b27af39332eb03563835e00c2845be255dc3c79: Status 404 returned error can't find the container with id 621ea3851504c6c63f19b9201b27af39332eb03563835e00c2845be255dc3c79
Sep 17 16:41:57.381497 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:57.381481    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:41:58.189233 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.189008    2566 apiserver.go:52] "Watching apiserver"
Sep 17 16:41:58.195451 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.195411    2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 16:41:58.195902 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.195877    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal","openshift-multus/multus-additional-cni-plugins-pcq95","openshift-multus/multus-cq7pn","openshift-multus/network-metrics-daemon-pfvrh","openshift-network-diagnostics/network-check-target-cd6rw","kube-system/global-pull-secret-syncer-xbdp6","kube-system/konnectivity-agent-z8fgx","openshift-cluster-node-tuning-operator/tuned-mtvzk","openshift-image-registry/node-ca-54cc4","openshift-network-operator/iptables-alerter-pnhbn","openshift-ovn-kubernetes/ovnkube-node-gblhr","kube-system/kube-apiserver-proxy-ip-10-0-141-66.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nc2kn","openshift-dns/node-resolver-gnzg4"]
Sep 17 16:41:58.198428 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.198407    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-z8fgx"
Sep 17 16:41:58.200770 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.200748    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nc2kn"
Sep 17 16:41:58.202011 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.201989    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 16:41:58.202154 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.202137    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-pbmvg\""
Sep 17 16:41:58.202222 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.201710    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 16:41:58.204536 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.204516    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.205744 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.205485    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9k5ns\""
Sep 17 16:41:58.205744 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.205534    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 16:41:58.205744 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.205598    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 16:41:58.205744 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.205673    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 16:41:58.205744 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.205723    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfvrh"
Sep 17 16:41:58.206045 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:58.205791    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfvrh" podUID="fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1"
Sep 17 16:41:58.206327 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.206308    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 16:41:58.206628 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.206612    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 16:41:58.206708 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.206643    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 16:41:58.206855 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.206842    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 16:41:58.207600 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.207363    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2pk8w\""
Sep 17 16:41:58.207709 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.207651    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd6rw"
Sep 17 16:41:58.207771 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:58.207706    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd6rw" podUID="ddae5eb7-0626-4732-b8e7-92daf97e56a4"
Sep 17 16:41:58.209860 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.209831    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbdp6"
Sep 17 16:41:58.209940 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:58.209904    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbdp6" podUID="2a0d6b45-1f66-4c14-988a-1ec74eac67ad"
Sep 17 16:41:58.212017 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.211995    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-54cc4"
Sep 17 16:41:58.215504 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.214286    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 16:41:58.215504 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.214489    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 16:41:58.215504 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.214688    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 16:41:58.215504 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.214856    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8g9nh\""
Sep 17 16:41:58.215504 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.215125    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gnzg4"
Sep 17 16:41:58.216753 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.216735    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pcq95"
Sep 17 16:41:58.217286 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.217268    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 16:41:58.217355 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.217329    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5xd54\""
Sep 17 16:41:58.217419 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.217273    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 16:41:58.218977 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.218959    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pnhbn"
Sep 17 16:41:58.220205 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.219696    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wffxt\""
Sep 17 16:41:58.220205 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.219854    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 16:41:58.220205 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.220042    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 16:41:58.220886 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.220860    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 16:41:58.221258 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.221237    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 16:41:58.221686 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.221583    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jkk7q\""
Sep 17 16:41:58.221686 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.221675    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.221929 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.221868    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 16:41:58.223953 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.223939    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 16:41:58.224254 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.224233    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 16:41:58.224342 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.224281    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 16:41:58.224406 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.224392    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 16:41:58.224655 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.224635    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vqnx5\""
Sep 17 16:41:58.224748 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.224667    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 16:41:58.224841 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.224828    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 16:41:58.225608 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.225587    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mtvzk"
Sep 17 16:41:58.227587 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.227565    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 16:41:58.227694 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.227671    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-r2qhr\""
Sep 17 16:41:58.227915 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.227901    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 16:41:58.235982 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.235963    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 16:41:58.237056 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.237041    2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 16:41:58.244871 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.244851    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-multus-conf-dir\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.244950 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.244883    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-host-run-ovn-kubernetes\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.244950 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.244909    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-host-var-lib-cni-multus\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.245065 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.244949    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-host-cni-netd\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.245065 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.244976    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/91477834-6648-4e75-8530-e01bdee5e57e-env-overrides\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.245065 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.244997    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/151db911-0f2b-48c8-8d4f-4fd4e3b67ebb-device-dir\") pod \"aws-ebs-csi-driver-node-nc2kn\" (UID: \"151db911-0f2b-48c8-8d4f-4fd4e3b67ebb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nc2kn"
Sep 17 16:41:58.245065 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.245021    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/367c9d70-f0c6-44ae-b24e-2ac0790b5cb7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pcq95\" (UID: \"367c9d70-f0c6-44ae-b24e-2ac0790b5cb7\") " pod="openshift-multus/multus-additional-cni-plugins-pcq95"
Sep 17 16:41:58.245065 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.245045    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/367c9d70-f0c6-44ae-b24e-2ac0790b5cb7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pcq95\" (UID: \"367c9d70-f0c6-44ae-b24e-2ac0790b5cb7\") " pod="openshift-multus/multus-additional-cni-plugins-pcq95"
Sep 17 16:41:58.245299 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.245067    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpq8z\" (UniqueName: \"kubernetes.io/projected/949596f3-e4a0-4b1e-b1c8-e091aac598a2-kube-api-access-vpq8z\") pod \"iptables-alerter-pnhbn\" (UID: \"949596f3-e4a0-4b1e-b1c8-e091aac598a2\") " pod="openshift-network-operator/iptables-alerter-pnhbn"
Sep 17 16:41:58.245299 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.245128    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4-etc-sysctl-conf\") pod \"tuned-mtvzk\" (UID: \"f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4\") " pod="openshift-cluster-node-tuning-operator/tuned-mtvzk"
Sep 17 16:41:58.245299 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.245171    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb24m\" (UniqueName: \"kubernetes.io/projected/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-kube-api-access-zb24m\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.245299 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.245198    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-etc-openvswitch\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.245299 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.245220    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-run-ovn\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.245299 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.245243    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/151db911-0f2b-48c8-8d4f-4fd4e3b67ebb-socket-dir\") pod \"aws-ebs-csi-driver-node-nc2kn\" (UID: \"151db911-0f2b-48c8-8d4f-4fd4e3b67ebb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nc2kn"
Sep 17 16:41:58.245299 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.245266    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/151db911-0f2b-48c8-8d4f-4fd4e3b67ebb-sys-fs\") pod \"aws-ebs-csi-driver-node-nc2kn\" (UID: \"151db911-0f2b-48c8-8d4f-4fd4e3b67ebb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nc2kn"
Sep 17 16:41:58.245299 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.245288    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4-etc-modprobe-d\") pod \"tuned-mtvzk\" (UID: \"f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4\") " pod="openshift-cluster-node-tuning-operator/tuned-mtvzk"
Sep 17 16:41:58.245715 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.245315    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4-var-lib-kubelet\") pod \"tuned-mtvzk\" (UID: \"f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4\") " pod="openshift-cluster-node-tuning-operator/tuned-mtvzk"
Sep 17 16:41:58.245715 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.245360    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r59pr\" (UniqueName: \"kubernetes.io/projected/f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4-kube-api-access-r59pr\") pod \"tuned-mtvzk\" (UID: \"f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4\") " pod="openshift-cluster-node-tuning-operator/tuned-mtvzk"
Sep 17 16:41:58.245715 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.245384    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a0d6b45-1f66-4c14-988a-1ec74eac67ad-original-pull-secret\") pod \"global-pull-secret-syncer-xbdp6\" (UID: \"2a0d6b45-1f66-4c14-988a-1ec74eac67ad\") " pod="kube-system/global-pull-secret-syncer-xbdp6"
Sep 17 16:41:58.245715 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.245458    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-host-kubelet\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.245715 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.245483    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/91477834-6648-4e75-8530-e01bdee5e57e-ovn-node-metrics-cert\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.245715 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.245508    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/151db911-0f2b-48c8-8d4f-4fd4e3b67ebb-registration-dir\") pod \"aws-ebs-csi-driver-node-nc2kn\" (UID: \"151db911-0f2b-48c8-8d4f-4fd4e3b67ebb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nc2kn"
Sep 17 16:41:58.245715 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.245529    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/151db911-0f2b-48c8-8d4f-4fd4e3b67ebb-etc-selinux\") pod \"aws-ebs-csi-driver-node-nc2kn\" (UID: \"151db911-0f2b-48c8-8d4f-4fd4e3b67ebb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nc2kn"
Sep 17 16:41:58.245715 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.245552    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1-metrics-certs\") pod \"network-metrics-daemon-pfvrh\" (UID: \"fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1\") " pod="openshift-multus/network-metrics-daemon-pfvrh"
Sep 17 16:41:58.245715 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.245585    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/96bf0c2f-8a48-491a-bb3d-73e417abfa26-host\") pod \"node-ca-54cc4\" (UID: \"96bf0c2f-8a48-491a-bb3d-73e417abfa26\") " pod="openshift-image-registry/node-ca-54cc4"
Sep 17 16:41:58.245715 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.245606    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/96bf0c2f-8a48-491a-bb3d-73e417abfa26-serviceca\") pod \"node-ca-54cc4\" (UID: \"96bf0c2f-8a48-491a-bb3d-73e417abfa26\") " pod="openshift-image-registry/node-ca-54cc4"
Sep 17 16:41:58.245715 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.245639    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4-run\") pod \"tuned-mtvzk\" (UID: \"f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4\") " pod="openshift-cluster-node-tuning-operator/tuned-mtvzk"
Sep 17 16:41:58.245715 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.245678    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/91477834-6648-4e75-8530-e01bdee5e57e-ovnkube-config\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.245715 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.245711    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr57f\" (UniqueName: \"kubernetes.io/projected/367c9d70-f0c6-44ae-b24e-2ac0790b5cb7-kube-api-access-zr57f\") pod \"multus-additional-cni-plugins-pcq95\" (UID: \"367c9d70-f0c6-44ae-b24e-2ac0790b5cb7\") " pod="openshift-multus/multus-additional-cni-plugins-pcq95"
Sep 17 16:41:58.246336 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.245733    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdp7c\" (UniqueName: \"kubernetes.io/projected/151db911-0f2b-48c8-8d4f-4fd4e3b67ebb-kube-api-access-gdp7c\") pod \"aws-ebs-csi-driver-node-nc2kn\" (UID: \"151db911-0f2b-48c8-8d4f-4fd4e3b67ebb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nc2kn"
Sep 17 16:41:58.246336 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.245751    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjd2j\" (UniqueName: \"kubernetes.io/projected/fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1-kube-api-access-sjd2j\") pod \"network-metrics-daemon-pfvrh\" (UID: \"fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1\") " pod="openshift-multus/network-metrics-daemon-pfvrh"
Sep 17 16:41:58.246336 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.245779    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfhsx\" (UniqueName: \"kubernetes.io/projected/ddae5eb7-0626-4732-b8e7-92daf97e56a4-kube-api-access-gfhsx\") pod \"network-check-target-cd6rw\" (UID: \"ddae5eb7-0626-4732-b8e7-92daf97e56a4\") " pod="openshift-network-diagnostics/network-check-target-cd6rw"
Sep 17 16:41:58.246336 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.245805    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/367c9d70-f0c6-44ae-b24e-2ac0790b5cb7-system-cni-dir\") pod \"multus-additional-cni-plugins-pcq95\" (UID: \"367c9d70-f0c6-44ae-b24e-2ac0790b5cb7\") " pod="openshift-multus/multus-additional-cni-plugins-pcq95"
Sep 17 16:41:58.246336 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.245842    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4-etc-sysconfig\") pod \"tuned-mtvzk\" (UID: \"f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4\") " pod="openshift-cluster-node-tuning-operator/tuned-mtvzk"
Sep 17 16:41:58.246336 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.245864    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4-host\") pod \"tuned-mtvzk\" (UID: \"f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4\") " pod="openshift-cluster-node-tuning-operator/tuned-mtvzk"
Sep 17 16:41:58.246336 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.245889    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-multus-cni-dir\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.246336 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.245932    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4-lib-modules\") pod \"tuned-mtvzk\" (UID: \"f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4\") " pod="openshift-cluster-node-tuning-operator/tuned-mtvzk"
Sep 17 16:41:58.246336 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.245962    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-cni-binary-copy\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.246336 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.245983    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-multus-daemon-config\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.246336 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.246003    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a0d6b45-1f66-4c14-988a-1ec74eac67ad-dbus\") pod \"global-pull-secret-syncer-xbdp6\" (UID: \"2a0d6b45-1f66-4c14-988a-1ec74eac67ad\") " pod="kube-system/global-pull-secret-syncer-xbdp6"
Sep 17 16:41:58.246336 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.246017    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/17d178d9-d43e-468a-9605-359baa45dc72-hosts-file\") pod \"node-resolver-gnzg4\" (UID: \"17d178d9-d43e-468a-9605-359baa45dc72\") " pod="openshift-dns/node-resolver-gnzg4"
Sep 17 16:41:58.246336 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.246035    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-host-run-netns\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.246336 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.246056    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-host-cni-bin\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.246336 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.246102    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/367c9d70-f0c6-44ae-b24e-2ac0790b5cb7-cnibin\") pod \"multus-additional-cni-plugins-pcq95\" (UID: \"367c9d70-f0c6-44ae-b24e-2ac0790b5cb7\") " pod="openshift-multus/multus-additional-cni-plugins-pcq95"
Sep 17 16:41:58.246336 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.246133    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4-etc-tuned\") pod \"tuned-mtvzk\" (UID: \"f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4\") " pod="openshift-cluster-node-tuning-operator/tuned-mtvzk"
Sep 17 16:41:58.247120 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.246157    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c10930b3-04d3-4b91-8931-a9638b8c5ea7-agent-certs\") pod \"konnectivity-agent-z8fgx\" (UID: \"c10930b3-04d3-4b91-8931-a9638b8c5ea7\") " pod="kube-system/konnectivity-agent-z8fgx"
Sep 17 16:41:58.247120 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.246178    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-host-var-lib-kubelet\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.247120 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.246200    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-run-openvswitch\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.247120 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.246226    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.247120 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.246251    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/91477834-6648-4e75-8530-e01bdee5e57e-ovnkube-script-lib\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.247120 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.246272    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4-etc-systemd\") pod \"tuned-mtvzk\" (UID: \"f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4\") " pod="openshift-cluster-node-tuning-operator/tuned-mtvzk"
Sep 17 16:41:58.247120 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.246301    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-cnibin\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.247120 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.246331    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-multus-socket-dir-parent\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.247120 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.246354    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-etc-kubernetes\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.247120 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.246376    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-log-socket\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.247120 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.246415    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vltn5\" (UniqueName: \"kubernetes.io/projected/91477834-6648-4e75-8530-e01bdee5e57e-kube-api-access-vltn5\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.247120 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.246454    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4-sys\") pod \"tuned-mtvzk\" (UID: \"f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4\") " pod="openshift-cluster-node-tuning-operator/tuned-mtvzk"
Sep 17 16:41:58.247120 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.246476    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-system-cni-dir\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.247120 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.246497    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-host-run-netns\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.247120 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.246518    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-hostroot\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.247120 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.246539    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-host-run-multus-certs\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.247120 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.246578    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-host-slash\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.247844 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.246600    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/151db911-0f2b-48c8-8d4f-4fd4e3b67ebb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nc2kn\" (UID: \"151db911-0f2b-48c8-8d4f-4fd4e3b67ebb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nc2kn"
Sep 17 16:41:58.247844 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.246640    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rml2\" (UniqueName: \"kubernetes.io/projected/96bf0c2f-8a48-491a-bb3d-73e417abfa26-kube-api-access-9rml2\") pod \"node-ca-54cc4\" (UID: \"96bf0c2f-8a48-491a-bb3d-73e417abfa26\") " pod="openshift-image-registry/node-ca-54cc4"
Sep 17 16:41:58.247844 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.246801    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/367c9d70-f0c6-44ae-b24e-2ac0790b5cb7-cni-binary-copy\") pod \"multus-additional-cni-plugins-pcq95\" (UID: \"367c9d70-f0c6-44ae-b24e-2ac0790b5cb7\") " pod="openshift-multus/multus-additional-cni-plugins-pcq95"
Sep 17 16:41:58.247844 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.246982    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4-etc-kubernetes\") pod \"tuned-mtvzk\" (UID: \"f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4\") " pod="openshift-cluster-node-tuning-operator/tuned-mtvzk"
Sep 17 16:41:58.247844 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.247025    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-host-run-k8s-cni-cncf-io\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.247844 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.247060    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17d178d9-d43e-468a-9605-359baa45dc72-tmp-dir\") pod \"node-resolver-gnzg4\" (UID: \"17d178d9-d43e-468a-9605-359baa45dc72\") " pod="openshift-dns/node-resolver-gnzg4"
Sep 17 16:41:58.247844 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.247089    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/949596f3-e4a0-4b1e-b1c8-e091aac598a2-host-slash\") pod \"iptables-alerter-pnhbn\" (UID: \"949596f3-e4a0-4b1e-b1c8-e091aac598a2\") " pod="openshift-network-operator/iptables-alerter-pnhbn"
Sep 17 16:41:58.247844 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.247114    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4-tmp\") pod \"tuned-mtvzk\" (UID: \"f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4\") " pod="openshift-cluster-node-tuning-operator/tuned-mtvzk"
Sep 17 16:41:58.247844 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.247145    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-run-systemd\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.247844 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.247178    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-var-lib-openvswitch\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.247844 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.247207    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/949596f3-e4a0-4b1e-b1c8-e091aac598a2-iptables-alerter-script\") pod \"iptables-alerter-pnhbn\" (UID: \"949596f3-e4a0-4b1e-b1c8-e091aac598a2\") " pod="openshift-network-operator/iptables-alerter-pnhbn"
Sep 17 16:41:58.247844 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.247236    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c10930b3-04d3-4b91-8931-a9638b8c5ea7-konnectivity-ca\") pod \"konnectivity-agent-z8fgx\" (UID: \"c10930b3-04d3-4b91-8931-a9638b8c5ea7\") " pod="kube-system/konnectivity-agent-z8fgx"
Sep 17 16:41:58.247844 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.247262    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a0d6b45-1f66-4c14-988a-1ec74eac67ad-kubelet-config\") pod \"global-pull-secret-syncer-xbdp6\" (UID: \"2a0d6b45-1f66-4c14-988a-1ec74eac67ad\") " pod="kube-system/global-pull-secret-syncer-xbdp6"
Sep 17 16:41:58.247844 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.247292    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrfkx\" (UniqueName: \"kubernetes.io/projected/17d178d9-d43e-468a-9605-359baa45dc72-kube-api-access-wrfkx\") pod \"node-resolver-gnzg4\" (UID: \"17d178d9-d43e-468a-9605-359baa45dc72\") " pod="openshift-dns/node-resolver-gnzg4"
Sep 17 16:41:58.247844 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.247327    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/367c9d70-f0c6-44ae-b24e-2ac0790b5cb7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pcq95\" (UID: \"367c9d70-f0c6-44ae-b24e-2ac0790b5cb7\") " pod="openshift-multus/multus-additional-cni-plugins-pcq95"
Sep 17 16:41:58.247844 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.247361    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4-etc-sysctl-d\") pod \"tuned-mtvzk\" (UID: \"f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4\") " pod="openshift-cluster-node-tuning-operator/tuned-mtvzk"
Sep 17 16:41:58.248411 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.247387    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-host-var-lib-cni-bin\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.248411 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.247419    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-systemd-units\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.248411 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.247462    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-node-log\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.256033 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.256011    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 16:41:58.284769 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.284743    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6hftp"
Sep 17 16:41:58.295306 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.295286    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6hftp"
Sep 17 16:41:58.344783 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.344706    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-66.ec2.internal" event={"ID":"dcbcba84d85e3930495a76b3760ea833","Type":"ContainerStarted","Data":"aabec4d49f56a628aa9a790906d3979122699c1e980bc8d723f82c2a817328c9"}
Sep 17 16:41:58.345961 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.345932    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal" event={"ID":"07f553a18377ccfd877b2a2f13bd91e5","Type":"ContainerStarted","Data":"621ea3851504c6c63f19b9201b27af39332eb03563835e00c2845be255dc3c79"}
Sep 17 16:41:58.349241 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.349215    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/151db911-0f2b-48c8-8d4f-4fd4e3b67ebb-etc-selinux\") pod \"aws-ebs-csi-driver-node-nc2kn\" (UID: \"151db911-0f2b-48c8-8d4f-4fd4e3b67ebb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nc2kn"
Sep 17 16:41:58.349334 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.349254    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1-metrics-certs\") pod \"network-metrics-daemon-pfvrh\" (UID: \"fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1\") " pod="openshift-multus/network-metrics-daemon-pfvrh"
Sep 17 16:41:58.349334 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.349276    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/96bf0c2f-8a48-491a-bb3d-73e417abfa26-host\") pod \"node-ca-54cc4\" (UID: \"96bf0c2f-8a48-491a-bb3d-73e417abfa26\") " pod="openshift-image-registry/node-ca-54cc4"
Sep 17 16:41:58.349334 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.349299    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/96bf0c2f-8a48-491a-bb3d-73e417abfa26-serviceca\") pod \"node-ca-54cc4\" (UID: \"96bf0c2f-8a48-491a-bb3d-73e417abfa26\") " pod="openshift-image-registry/node-ca-54cc4"
Sep 17 16:41:58.349334 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.349320    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4-run\") pod \"tuned-mtvzk\" (UID: \"f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4\") " pod="openshift-cluster-node-tuning-operator/tuned-mtvzk"
Sep 17 16:41:58.349608 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.349342    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/91477834-6648-4e75-8530-e01bdee5e57e-ovnkube-config\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.349608 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.349343    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/96bf0c2f-8a48-491a-bb3d-73e417abfa26-host\") pod \"node-ca-54cc4\" (UID: \"96bf0c2f-8a48-491a-bb3d-73e417abfa26\") " pod="openshift-image-registry/node-ca-54cc4"
Sep 17 16:41:58.349608 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.349369    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr57f\" (UniqueName: \"kubernetes.io/projected/367c9d70-f0c6-44ae-b24e-2ac0790b5cb7-kube-api-access-zr57f\") pod \"multus-additional-cni-plugins-pcq95\" (UID: \"367c9d70-f0c6-44ae-b24e-2ac0790b5cb7\") " pod="openshift-multus/multus-additional-cni-plugins-pcq95"
Sep 17 16:41:58.349608 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.349388    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/151db911-0f2b-48c8-8d4f-4fd4e3b67ebb-etc-selinux\") pod \"aws-ebs-csi-driver-node-nc2kn\" (UID: \"151db911-0f2b-48c8-8d4f-4fd4e3b67ebb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nc2kn"
Sep 17 16:41:58.349608 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.349398    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdp7c\" (UniqueName: \"kubernetes.io/projected/151db911-0f2b-48c8-8d4f-4fd4e3b67ebb-kube-api-access-gdp7c\") pod \"aws-ebs-csi-driver-node-nc2kn\" (UID: \"151db911-0f2b-48c8-8d4f-4fd4e3b67ebb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nc2kn"
Sep 17 16:41:58.349608 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.349423    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjd2j\" (UniqueName: \"kubernetes.io/projected/fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1-kube-api-access-sjd2j\") pod \"network-metrics-daemon-pfvrh\" (UID: \"fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1\") " pod="openshift-multus/network-metrics-daemon-pfvrh"
Sep 17 16:41:58.349608 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.349465    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfhsx\" (UniqueName: \"kubernetes.io/projected/ddae5eb7-0626-4732-b8e7-92daf97e56a4-kube-api-access-gfhsx\") pod \"network-check-target-cd6rw\" (UID: \"ddae5eb7-0626-4732-b8e7-92daf97e56a4\") " pod="openshift-network-diagnostics/network-check-target-cd6rw"
Sep 17 16:41:58.349608 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:58.349399    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:41:58.349608 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.349480    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4-run\") pod \"tuned-mtvzk\" (UID: \"f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4\") " pod="openshift-cluster-node-tuning-operator/tuned-mtvzk"
Sep 17 16:41:58.349608 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.349491    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/367c9d70-f0c6-44ae-b24e-2ac0790b5cb7-system-cni-dir\") pod \"multus-additional-cni-plugins-pcq95\" (UID: \"367c9d70-f0c6-44ae-b24e-2ac0790b5cb7\") " pod="openshift-multus/multus-additional-cni-plugins-pcq95"
Sep 17 16:41:58.349608 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.349514    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4-etc-sysconfig\") pod \"tuned-mtvzk\" (UID: \"f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4\") " pod="openshift-cluster-node-tuning-operator/tuned-mtvzk"
Sep 17 16:41:58.349608 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:58.349529    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1-metrics-certs podName:fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1 nodeName:}" failed. No retries permitted until 2026-09-17 16:41:58.849512274 +0000 UTC m=+4.743991391 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1-metrics-certs") pod "network-metrics-daemon-pfvrh" (UID: "fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:41:58.349608 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.349568    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4-host\") pod \"tuned-mtvzk\" (UID: \"f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4\") " pod="openshift-cluster-node-tuning-operator/tuned-mtvzk"
Sep 17 16:41:58.349608 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.349574    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4-etc-sysconfig\") pod \"tuned-mtvzk\" (UID: \"f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4\") " pod="openshift-cluster-node-tuning-operator/tuned-mtvzk"
Sep 17 16:41:58.349608 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.349586    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-multus-cni-dir\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.350282 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.349624    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/367c9d70-f0c6-44ae-b24e-2ac0790b5cb7-system-cni-dir\") pod \"multus-additional-cni-plugins-pcq95\" (UID: \"367c9d70-f0c6-44ae-b24e-2ac0790b5cb7\") " pod="openshift-multus/multus-additional-cni-plugins-pcq95"
Sep 17 16:41:58.350282 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.349721    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-multus-cni-dir\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.350282 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.349739    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4-lib-modules\") pod \"tuned-mtvzk\" (UID: \"f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4\") " pod="openshift-cluster-node-tuning-operator/tuned-mtvzk"
Sep 17 16:41:58.350282 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.349766    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4-host\") pod \"tuned-mtvzk\" (UID: \"f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4\") " pod="openshift-cluster-node-tuning-operator/tuned-mtvzk"
Sep 17 16:41:58.350282 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.349770    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-cni-binary-copy\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.350282 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.349786    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/96bf0c2f-8a48-491a-bb3d-73e417abfa26-serviceca\") pod \"node-ca-54cc4\" (UID: \"96bf0c2f-8a48-491a-bb3d-73e417abfa26\") " pod="openshift-image-registry/node-ca-54cc4"
Sep 17 16:41:58.350282 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.349795    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-multus-daemon-config\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.350282 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.349815    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a0d6b45-1f66-4c14-988a-1ec74eac67ad-dbus\") pod \"global-pull-secret-syncer-xbdp6\" (UID: \"2a0d6b45-1f66-4c14-988a-1ec74eac67ad\") " pod="kube-system/global-pull-secret-syncer-xbdp6"
Sep 17 16:41:58.350282 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.349839    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/17d178d9-d43e-468a-9605-359baa45dc72-hosts-file\") pod \"node-resolver-gnzg4\" (UID: \"17d178d9-d43e-468a-9605-359baa45dc72\") " pod="openshift-dns/node-resolver-gnzg4"
Sep 17 16:41:58.350282 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.349863    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-host-run-netns\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.350282 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.349876    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4-lib-modules\") pod \"tuned-mtvzk\" (UID: \"f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4\") " pod="openshift-cluster-node-tuning-operator/tuned-mtvzk"
Sep 17 16:41:58.350282 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.349886    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-host-cni-bin\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.350282 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.349892    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a0d6b45-1f66-4c14-988a-1ec74eac67ad-dbus\") pod \"global-pull-secret-syncer-xbdp6\" (UID: \"2a0d6b45-1f66-4c14-988a-1ec74eac67ad\") " pod="kube-system/global-pull-secret-syncer-xbdp6"
Sep 17 16:41:58.350282 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.349901    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/91477834-6648-4e75-8530-e01bdee5e57e-ovnkube-config\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.350282 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.349919    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/367c9d70-f0c6-44ae-b24e-2ac0790b5cb7-cnibin\") pod \"multus-additional-cni-plugins-pcq95\" (UID: \"367c9d70-f0c6-44ae-b24e-2ac0790b5cb7\") " pod="openshift-multus/multus-additional-cni-plugins-pcq95"
Sep 17 16:41:58.350282 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.349930    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-host-run-netns\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.350282 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.349936    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/17d178d9-d43e-468a-9605-359baa45dc72-hosts-file\") pod \"node-resolver-gnzg4\" (UID: \"17d178d9-d43e-468a-9605-359baa45dc72\") " pod="openshift-dns/node-resolver-gnzg4"
Sep 17 16:41:58.350282 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.349944    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4-etc-tuned\") pod \"tuned-mtvzk\" (UID: \"f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4\") " pod="openshift-cluster-node-tuning-operator/tuned-mtvzk"
Sep 17 16:41:58.351101 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.349936    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-host-cni-bin\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.351101 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.349956    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/367c9d70-f0c6-44ae-b24e-2ac0790b5cb7-cnibin\") pod \"multus-additional-cni-plugins-pcq95\" (UID: \"367c9d70-f0c6-44ae-b24e-2ac0790b5cb7\") " pod="openshift-multus/multus-additional-cni-plugins-pcq95"
Sep 17 16:41:58.351101 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.349972    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c10930b3-04d3-4b91-8931-a9638b8c5ea7-agent-certs\") pod \"konnectivity-agent-z8fgx\" (UID: \"c10930b3-04d3-4b91-8931-a9638b8c5ea7\") " pod="kube-system/konnectivity-agent-z8fgx"
Sep 17 16:41:58.351101 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.349994    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-host-var-lib-kubelet\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.351101 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350019    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-run-openvswitch\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.351101 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350041    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.351101 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350058    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-host-var-lib-kubelet\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.351101 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350064    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/91477834-6648-4e75-8530-e01bdee5e57e-ovnkube-script-lib\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.351101 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350086    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4-etc-systemd\") pod \"tuned-mtvzk\" (UID: \"f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4\") " pod="openshift-cluster-node-tuning-operator/tuned-mtvzk"
Sep 17 16:41:58.351101 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350108    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.351101 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350107    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-cnibin\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.351101 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350153    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-cnibin\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.351101 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350155    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-multus-socket-dir-parent\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.351101 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350181    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-multus-socket-dir-parent\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.351101 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350188    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-etc-kubernetes\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.351101 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350088    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-run-openvswitch\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.351101 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350226    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4-etc-systemd\") pod \"tuned-mtvzk\" (UID: \"f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4\") " pod="openshift-cluster-node-tuning-operator/tuned-mtvzk"
Sep 17 16:41:58.351101 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350212    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-log-socket\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.352000 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350251    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-log-socket\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.352000 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350247    2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 16:41:58.352000 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350264    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-etc-kubernetes\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.352000 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350262    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vltn5\" (UniqueName: \"kubernetes.io/projected/91477834-6648-4e75-8530-e01bdee5e57e-kube-api-access-vltn5\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.352000 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350305    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4-sys\") pod \"tuned-mtvzk\" (UID: \"f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4\") " pod="openshift-cluster-node-tuning-operator/tuned-mtvzk"
Sep 17 16:41:58.352000 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350330    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-system-cni-dir\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.352000 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350350    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-host-run-netns\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.352000 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350366    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-cni-binary-copy\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.352000 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350366    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-multus-daemon-config\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.352000 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350371    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-hostroot\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.352000 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350418    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4-sys\") pod \"tuned-mtvzk\" (UID: \"f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4\") " pod="openshift-cluster-node-tuning-operator/tuned-mtvzk"
Sep 17 16:41:58.352000 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350421    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-host-run-multus-certs\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.352000 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350474    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-host-run-multus-certs\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.352000 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350478    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-host-run-netns\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.352000 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350494    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-host-slash\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.352000 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350516    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-hostroot\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.352000 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350526    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/151db911-0f2b-48c8-8d4f-4fd4e3b67ebb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nc2kn\" (UID: \"151db911-0f2b-48c8-8d4f-4fd4e3b67ebb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nc2kn"
Sep 17 16:41:58.352000 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350543    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-system-cni-dir\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.352000 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350546    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rml2\" (UniqueName: \"kubernetes.io/projected/96bf0c2f-8a48-491a-bb3d-73e417abfa26-kube-api-access-9rml2\") pod \"node-ca-54cc4\" (UID: \"96bf0c2f-8a48-491a-bb3d-73e417abfa26\") " pod="openshift-image-registry/node-ca-54cc4"
Sep 17 16:41:58.352810 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350553    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-host-slash\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.352810 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350575    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/367c9d70-f0c6-44ae-b24e-2ac0790b5cb7-cni-binary-copy\") pod \"multus-additional-cni-plugins-pcq95\" (UID: \"367c9d70-f0c6-44ae-b24e-2ac0790b5cb7\") " pod="openshift-multus/multus-additional-cni-plugins-pcq95"
Sep 17 16:41:58.352810 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350593    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/151db911-0f2b-48c8-8d4f-4fd4e3b67ebb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nc2kn\" (UID: \"151db911-0f2b-48c8-8d4f-4fd4e3b67ebb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nc2kn"
Sep 17 16:41:58.352810 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350599    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4-etc-kubernetes\") pod \"tuned-mtvzk\" (UID: \"f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4\") " pod="openshift-cluster-node-tuning-operator/tuned-mtvzk"
Sep 17 16:41:58.352810 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350628    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4-etc-kubernetes\") pod \"tuned-mtvzk\" (UID: \"f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4\") " pod="openshift-cluster-node-tuning-operator/tuned-mtvzk"
Sep 17 16:41:58.352810 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350652    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-host-run-k8s-cni-cncf-io\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.352810 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350679    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17d178d9-d43e-468a-9605-359baa45dc72-tmp-dir\") pod \"node-resolver-gnzg4\" (UID: \"17d178d9-d43e-468a-9605-359baa45dc72\") " pod="openshift-dns/node-resolver-gnzg4"
Sep 17 16:41:58.352810 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350688    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-host-run-k8s-cni-cncf-io\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.352810 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350702    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/949596f3-e4a0-4b1e-b1c8-e091aac598a2-host-slash\") pod \"iptables-alerter-pnhbn\" (UID: \"949596f3-e4a0-4b1e-b1c8-e091aac598a2\") " pod="openshift-network-operator/iptables-alerter-pnhbn"
Sep 17 16:41:58.352810 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350722    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4-tmp\") pod \"tuned-mtvzk\" (UID: \"f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4\") " pod="openshift-cluster-node-tuning-operator/tuned-mtvzk"
Sep 17 16:41:58.352810 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350738    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/91477834-6648-4e75-8530-e01bdee5e57e-ovnkube-script-lib\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.352810 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350743    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-run-systemd\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.352810 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350773    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-run-systemd\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.352810 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350792    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-var-lib-openvswitch\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.352810 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350818    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/949596f3-e4a0-4b1e-b1c8-e091aac598a2-host-slash\") pod \"iptables-alerter-pnhbn\" (UID: \"949596f3-e4a0-4b1e-b1c8-e091aac598a2\") " pod="openshift-network-operator/iptables-alerter-pnhbn"
Sep 17 16:41:58.352810 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350839    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-var-lib-openvswitch\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.352810 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350859    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/949596f3-e4a0-4b1e-b1c8-e091aac598a2-iptables-alerter-script\") pod \"iptables-alerter-pnhbn\" (UID: \"949596f3-e4a0-4b1e-b1c8-e091aac598a2\") " pod="openshift-network-operator/iptables-alerter-pnhbn"
Sep 17 16:41:58.353662 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350885    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c10930b3-04d3-4b91-8931-a9638b8c5ea7-konnectivity-ca\") pod \"konnectivity-agent-z8fgx\" (UID: \"c10930b3-04d3-4b91-8931-a9638b8c5ea7\") " pod="kube-system/konnectivity-agent-z8fgx"
Sep 17 16:41:58.353662 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350909    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a0d6b45-1f66-4c14-988a-1ec74eac67ad-kubelet-config\") pod \"global-pull-secret-syncer-xbdp6\" (UID: \"2a0d6b45-1f66-4c14-988a-1ec74eac67ad\") " pod="kube-system/global-pull-secret-syncer-xbdp6"
Sep 17 16:41:58.353662 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350931    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrfkx\" (UniqueName: \"kubernetes.io/projected/17d178d9-d43e-468a-9605-359baa45dc72-kube-api-access-wrfkx\") pod \"node-resolver-gnzg4\" (UID: \"17d178d9-d43e-468a-9605-359baa45dc72\") " pod="openshift-dns/node-resolver-gnzg4"
Sep 17 16:41:58.353662 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350949    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/367c9d70-f0c6-44ae-b24e-2ac0790b5cb7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pcq95\" (UID: \"367c9d70-f0c6-44ae-b24e-2ac0790b5cb7\") " pod="openshift-multus/multus-additional-cni-plugins-pcq95"
Sep 17 16:41:58.353662 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350966    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4-etc-sysctl-d\") pod \"tuned-mtvzk\" (UID: \"f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4\") " pod="openshift-cluster-node-tuning-operator/tuned-mtvzk"
Sep 17 16:41:58.353662 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350987    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-host-var-lib-cni-bin\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.353662 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.350989    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17d178d9-d43e-468a-9605-359baa45dc72-tmp-dir\") pod \"node-resolver-gnzg4\" (UID: \"17d178d9-d43e-468a-9605-359baa45dc72\") " pod="openshift-dns/node-resolver-gnzg4"
Sep 17 16:41:58.353662 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.351001    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-systemd-units\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.353662 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.351023    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-node-log\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.353662 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.351038    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-multus-conf-dir\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.353662 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.351071    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-host-run-ovn-kubernetes\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.353662 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.351094    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-host-var-lib-cni-multus\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.353662 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.351111    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-host-cni-netd\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.353662 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.351123    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/367c9d70-f0c6-44ae-b24e-2ac0790b5cb7-cni-binary-copy\") pod \"multus-additional-cni-plugins-pcq95\" (UID: \"367c9d70-f0c6-44ae-b24e-2ac0790b5cb7\") " pod="openshift-multus/multus-additional-cni-plugins-pcq95"
Sep 17 16:41:58.353662 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.351136    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/91477834-6648-4e75-8530-e01bdee5e57e-env-overrides\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.353662 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.351153    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/367c9d70-f0c6-44ae-b24e-2ac0790b5cb7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pcq95\" (UID: \"367c9d70-f0c6-44ae-b24e-2ac0790b5cb7\") " pod="openshift-multus/multus-additional-cni-plugins-pcq95"
Sep 17 16:41:58.353662 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.351167    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/151db911-0f2b-48c8-8d4f-4fd4e3b67ebb-device-dir\") pod \"aws-ebs-csi-driver-node-nc2kn\" (UID: \"151db911-0f2b-48c8-8d4f-4fd4e3b67ebb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nc2kn"
Sep 17 16:41:58.354477 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.351194    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/367c9d70-f0c6-44ae-b24e-2ac0790b5cb7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pcq95\" (UID: \"367c9d70-f0c6-44ae-b24e-2ac0790b5cb7\") " pod="openshift-multus/multus-additional-cni-plugins-pcq95"
Sep 17 16:41:58.354477 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.351203    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a0d6b45-1f66-4c14-988a-1ec74eac67ad-kubelet-config\") pod \"global-pull-secret-syncer-xbdp6\" (UID: \"2a0d6b45-1f66-4c14-988a-1ec74eac67ad\") " pod="kube-system/global-pull-secret-syncer-xbdp6"
Sep 17 16:41:58.354477 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.351221    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/367c9d70-f0c6-44ae-b24e-2ac0790b5cb7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pcq95\" (UID: \"367c9d70-f0c6-44ae-b24e-2ac0790b5cb7\") " pod="openshift-multus/multus-additional-cni-plugins-pcq95"
Sep 17 16:41:58.354477 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.351247    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpq8z\" (UniqueName: \"kubernetes.io/projected/949596f3-e4a0-4b1e-b1c8-e091aac598a2-kube-api-access-vpq8z\") pod \"iptables-alerter-pnhbn\" (UID: \"949596f3-e4a0-4b1e-b1c8-e091aac598a2\") " pod="openshift-network-operator/iptables-alerter-pnhbn"
Sep 17 16:41:58.354477 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.351271    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4-etc-sysctl-conf\") pod \"tuned-mtvzk\" (UID: \"f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4\") " pod="openshift-cluster-node-tuning-operator/tuned-mtvzk"
Sep 17 16:41:58.354477 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.351295    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb24m\" (UniqueName: \"kubernetes.io/projected/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-kube-api-access-zb24m\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.354477 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.351318    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-etc-openvswitch\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.354477 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.351340    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-run-ovn\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.354477 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.351361    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/151db911-0f2b-48c8-8d4f-4fd4e3b67ebb-socket-dir\") pod \"aws-ebs-csi-driver-node-nc2kn\" (UID: \"151db911-0f2b-48c8-8d4f-4fd4e3b67ebb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nc2kn"
Sep 17 16:41:58.354477 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.351384    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/151db911-0f2b-48c8-8d4f-4fd4e3b67ebb-sys-fs\") pod \"aws-ebs-csi-driver-node-nc2kn\" (UID: \"151db911-0f2b-48c8-8d4f-4fd4e3b67ebb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nc2kn"
Sep 17 16:41:58.354477 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.351408    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4-etc-modprobe-d\") pod \"tuned-mtvzk\" (UID: \"f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4\") " pod="openshift-cluster-node-tuning-operator/tuned-mtvzk"
Sep 17 16:41:58.354477 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.351419    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-node-log\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.354477 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.351451    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4-var-lib-kubelet\") pod \"tuned-mtvzk\" (UID: \"f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4\") " pod="openshift-cluster-node-tuning-operator/tuned-mtvzk"
Sep 17 16:41:58.354477 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.351474    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/91477834-6648-4e75-8530-e01bdee5e57e-env-overrides\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.354477 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.351514    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r59pr\" (UniqueName: \"kubernetes.io/projected/f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4-kube-api-access-r59pr\") pod \"tuned-mtvzk\" (UID: \"f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4\") " pod="openshift-cluster-node-tuning-operator/tuned-mtvzk"
Sep 17 16:41:58.354477 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.351522    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-multus-conf-dir\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.354477 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.351577    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/151db911-0f2b-48c8-8d4f-4fd4e3b67ebb-device-dir\") pod \"aws-ebs-csi-driver-node-nc2kn\" (UID: \"151db911-0f2b-48c8-8d4f-4fd4e3b67ebb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nc2kn"
Sep 17 16:41:58.355255 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.351598    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4-etc-sysctl-d\") pod \"tuned-mtvzk\" (UID: \"f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4\") " pod="openshift-cluster-node-tuning-operator/tuned-mtvzk"
Sep 17 16:41:58.355255 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:58.351605    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 16:41:58.355255 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.351609    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-host-run-ovn-kubernetes\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.355255 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.351638    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-host-var-lib-cni-multus\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.355255 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.351643    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-host-var-lib-cni-bin\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.355255 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.351543    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a0d6b45-1f66-4c14-988a-1ec74eac67ad-original-pull-secret\") pod \"global-pull-secret-syncer-xbdp6\" (UID: \"2a0d6b45-1f66-4c14-988a-1ec74eac67ad\") " pod="kube-system/global-pull-secret-syncer-xbdp6"
Sep 17 16:41:58.355255 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:58.351661    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a0d6b45-1f66-4c14-988a-1ec74eac67ad-original-pull-secret podName:2a0d6b45-1f66-4c14-988a-1ec74eac67ad nodeName:}" failed. No retries permitted until 2026-09-17 16:41:58.851645006 +0000 UTC m=+4.746124140 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a0d6b45-1f66-4c14-988a-1ec74eac67ad-original-pull-secret") pod "global-pull-secret-syncer-xbdp6" (UID: "2a0d6b45-1f66-4c14-988a-1ec74eac67ad") : object "kube-system"/"original-pull-secret" not registered
Sep 17 16:41:58.355255 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.351673    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-systemd-units\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.355255 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.351684    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-host-cni-netd\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.355255 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.351731    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/151db911-0f2b-48c8-8d4f-4fd4e3b67ebb-sys-fs\") pod \"aws-ebs-csi-driver-node-nc2kn\" (UID: \"151db911-0f2b-48c8-8d4f-4fd4e3b67ebb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nc2kn"
Sep 17 16:41:58.355255 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.351928    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/949596f3-e4a0-4b1e-b1c8-e091aac598a2-iptables-alerter-script\") pod \"iptables-alerter-pnhbn\" (UID: \"949596f3-e4a0-4b1e-b1c8-e091aac598a2\") " pod="openshift-network-operator/iptables-alerter-pnhbn"
Sep 17 16:41:58.355255 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.351995    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-run-ovn\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.355255 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.352038    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-etc-openvswitch\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.355255 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.352077    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/367c9d70-f0c6-44ae-b24e-2ac0790b5cb7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pcq95\" (UID: \"367c9d70-f0c6-44ae-b24e-2ac0790b5cb7\") " pod="openshift-multus/multus-additional-cni-plugins-pcq95"
Sep 17 16:41:58.355255 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.352117    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/367c9d70-f0c6-44ae-b24e-2ac0790b5cb7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pcq95\" (UID: \"367c9d70-f0c6-44ae-b24e-2ac0790b5cb7\") " pod="openshift-multus/multus-additional-cni-plugins-pcq95"
Sep 17 16:41:58.355255 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.352162    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4-etc-sysctl-conf\") pod \"tuned-mtvzk\" (UID: \"f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4\") " pod="openshift-cluster-node-tuning-operator/tuned-mtvzk"
Sep 17 16:41:58.355255 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.352170    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/151db911-0f2b-48c8-8d4f-4fd4e3b67ebb-socket-dir\") pod \"aws-ebs-csi-driver-node-nc2kn\" (UID: \"151db911-0f2b-48c8-8d4f-4fd4e3b67ebb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nc2kn"
Sep 17 16:41:58.356052 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.352195    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-host-kubelet\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.356052 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.352213    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/91477834-6648-4e75-8530-e01bdee5e57e-ovn-node-metrics-cert\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.356052 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.352219    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4-var-lib-kubelet\") pod \"tuned-mtvzk\" (UID: \"f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4\") " pod="openshift-cluster-node-tuning-operator/tuned-mtvzk"
Sep 17 16:41:58.356052 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.352229    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/151db911-0f2b-48c8-8d4f-4fd4e3b67ebb-registration-dir\") pod \"aws-ebs-csi-driver-node-nc2kn\" (UID: \"151db911-0f2b-48c8-8d4f-4fd4e3b67ebb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nc2kn"
Sep 17 16:41:58.356052 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.352256    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4-etc-modprobe-d\") pod \"tuned-mtvzk\" (UID: \"f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4\") " pod="openshift-cluster-node-tuning-operator/tuned-mtvzk"
Sep 17 16:41:58.356052 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.352263    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/91477834-6648-4e75-8530-e01bdee5e57e-host-kubelet\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.356052 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.351423    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c10930b3-04d3-4b91-8931-a9638b8c5ea7-konnectivity-ca\") pod \"konnectivity-agent-z8fgx\" (UID: \"c10930b3-04d3-4b91-8931-a9638b8c5ea7\") " pod="kube-system/konnectivity-agent-z8fgx"
Sep 17 16:41:58.356052 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.352291    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/151db911-0f2b-48c8-8d4f-4fd4e3b67ebb-registration-dir\") pod \"aws-ebs-csi-driver-node-nc2kn\" (UID: \"151db911-0f2b-48c8-8d4f-4fd4e3b67ebb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nc2kn"
Sep 17 16:41:58.356052 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.354556    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4-etc-tuned\") pod \"tuned-mtvzk\" (UID: \"f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4\") " pod="openshift-cluster-node-tuning-operator/tuned-mtvzk"
Sep 17 16:41:58.356052 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.354678    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/91477834-6648-4e75-8530-e01bdee5e57e-ovn-node-metrics-cert\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.356052 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.354730    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c10930b3-04d3-4b91-8931-a9638b8c5ea7-agent-certs\") pod \"konnectivity-agent-z8fgx\" (UID: \"c10930b3-04d3-4b91-8931-a9638b8c5ea7\") " pod="kube-system/konnectivity-agent-z8fgx"
Sep 17 16:41:58.356052 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.355769    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4-tmp\") pod \"tuned-mtvzk\" (UID: \"f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4\") " pod="openshift-cluster-node-tuning-operator/tuned-mtvzk"
Sep 17 16:41:58.370170 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:58.370146    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 16:41:58.370264 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:58.370176    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 16:41:58.370264 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:58.370191    2566 projected.go:196] Error preparing data for projected volume kube-api-access-gfhsx for pod openshift-network-diagnostics/network-check-target-cd6rw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:41:58.370264 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:58.370256    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ddae5eb7-0626-4732-b8e7-92daf97e56a4-kube-api-access-gfhsx podName:ddae5eb7-0626-4732-b8e7-92daf97e56a4 nodeName:}" failed. No retries permitted until 2026-09-17 16:41:58.870239218 +0000 UTC m=+4.764718350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gfhsx" (UniqueName: "kubernetes.io/projected/ddae5eb7-0626-4732-b8e7-92daf97e56a4-kube-api-access-gfhsx") pod "network-check-target-cd6rw" (UID: "ddae5eb7-0626-4732-b8e7-92daf97e56a4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:41:58.370976 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.370951    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjd2j\" (UniqueName: \"kubernetes.io/projected/fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1-kube-api-access-sjd2j\") pod \"network-metrics-daemon-pfvrh\" (UID: \"fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1\") " pod="openshift-multus/network-metrics-daemon-pfvrh"
Sep 17 16:41:58.371448 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.371411    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb24m\" (UniqueName: \"kubernetes.io/projected/b19f9bd7-7bff-4572-aedc-07a2c4b4a975-kube-api-access-zb24m\") pod \"multus-cq7pn\" (UID: \"b19f9bd7-7bff-4572-aedc-07a2c4b4a975\") " pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.372354 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.372329    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrfkx\" (UniqueName: \"kubernetes.io/projected/17d178d9-d43e-468a-9605-359baa45dc72-kube-api-access-wrfkx\") pod \"node-resolver-gnzg4\" (UID: \"17d178d9-d43e-468a-9605-359baa45dc72\") " pod="openshift-dns/node-resolver-gnzg4"
Sep 17 16:41:58.372609 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.372579    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdp7c\" (UniqueName: \"kubernetes.io/projected/151db911-0f2b-48c8-8d4f-4fd4e3b67ebb-kube-api-access-gdp7c\") pod \"aws-ebs-csi-driver-node-nc2kn\" (UID: \"151db911-0f2b-48c8-8d4f-4fd4e3b67ebb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nc2kn"
Sep 17 16:41:58.373910 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.373866    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpq8z\" (UniqueName: \"kubernetes.io/projected/949596f3-e4a0-4b1e-b1c8-e091aac598a2-kube-api-access-vpq8z\") pod \"iptables-alerter-pnhbn\" (UID: \"949596f3-e4a0-4b1e-b1c8-e091aac598a2\") " pod="openshift-network-operator/iptables-alerter-pnhbn"
Sep 17 16:41:58.374027 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.374004    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rml2\" (UniqueName: \"kubernetes.io/projected/96bf0c2f-8a48-491a-bb3d-73e417abfa26-kube-api-access-9rml2\") pod \"node-ca-54cc4\" (UID: \"96bf0c2f-8a48-491a-bb3d-73e417abfa26\") " pod="openshift-image-registry/node-ca-54cc4"
Sep 17 16:41:58.374802 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.374784    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vltn5\" (UniqueName: \"kubernetes.io/projected/91477834-6648-4e75-8530-e01bdee5e57e-kube-api-access-vltn5\") pod \"ovnkube-node-gblhr\" (UID: \"91477834-6648-4e75-8530-e01bdee5e57e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.375282 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.375267    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr57f\" (UniqueName: \"kubernetes.io/projected/367c9d70-f0c6-44ae-b24e-2ac0790b5cb7-kube-api-access-zr57f\") pod \"multus-additional-cni-plugins-pcq95\" (UID: \"367c9d70-f0c6-44ae-b24e-2ac0790b5cb7\") " pod="openshift-multus/multus-additional-cni-plugins-pcq95"
Sep 17 16:41:58.375757 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.375736    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r59pr\" (UniqueName: \"kubernetes.io/projected/f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4-kube-api-access-r59pr\") pod \"tuned-mtvzk\" (UID: \"f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4\") " pod="openshift-cluster-node-tuning-operator/tuned-mtvzk"
Sep 17 16:41:58.512647 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.512577    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-z8fgx"
Sep 17 16:41:58.519270 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.519244    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nc2kn"
Sep 17 16:41:58.526724 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.526705    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 16:41:58.529404 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.529390    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cq7pn"
Sep 17 16:41:58.535866 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.535849    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-54cc4"
Sep 17 16:41:58.543357 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.543336    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gnzg4"
Sep 17 16:41:58.550913 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.550896    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pcq95"
Sep 17 16:41:58.559658 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.559641    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pnhbn"
Sep 17 16:41:58.570229 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.570208    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:41:58.574828 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.574811    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mtvzk"
Sep 17 16:41:58.855638 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.855564    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a0d6b45-1f66-4c14-988a-1ec74eac67ad-original-pull-secret\") pod \"global-pull-secret-syncer-xbdp6\" (UID: \"2a0d6b45-1f66-4c14-988a-1ec74eac67ad\") " pod="kube-system/global-pull-secret-syncer-xbdp6"
Sep 17 16:41:58.855638 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.855599    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1-metrics-certs\") pod \"network-metrics-daemon-pfvrh\" (UID: \"fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1\") " pod="openshift-multus/network-metrics-daemon-pfvrh"
Sep 17 16:41:58.855826 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:58.855704    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:41:58.855826 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:58.855704    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 16:41:58.855826 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:58.855763    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1-metrics-certs podName:fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1 nodeName:}" failed. No retries permitted until 2026-09-17 16:41:59.855744875 +0000 UTC m=+5.750223995 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1-metrics-certs") pod "network-metrics-daemon-pfvrh" (UID: "fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:41:58.855826 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:58.855778    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a0d6b45-1f66-4c14-988a-1ec74eac67ad-original-pull-secret podName:2a0d6b45-1f66-4c14-988a-1ec74eac67ad nodeName:}" failed. No retries permitted until 2026-09-17 16:41:59.855772349 +0000 UTC m=+5.750251466 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a0d6b45-1f66-4c14-988a-1ec74eac67ad-original-pull-secret") pod "global-pull-secret-syncer-xbdp6" (UID: "2a0d6b45-1f66-4c14-988a-1ec74eac67ad") : object "kube-system"/"original-pull-secret" not registered
Sep 17 16:41:58.956816 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:58.956787    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfhsx\" (UniqueName: \"kubernetes.io/projected/ddae5eb7-0626-4732-b8e7-92daf97e56a4-kube-api-access-gfhsx\") pod \"network-check-target-cd6rw\" (UID: \"ddae5eb7-0626-4732-b8e7-92daf97e56a4\") " pod="openshift-network-diagnostics/network-check-target-cd6rw"
Sep 17 16:41:58.956960 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:58.956942    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 16:41:58.957009 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:58.956962    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 16:41:58.957009 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:58.956971    2566 projected.go:196] Error preparing data for projected volume kube-api-access-gfhsx for pod openshift-network-diagnostics/network-check-target-cd6rw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:41:58.957078 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:58.957025    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ddae5eb7-0626-4732-b8e7-92daf97e56a4-kube-api-access-gfhsx podName:ddae5eb7-0626-4732-b8e7-92daf97e56a4 nodeName:}" failed. No retries permitted until 2026-09-17 16:41:59.957008221 +0000 UTC m=+5.851487540 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gfhsx" (UniqueName: "kubernetes.io/projected/ddae5eb7-0626-4732-b8e7-92daf97e56a4-kube-api-access-gfhsx") pod "network-check-target-cd6rw" (UID: "ddae5eb7-0626-4732-b8e7-92daf97e56a4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:41:59.274637 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:59.274594    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod367c9d70_f0c6_44ae_b24e_2ac0790b5cb7.slice/crio-2a147bf3376d548c8e06046ccbdf0246ca1adfda472596c74d36cb2972b348f4 WatchSource:0}: Error finding container 2a147bf3376d548c8e06046ccbdf0246ca1adfda472596c74d36cb2972b348f4: Status 404 returned error can't find the container with id 2a147bf3376d548c8e06046ccbdf0246ca1adfda472596c74d36cb2972b348f4
Sep 17 16:41:59.276768 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:59.276046    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc10930b3_04d3_4b91_8931_a9638b8c5ea7.slice/crio-21033a81d0690ddebdebeab33b30a04e81e7c78fa6e67e8a01f9ca97580ab258 WatchSource:0}: Error finding container 21033a81d0690ddebdebeab33b30a04e81e7c78fa6e67e8a01f9ca97580ab258: Status 404 returned error can't find the container with id 21033a81d0690ddebdebeab33b30a04e81e7c78fa6e67e8a01f9ca97580ab258
Sep 17 16:41:59.280964 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:59.280920    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod949596f3_e4a0_4b1e_b1c8_e091aac598a2.slice/crio-4d5b2f3fcb5f771b496a045f673ff8d2f622cdd7f29ce8ca5afbaaad4d2f4c13 WatchSource:0}: Error finding container 4d5b2f3fcb5f771b496a045f673ff8d2f622cdd7f29ce8ca5afbaaad4d2f4c13: Status 404 returned error can't find the container with id 4d5b2f3fcb5f771b496a045f673ff8d2f622cdd7f29ce8ca5afbaaad4d2f4c13
Sep 17 16:41:59.281908 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:59.281780    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91477834_6648_4e75_8530_e01bdee5e57e.slice/crio-bbe367fe009d6adfd6c12a4c91c3a97d5b7e5c50ffcb75792d138391240433ed WatchSource:0}: Error finding container bbe367fe009d6adfd6c12a4c91c3a97d5b7e5c50ffcb75792d138391240433ed: Status 404 returned error can't find the container with id bbe367fe009d6adfd6c12a4c91c3a97d5b7e5c50ffcb75792d138391240433ed
Sep 17 16:41:59.282759 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:41:59.282721    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1c8c31e_b0af_41c4_9e2b_2cb79aacccd4.slice/crio-86fed12e16deafc6b378883df8de5513ba121347e43d359ca4590b9b5256c5b9 WatchSource:0}: Error finding container 86fed12e16deafc6b378883df8de5513ba121347e43d359ca4590b9b5256c5b9: Status 404 returned error can't find the container with id 86fed12e16deafc6b378883df8de5513ba121347e43d359ca4590b9b5256c5b9
Sep 17 16:41:59.299005 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:59.298979    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 16:36:58 +0000 UTC" deadline="2028-04-01 20:27:11.864540739 +0000 UTC"
Sep 17 16:41:59.299005 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:59.299003    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13491h45m12.565540522s"
Sep 17 16:41:59.340582 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:59.340563    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfvrh"
Sep 17 16:41:59.340690 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:59.340666    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfvrh" podUID="fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1"
Sep 17 16:41:59.349276 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:59.349254    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gblhr" event={"ID":"91477834-6648-4e75-8530-e01bdee5e57e","Type":"ContainerStarted","Data":"bbe367fe009d6adfd6c12a4c91c3a97d5b7e5c50ffcb75792d138391240433ed"}
Sep 17 16:41:59.350379 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:59.350354    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pcq95" event={"ID":"367c9d70-f0c6-44ae-b24e-2ac0790b5cb7","Type":"ContainerStarted","Data":"2a147bf3376d548c8e06046ccbdf0246ca1adfda472596c74d36cb2972b348f4"}
Sep 17 16:41:59.351573 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:59.351539    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mtvzk" event={"ID":"f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4","Type":"ContainerStarted","Data":"86fed12e16deafc6b378883df8de5513ba121347e43d359ca4590b9b5256c5b9"}
Sep 17 16:41:59.352860 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:59.352789    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pnhbn" event={"ID":"949596f3-e4a0-4b1e-b1c8-e091aac598a2","Type":"ContainerStarted","Data":"4d5b2f3fcb5f771b496a045f673ff8d2f622cdd7f29ce8ca5afbaaad4d2f4c13"}
Sep 17 16:41:59.354013 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:59.353988    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-z8fgx" event={"ID":"c10930b3-04d3-4b91-8931-a9638b8c5ea7","Type":"ContainerStarted","Data":"21033a81d0690ddebdebeab33b30a04e81e7c78fa6e67e8a01f9ca97580ab258"}
Sep 17 16:41:59.355187 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:59.355168    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cq7pn" event={"ID":"b19f9bd7-7bff-4572-aedc-07a2c4b4a975","Type":"ContainerStarted","Data":"ecf6cd403f3a4c8f4dd4b05f43f46fd1719a4039ad3457b4d17b711abaa52c40"}
Sep 17 16:41:59.356251 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:59.356230    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gnzg4" event={"ID":"17d178d9-d43e-468a-9605-359baa45dc72","Type":"ContainerStarted","Data":"19bb7ef2b7d1b675a0ddcb41b82044ea63df38ec6670c9932b4155cb545f7319"}
Sep 17 16:41:59.357545 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:59.357520    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nc2kn" event={"ID":"151db911-0f2b-48c8-8d4f-4fd4e3b67ebb","Type":"ContainerStarted","Data":"9ff4e8bff7f4030dd70635f2b8e1dfad1282c164743b2b2223382adb044018e2"}
Sep 17 16:41:59.358608 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:59.358586    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-54cc4" event={"ID":"96bf0c2f-8a48-491a-bb3d-73e417abfa26","Type":"ContainerStarted","Data":"27a1206f1b44fb2e9e79bf0834f343ef96e03e747619228d3462f36b83f35c3f"}
Sep 17 16:41:59.594692 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:59.594670    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 16:41:59.862791 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:59.862708    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a0d6b45-1f66-4c14-988a-1ec74eac67ad-original-pull-secret\") pod \"global-pull-secret-syncer-xbdp6\" (UID: \"2a0d6b45-1f66-4c14-988a-1ec74eac67ad\") " pod="kube-system/global-pull-secret-syncer-xbdp6"
Sep 17 16:41:59.862791 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:59.862763    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1-metrics-certs\") pod \"network-metrics-daemon-pfvrh\" (UID: \"fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1\") " pod="openshift-multus/network-metrics-daemon-pfvrh"
Sep 17 16:41:59.862998 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:59.862901    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:41:59.862998 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:59.862965    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1-metrics-certs podName:fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:01.862946639 +0000 UTC m=+7.757425761 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1-metrics-certs") pod "network-metrics-daemon-pfvrh" (UID: "fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:41:59.863379 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:59.863356    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 16:41:59.863485 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:59.863418    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a0d6b45-1f66-4c14-988a-1ec74eac67ad-original-pull-secret podName:2a0d6b45-1f66-4c14-988a-1ec74eac67ad nodeName:}" failed. No retries permitted until 2026-09-17 16:42:01.863401 +0000 UTC m=+7.757880136 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a0d6b45-1f66-4c14-988a-1ec74eac67ad-original-pull-secret") pod "global-pull-secret-syncer-xbdp6" (UID: "2a0d6b45-1f66-4c14-988a-1ec74eac67ad") : object "kube-system"/"original-pull-secret" not registered
Sep 17 16:41:59.963960 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:41:59.963925    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfhsx\" (UniqueName: \"kubernetes.io/projected/ddae5eb7-0626-4732-b8e7-92daf97e56a4-kube-api-access-gfhsx\") pod \"network-check-target-cd6rw\" (UID: \"ddae5eb7-0626-4732-b8e7-92daf97e56a4\") " pod="openshift-network-diagnostics/network-check-target-cd6rw"
Sep 17 16:41:59.964116 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:59.964080    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 16:41:59.964116 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:59.964103    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 16:41:59.964116 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:59.964117    2566 projected.go:196] Error preparing data for projected volume kube-api-access-gfhsx for pod openshift-network-diagnostics/network-check-target-cd6rw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:41:59.964278 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:41:59.964180    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ddae5eb7-0626-4732-b8e7-92daf97e56a4-kube-api-access-gfhsx podName:ddae5eb7-0626-4732-b8e7-92daf97e56a4 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:01.964163022 +0000 UTC m=+7.858642162 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gfhsx" (UniqueName: "kubernetes.io/projected/ddae5eb7-0626-4732-b8e7-92daf97e56a4-kube-api-access-gfhsx") pod "network-check-target-cd6rw" (UID: "ddae5eb7-0626-4732-b8e7-92daf97e56a4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:42:00.198770 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:00.198638    2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 16:42:00.299192 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:00.299125    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 16:36:58 +0000 UTC" deadline="2028-04-17 02:54:50.167016163 +0000 UTC"
Sep 17 16:42:00.299192 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:00.299190    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13858h12m49.86783062s"
Sep 17 16:42:00.347552 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:00.347520    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbdp6"
Sep 17 16:42:00.347697 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:00.347661    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbdp6" podUID="2a0d6b45-1f66-4c14-988a-1ec74eac67ad"
Sep 17 16:42:00.347777 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:00.347747    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd6rw"
Sep 17 16:42:00.347846 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:00.347825    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd6rw" podUID="ddae5eb7-0626-4732-b8e7-92daf97e56a4"
Sep 17 16:42:00.373234 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:00.373203    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-66.ec2.internal" event={"ID":"dcbcba84d85e3930495a76b3760ea833","Type":"ContainerStarted","Data":"80dfd0602aabba47234a0a9a41a8d57d560ea5406c7b4c690ee53014564b6ac6"}
Sep 17 16:42:00.391977 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:00.390328    2566 generic.go:356] "Generic (PLEG): container finished" podID="07f553a18377ccfd877b2a2f13bd91e5" containerID="cbad0f218d4fe8e14659fb2d70194b777b5b497ffc4d57f9e8841c15c1dfe7c5" exitCode=0
Sep 17 16:42:00.391977 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:00.390376    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal" event={"ID":"07f553a18377ccfd877b2a2f13bd91e5","Type":"ContainerDied","Data":"cbad0f218d4fe8e14659fb2d70194b777b5b497ffc4d57f9e8841c15c1dfe7c5"}
Sep 17 16:42:00.391977 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:00.391297    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-66.ec2.internal" podStartSLOduration=3.391279423 podStartE2EDuration="3.391279423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:41:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:00.39075301 +0000 UTC m=+6.285232150" watchObservedRunningTime="2026-09-17 16:42:00.391279423 +0000 UTC m=+6.285758563"
Sep 17 16:42:01.341741 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:01.341172    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfvrh"
Sep 17 16:42:01.341741 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:01.341316    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfvrh" podUID="fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1"
Sep 17 16:42:01.409082 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:01.409016    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal" event={"ID":"07f553a18377ccfd877b2a2f13bd91e5","Type":"ContainerStarted","Data":"9d7df6b23a90e7e245981fb4016c9389f020052dd6b110984ddf9a5815012645"}
Sep 17 16:42:01.428709 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:01.428424    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal" podStartSLOduration=4.42840353 podStartE2EDuration="4.42840353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:41:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:01.427514059 +0000 UTC m=+7.321993202" watchObservedRunningTime="2026-09-17 16:42:01.42840353 +0000 UTC m=+7.322882666"
Sep 17 16:42:01.880563 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:01.880504    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a0d6b45-1f66-4c14-988a-1ec74eac67ad-original-pull-secret\") pod \"global-pull-secret-syncer-xbdp6\" (UID: \"2a0d6b45-1f66-4c14-988a-1ec74eac67ad\") " pod="kube-system/global-pull-secret-syncer-xbdp6"
Sep 17 16:42:01.880563 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:01.880561    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1-metrics-certs\") pod \"network-metrics-daemon-pfvrh\" (UID: \"fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1\") " pod="openshift-multus/network-metrics-daemon-pfvrh"
Sep 17 16:42:01.880800 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:01.880745    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:42:01.880863 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:01.880807    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1-metrics-certs podName:fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:05.880789272 +0000 UTC m=+11.775268412 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1-metrics-certs") pod "network-metrics-daemon-pfvrh" (UID: "fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:42:01.881222 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:01.881188    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 16:42:01.881317 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:01.881259    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a0d6b45-1f66-4c14-988a-1ec74eac67ad-original-pull-secret podName:2a0d6b45-1f66-4c14-988a-1ec74eac67ad nodeName:}" failed. No retries permitted until 2026-09-17 16:42:05.88124258 +0000 UTC m=+11.775721703 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a0d6b45-1f66-4c14-988a-1ec74eac67ad-original-pull-secret") pod "global-pull-secret-syncer-xbdp6" (UID: "2a0d6b45-1f66-4c14-988a-1ec74eac67ad") : object "kube-system"/"original-pull-secret" not registered
Sep 17 16:42:01.981643 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:01.981603    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfhsx\" (UniqueName: \"kubernetes.io/projected/ddae5eb7-0626-4732-b8e7-92daf97e56a4-kube-api-access-gfhsx\") pod \"network-check-target-cd6rw\" (UID: \"ddae5eb7-0626-4732-b8e7-92daf97e56a4\") " pod="openshift-network-diagnostics/network-check-target-cd6rw"
Sep 17 16:42:01.981818 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:01.981799    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 16:42:01.981895 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:01.981826    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 16:42:01.981895 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:01.981839    2566 projected.go:196] Error preparing data for projected volume kube-api-access-gfhsx for pod openshift-network-diagnostics/network-check-target-cd6rw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:42:01.982028 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:01.981900    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ddae5eb7-0626-4732-b8e7-92daf97e56a4-kube-api-access-gfhsx podName:ddae5eb7-0626-4732-b8e7-92daf97e56a4 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:05.981881096 +0000 UTC m=+11.876360232 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gfhsx" (UniqueName: "kubernetes.io/projected/ddae5eb7-0626-4732-b8e7-92daf97e56a4-kube-api-access-gfhsx") pod "network-check-target-cd6rw" (UID: "ddae5eb7-0626-4732-b8e7-92daf97e56a4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:42:02.345213 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:02.345182    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbdp6"
Sep 17 16:42:02.345685 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:02.345309    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbdp6" podUID="2a0d6b45-1f66-4c14-988a-1ec74eac67ad"
Sep 17 16:42:02.345685 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:02.345678    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd6rw"
Sep 17 16:42:02.345797 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:02.345769    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd6rw" podUID="ddae5eb7-0626-4732-b8e7-92daf97e56a4"
Sep 17 16:42:03.341379 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:03.341346    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfvrh"
Sep 17 16:42:03.341588 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:03.341564    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfvrh" podUID="fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1"
Sep 17 16:42:04.346787 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:04.346755    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd6rw"
Sep 17 16:42:04.347162 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:04.347036    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd6rw" podUID="ddae5eb7-0626-4732-b8e7-92daf97e56a4"
Sep 17 16:42:04.347632 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:04.347614    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbdp6"
Sep 17 16:42:04.347749 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:04.347728    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbdp6" podUID="2a0d6b45-1f66-4c14-988a-1ec74eac67ad"
Sep 17 16:42:05.340996 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:05.340572    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfvrh"
Sep 17 16:42:05.340996 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:05.340689    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfvrh" podUID="fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1"
Sep 17 16:42:05.914856 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:05.914783    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a0d6b45-1f66-4c14-988a-1ec74eac67ad-original-pull-secret\") pod \"global-pull-secret-syncer-xbdp6\" (UID: \"2a0d6b45-1f66-4c14-988a-1ec74eac67ad\") " pod="kube-system/global-pull-secret-syncer-xbdp6"
Sep 17 16:42:05.914856 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:05.914837    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1-metrics-certs\") pod \"network-metrics-daemon-pfvrh\" (UID: \"fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1\") " pod="openshift-multus/network-metrics-daemon-pfvrh"
Sep 17 16:42:05.915393 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:05.915004    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:42:05.915393 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:05.915065    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1-metrics-certs podName:fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:13.915047642 +0000 UTC m=+19.809526764 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1-metrics-certs") pod "network-metrics-daemon-pfvrh" (UID: "fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:42:05.915537 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:05.915455    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 16:42:05.915537 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:05.915504    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a0d6b45-1f66-4c14-988a-1ec74eac67ad-original-pull-secret podName:2a0d6b45-1f66-4c14-988a-1ec74eac67ad nodeName:}" failed. No retries permitted until 2026-09-17 16:42:13.915488955 +0000 UTC m=+19.809968078 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a0d6b45-1f66-4c14-988a-1ec74eac67ad-original-pull-secret") pod "global-pull-secret-syncer-xbdp6" (UID: "2a0d6b45-1f66-4c14-988a-1ec74eac67ad") : object "kube-system"/"original-pull-secret" not registered
Sep 17 16:42:06.015877 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:06.015837    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfhsx\" (UniqueName: \"kubernetes.io/projected/ddae5eb7-0626-4732-b8e7-92daf97e56a4-kube-api-access-gfhsx\") pod \"network-check-target-cd6rw\" (UID: \"ddae5eb7-0626-4732-b8e7-92daf97e56a4\") " pod="openshift-network-diagnostics/network-check-target-cd6rw"
Sep 17 16:42:06.016067 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:06.016020    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 16:42:06.016067 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:06.016039    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 16:42:06.016067 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:06.016051    2566 projected.go:196] Error preparing data for projected volume kube-api-access-gfhsx for pod openshift-network-diagnostics/network-check-target-cd6rw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:42:06.016237 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:06.016101    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ddae5eb7-0626-4732-b8e7-92daf97e56a4-kube-api-access-gfhsx podName:ddae5eb7-0626-4732-b8e7-92daf97e56a4 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:14.016088972 +0000 UTC m=+19.910568091 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gfhsx" (UniqueName: "kubernetes.io/projected/ddae5eb7-0626-4732-b8e7-92daf97e56a4-kube-api-access-gfhsx") pod "network-check-target-cd6rw" (UID: "ddae5eb7-0626-4732-b8e7-92daf97e56a4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:42:06.341686 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:06.341647    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd6rw"
Sep 17 16:42:06.341846 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:06.341737    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd6rw" podUID="ddae5eb7-0626-4732-b8e7-92daf97e56a4"
Sep 17 16:42:06.342154 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:06.342132    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbdp6"
Sep 17 16:42:06.342256 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:06.342235    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbdp6" podUID="2a0d6b45-1f66-4c14-988a-1ec74eac67ad"
Sep 17 16:42:07.340937 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:07.340899    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfvrh"
Sep 17 16:42:07.341372 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:07.341037    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfvrh" podUID="fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1"
Sep 17 16:42:08.342740 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:08.342707    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd6rw"
Sep 17 16:42:08.343123 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:08.342819    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd6rw" podUID="ddae5eb7-0626-4732-b8e7-92daf97e56a4"
Sep 17 16:42:08.343180 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:08.343136    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbdp6"
Sep 17 16:42:08.343254 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:08.343220    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbdp6" podUID="2a0d6b45-1f66-4c14-988a-1ec74eac67ad"
Sep 17 16:42:09.340579 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:09.340539    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfvrh"
Sep 17 16:42:09.340733 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:09.340680    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfvrh" podUID="fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1"
Sep 17 16:42:10.341055 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:10.341014    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbdp6"
Sep 17 16:42:10.341462 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:10.341028    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd6rw"
Sep 17 16:42:10.341462 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:10.341160    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbdp6" podUID="2a0d6b45-1f66-4c14-988a-1ec74eac67ad"
Sep 17 16:42:10.341462 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:10.341233    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd6rw" podUID="ddae5eb7-0626-4732-b8e7-92daf97e56a4"
Sep 17 16:42:11.341334 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:11.341299    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfvrh"
Sep 17 16:42:11.341813 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:11.341423    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfvrh" podUID="fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1"
Sep 17 16:42:12.340990 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:12.340943    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbdp6"
Sep 17 16:42:12.341159 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:12.340943    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd6rw"
Sep 17 16:42:12.341218 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:12.341174    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd6rw" podUID="ddae5eb7-0626-4732-b8e7-92daf97e56a4"
Sep 17 16:42:12.341398 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:12.341368    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbdp6" podUID="2a0d6b45-1f66-4c14-988a-1ec74eac67ad"
Sep 17 16:42:13.340946 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:13.340914    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfvrh"
Sep 17 16:42:13.341083 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:13.341023    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfvrh" podUID="fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1"
Sep 17 16:42:13.969903 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:13.969859    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a0d6b45-1f66-4c14-988a-1ec74eac67ad-original-pull-secret\") pod \"global-pull-secret-syncer-xbdp6\" (UID: \"2a0d6b45-1f66-4c14-988a-1ec74eac67ad\") " pod="kube-system/global-pull-secret-syncer-xbdp6"
Sep 17 16:42:13.969903 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:13.969911    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1-metrics-certs\") pod \"network-metrics-daemon-pfvrh\" (UID: \"fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1\") " pod="openshift-multus/network-metrics-daemon-pfvrh"
Sep 17 16:42:13.970373 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:13.970056    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 16:42:13.970373 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:13.970073    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:42:13.970373 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:13.970127    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1-metrics-certs podName:fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:29.970107762 +0000 UTC m=+35.864586899 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1-metrics-certs") pod "network-metrics-daemon-pfvrh" (UID: "fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:42:13.970373 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:13.970144    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a0d6b45-1f66-4c14-988a-1ec74eac67ad-original-pull-secret podName:2a0d6b45-1f66-4c14-988a-1ec74eac67ad nodeName:}" failed. No retries permitted until 2026-09-17 16:42:29.970134977 +0000 UTC m=+35.864614099 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a0d6b45-1f66-4c14-988a-1ec74eac67ad-original-pull-secret") pod "global-pull-secret-syncer-xbdp6" (UID: "2a0d6b45-1f66-4c14-988a-1ec74eac67ad") : object "kube-system"/"original-pull-secret" not registered
Sep 17 16:42:14.070663 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:14.070631    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfhsx\" (UniqueName: \"kubernetes.io/projected/ddae5eb7-0626-4732-b8e7-92daf97e56a4-kube-api-access-gfhsx\") pod \"network-check-target-cd6rw\" (UID: \"ddae5eb7-0626-4732-b8e7-92daf97e56a4\") " pod="openshift-network-diagnostics/network-check-target-cd6rw"
Sep 17 16:42:14.070825 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:14.070761    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 16:42:14.070825 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:14.070782    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 16:42:14.070825 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:14.070792    2566 projected.go:196] Error preparing data for projected volume kube-api-access-gfhsx for pod openshift-network-diagnostics/network-check-target-cd6rw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:42:14.070961 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:14.070841    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ddae5eb7-0626-4732-b8e7-92daf97e56a4-kube-api-access-gfhsx podName:ddae5eb7-0626-4732-b8e7-92daf97e56a4 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:30.070824905 +0000 UTC m=+35.965304022 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gfhsx" (UniqueName: "kubernetes.io/projected/ddae5eb7-0626-4732-b8e7-92daf97e56a4-kube-api-access-gfhsx") pod "network-check-target-cd6rw" (UID: "ddae5eb7-0626-4732-b8e7-92daf97e56a4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:42:14.340646 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:14.340611    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd6rw"
Sep 17 16:42:14.340824 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:14.340665    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbdp6"
Sep 17 16:42:14.340824 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:14.340769    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd6rw" podUID="ddae5eb7-0626-4732-b8e7-92daf97e56a4"
Sep 17 16:42:14.340824 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:14.340790    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbdp6" podUID="2a0d6b45-1f66-4c14-988a-1ec74eac67ad"
Sep 17 16:42:15.341030 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:15.340997    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfvrh"
Sep 17 16:42:15.341497 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:15.341130    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfvrh" podUID="fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1"
Sep 17 16:42:16.341333 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:16.341300    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd6rw"
Sep 17 16:42:16.341782 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:16.341401    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd6rw" podUID="ddae5eb7-0626-4732-b8e7-92daf97e56a4"
Sep 17 16:42:16.341782 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:16.341468    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbdp6"
Sep 17 16:42:16.341782 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:16.341579    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbdp6" podUID="2a0d6b45-1f66-4c14-988a-1ec74eac67ad"
Sep 17 16:42:17.341562 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:17.341535    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfvrh"
Sep 17 16:42:17.341868 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:17.341695    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfvrh" podUID="fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1"
Sep 17 16:42:17.441813 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:17.441778    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-z8fgx" event={"ID":"c10930b3-04d3-4b91-8931-a9638b8c5ea7","Type":"ContainerStarted","Data":"dcf3b890ca4ae02cf37c4953cae4564fd71eed3622217a049fb46597e5a10117"}
Sep 17 16:42:17.443455 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:17.443408    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cq7pn" event={"ID":"b19f9bd7-7bff-4572-aedc-07a2c4b4a975","Type":"ContainerStarted","Data":"038cda0bef2d210b6f9a30ecc3f7cc19729fc7925d197c56051557c286c271aa"}
Sep 17 16:42:17.445108 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:17.445079    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gnzg4" event={"ID":"17d178d9-d43e-468a-9605-359baa45dc72","Type":"ContainerStarted","Data":"500fbad6721bc161fd0d066f9b48c04179fcaf4d4683174da6e56782d163737c"}
Sep 17 16:42:17.447054 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:17.447030    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nc2kn" event={"ID":"151db911-0f2b-48c8-8d4f-4fd4e3b67ebb","Type":"ContainerStarted","Data":"8e9c9e37d9794080a259ac789acb3855a9ef7f11190988c394820bda10b186eb"}
Sep 17 16:42:17.448550 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:17.448525    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-54cc4" event={"ID":"96bf0c2f-8a48-491a-bb3d-73e417abfa26","Type":"ContainerStarted","Data":"62b18e8ccb9acab4cafdd4ad6fb538d5f6cb878acb77a7e31a0886d2bb0a23da"}
Sep 17 16:42:17.450761 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:17.450738    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gblhr" event={"ID":"91477834-6648-4e75-8530-e01bdee5e57e","Type":"ContainerStarted","Data":"94b8cbda96388fb8c1ef04db61b913fd77d827551f7221b226b7ff3d329e58b1"}
Sep 17 16:42:17.450854 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:17.450811    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gblhr" event={"ID":"91477834-6648-4e75-8530-e01bdee5e57e","Type":"ContainerStarted","Data":"f6abe1e7e0e4468aefccac1e4ab0aef4674af897c4b2e87e4448b89f7d08b652"}
Sep 17 16:42:17.452466 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:17.452424    2566 generic.go:356] "Generic (PLEG): container finished" podID="367c9d70-f0c6-44ae-b24e-2ac0790b5cb7" containerID="db87122e439ef2a39cdca09555fe15bcf1558ff6398b1ea6080483b7056f5874" exitCode=0
Sep 17 16:42:17.452568 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:17.452527    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pcq95" event={"ID":"367c9d70-f0c6-44ae-b24e-2ac0790b5cb7","Type":"ContainerDied","Data":"db87122e439ef2a39cdca09555fe15bcf1558ff6398b1ea6080483b7056f5874"}
Sep 17 16:42:17.454183 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:17.454149    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mtvzk" event={"ID":"f1c8c31e-b0af-41c4-9e2b-2cb79aacccd4","Type":"ContainerStarted","Data":"2c3dacef13b388fcd8303cc74add48cda6d6df829f6242d0b3a0a3acc2ccf7ba"}
Sep 17 16:42:17.465909 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:17.465870    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-z8fgx" podStartSLOduration=7.452001968 podStartE2EDuration="21.465858577s" totalImagesPullingTime="14.013856609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:41:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:17.465670548 +0000 UTC m=+23.360149687" watchObservedRunningTime="2026-09-17 16:42:17.465858577 +0000 UTC m=+23.360337715"
Sep 17 16:42:17.485204 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:17.485161    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gnzg4" podStartSLOduration=3.938922083 podStartE2EDuration="21.485150467s" totalImagesPullingTime="17.546228384s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:41:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:17.484918707 +0000 UTC m=+23.379397846" watchObservedRunningTime="2026-09-17 16:42:17.485150467 +0000 UTC m=+23.379629606"
Sep 17 16:42:17.559049 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:17.558980    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mtvzk" podStartSLOduration=4.007701128 podStartE2EDuration="21.558957076s" totalImagesPullingTime="17.551255948s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:41:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:17.529919996 +0000 UTC m=+23.424399137" watchObservedRunningTime="2026-09-17 16:42:17.558957076 +0000 UTC m=+23.453436219"
Sep 17 16:42:17.581333 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:17.581252    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-cq7pn" podStartSLOduration=3.994340721 podStartE2EDuration="21.5812008s" totalImagesPullingTime="17.586860079s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:41:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:17.558854257 +0000 UTC m=+23.453333397" watchObservedRunningTime="2026-09-17 16:42:17.5812008 +0000 UTC m=+23.475679937"
Sep 17 16:42:17.581698 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:17.581663    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-54cc4" podStartSLOduration=4.034670815 podStartE2EDuration="21.581651272s" totalImagesPullingTime="17.546980457s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:41:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:17.580942239 +0000 UTC m=+23.475421378" watchObservedRunningTime="2026-09-17 16:42:17.581651272 +0000 UTC m=+23.476130413"
Sep 17 16:42:17.753109 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:17.752891    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-z8fgx"
Sep 17 16:42:17.753724 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:17.753701    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-z8fgx"
Sep 17 16:42:18.215903 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:18.215883    2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 16:42:18.307389 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:18.307285    2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T16:42:18.215899727Z","UUID":"ede35ee1-79f6-4c17-8e2c-fe2220a7d6a7","Handler":null,"Name":"","Endpoint":""}
Sep 17 16:42:18.309237 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:18.309216    2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 16:42:18.309362 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:18.309249    2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 16:42:18.341016 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:18.340948    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbdp6"
Sep 17 16:42:18.341144 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:18.341118    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd6rw"
Sep 17 16:42:18.341225 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:18.341194    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd6rw" podUID="ddae5eb7-0626-4732-b8e7-92daf97e56a4"
Sep 17 16:42:18.341225 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:18.341113    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbdp6" podUID="2a0d6b45-1f66-4c14-988a-1ec74eac67ad"
Sep 17 16:42:18.458640 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:18.458597    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nc2kn" event={"ID":"151db911-0f2b-48c8-8d4f-4fd4e3b67ebb","Type":"ContainerStarted","Data":"e8ed1271de52a5f6c218328872cae3fc112bb5fb451c39910ccc39f04ab9a4cf"}
Sep 17 16:42:18.461992 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:18.461962    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gblhr_91477834-6648-4e75-8530-e01bdee5e57e/ovn-acl-logging/0.log"
Sep 17 16:42:18.462543 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:18.462517    2566 generic.go:356] "Generic (PLEG): container finished" podID="91477834-6648-4e75-8530-e01bdee5e57e" containerID="94b8cbda96388fb8c1ef04db61b913fd77d827551f7221b226b7ff3d329e58b1" exitCode=1
Sep 17 16:42:18.462632 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:18.462587    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gblhr" event={"ID":"91477834-6648-4e75-8530-e01bdee5e57e","Type":"ContainerDied","Data":"94b8cbda96388fb8c1ef04db61b913fd77d827551f7221b226b7ff3d329e58b1"}
Sep 17 16:42:18.462705 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:18.462670    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gblhr" event={"ID":"91477834-6648-4e75-8530-e01bdee5e57e","Type":"ContainerStarted","Data":"6815fc5498e3c53842fffc41035d6bdb2ee1682f04c57fb78be5f3b0ef4b5007"}
Sep 17 16:42:18.462759 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:18.462740    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gblhr" event={"ID":"91477834-6648-4e75-8530-e01bdee5e57e","Type":"ContainerStarted","Data":"30de3aa448acfa06ec7e8897b12f61be972e3c75422e01d7a972abe2a1af1e8e"}
Sep 17 16:42:18.462812 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:18.462780    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gblhr" event={"ID":"91477834-6648-4e75-8530-e01bdee5e57e","Type":"ContainerStarted","Data":"41c4ddec126e6e63fa6e917e13483edb8fa86be402a2b6a98428e93917f4b4e3"}
Sep 17 16:42:18.462855 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:18.462835    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gblhr" event={"ID":"91477834-6648-4e75-8530-e01bdee5e57e","Type":"ContainerStarted","Data":"e1957e4fdceab45a748130528e0c2073ba34c0835598f56c4727ee45809301f0"}
Sep 17 16:42:18.464314 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:18.464281    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pnhbn" event={"ID":"949596f3-e4a0-4b1e-b1c8-e091aac598a2","Type":"ContainerStarted","Data":"ffcbf884ea2ad233b077fc8c67160f7e47cd18582a6559cb6ea37fbcd029c20b"}
Sep 17 16:42:18.464764 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:18.464745    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-z8fgx"
Sep 17 16:42:18.465239 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:18.465212    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-z8fgx"
Sep 17 16:42:18.498823 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:18.498778    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pnhbn" podStartSLOduration=4.947267736 podStartE2EDuration="22.498766121s" totalImagesPullingTime="17.551498385s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:41:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:18.482865133 +0000 UTC m=+24.377344295" watchObservedRunningTime="2026-09-17 16:42:18.498766121 +0000 UTC m=+24.393245261"
Sep 17 16:42:19.340698 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:19.340669    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfvrh"
Sep 17 16:42:19.340892 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:19.340817    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfvrh" podUID="fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1"
Sep 17 16:42:19.468856 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:19.468823    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nc2kn" event={"ID":"151db911-0f2b-48c8-8d4f-4fd4e3b67ebb","Type":"ContainerStarted","Data":"dc889457bb058f84e7b84f72da263ab4d5aea7c7ff23ea201937d31e97f69b77"}
Sep 17 16:42:19.488236 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:19.488190    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nc2kn" podStartSLOduration=3.7941246299999998 podStartE2EDuration="23.488174505s" totalImagesPullingTime="19.694049875s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:41:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:19.488150842 +0000 UTC m=+25.382629965" watchObservedRunningTime="2026-09-17 16:42:19.488174505 +0000 UTC m=+25.382653641"
Sep 17 16:42:20.341150 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:20.341053    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbdp6"
Sep 17 16:42:20.341310 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:20.341185    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbdp6" podUID="2a0d6b45-1f66-4c14-988a-1ec74eac67ad"
Sep 17 16:42:20.341310 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:20.341248    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd6rw"
Sep 17 16:42:20.341397 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:20.341360    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd6rw" podUID="ddae5eb7-0626-4732-b8e7-92daf97e56a4"
Sep 17 16:42:20.474291 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:20.474261    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gblhr_91477834-6648-4e75-8530-e01bdee5e57e/ovn-acl-logging/0.log"
Sep 17 16:42:20.474743 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:20.474722    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gblhr" event={"ID":"91477834-6648-4e75-8530-e01bdee5e57e","Type":"ContainerStarted","Data":"659bde43822ebbac0d6ddc1d29ec6dcb21e1023954b7f293af785c6d88635ecb"}
Sep 17 16:42:21.340901 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:21.340870    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfvrh"
Sep 17 16:42:21.341087 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:21.340985    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfvrh" podUID="fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1"
Sep 17 16:42:22.341213 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:22.341037    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbdp6"
Sep 17 16:42:22.341909 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:22.341233    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd6rw"
Sep 17 16:42:22.341909 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:22.341348    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbdp6" podUID="2a0d6b45-1f66-4c14-988a-1ec74eac67ad"
Sep 17 16:42:22.341909 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:22.341499    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd6rw" podUID="ddae5eb7-0626-4732-b8e7-92daf97e56a4"
Sep 17 16:42:22.483403 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:22.483327    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gblhr_91477834-6648-4e75-8530-e01bdee5e57e/ovn-acl-logging/0.log"
Sep 17 16:42:22.483717 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:22.483691    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gblhr" event={"ID":"91477834-6648-4e75-8530-e01bdee5e57e","Type":"ContainerStarted","Data":"13aa9d35ada39d08382b431919c264fcef037325b2a12f071d928ffacde2f99a"}
Sep 17 16:42:22.483983 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:22.483955    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:42:22.484202 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:22.484172    2566 scope.go:117] "RemoveContainer" containerID="94b8cbda96388fb8c1ef04db61b913fd77d827551f7221b226b7ff3d329e58b1"
Sep 17 16:42:22.485649 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:22.485627    2566 generic.go:356] "Generic (PLEG): container finished" podID="367c9d70-f0c6-44ae-b24e-2ac0790b5cb7" containerID="1a2679c72348a4e08a2620e569024c8e6d18116b637008f80a8f84993ceb4c63" exitCode=0
Sep 17 16:42:22.485731 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:22.485664    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pcq95" event={"ID":"367c9d70-f0c6-44ae-b24e-2ac0790b5cb7","Type":"ContainerDied","Data":"1a2679c72348a4e08a2620e569024c8e6d18116b637008f80a8f84993ceb4c63"}
Sep 17 16:42:22.500745 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:22.500721    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:42:23.341392 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:23.341362    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfvrh"
Sep 17 16:42:23.341799 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:23.341508    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfvrh" podUID="fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1"
Sep 17 16:42:23.493654 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:23.493621    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gblhr_91477834-6648-4e75-8530-e01bdee5e57e/ovn-acl-logging/0.log"
Sep 17 16:42:23.494159 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:23.494132    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gblhr" event={"ID":"91477834-6648-4e75-8530-e01bdee5e57e","Type":"ContainerStarted","Data":"2d91cc7fd20b6d116e96b5071d56baabeccb1c4b76576589cb161c3231a4ade4"}
Sep 17 16:42:23.495417 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:23.495391    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:42:23.495541 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:23.495509    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:42:23.518490 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:23.518463    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:42:23.540972 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:23.540927    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gblhr" podStartSLOduration=9.695080443 podStartE2EDuration="27.54091325s" totalImagesPullingTime="17.845832807s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:41:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:23.538773355 +0000 UTC m=+29.433252508" watchObservedRunningTime="2026-09-17 16:42:23.54091325 +0000 UTC m=+29.435392388"
Sep 17 16:42:24.239360 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:24.239110    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cd6rw"]
Sep 17 16:42:24.239551 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:24.239409    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd6rw"
Sep 17 16:42:24.239551 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:24.239534    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd6rw" podUID="ddae5eb7-0626-4732-b8e7-92daf97e56a4"
Sep 17 16:42:24.241700 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:24.241673    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pfvrh"]
Sep 17 16:42:24.241816 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:24.241786    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfvrh"
Sep 17 16:42:24.241910 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:24.241887    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfvrh" podUID="fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1"
Sep 17 16:42:24.242455 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:24.242276    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xbdp6"]
Sep 17 16:42:24.242455 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:24.242373    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbdp6"
Sep 17 16:42:24.242615 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:24.242501    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbdp6" podUID="2a0d6b45-1f66-4c14-988a-1ec74eac67ad"
Sep 17 16:42:24.498514 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:24.498414    2566 generic.go:356] "Generic (PLEG): container finished" podID="367c9d70-f0c6-44ae-b24e-2ac0790b5cb7" containerID="190b6ca05712fa2197b51e2af517ec0faf92af31a2c8cde2edaa65efd79e3421" exitCode=0
Sep 17 16:42:24.498514 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:24.498496    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pcq95" event={"ID":"367c9d70-f0c6-44ae-b24e-2ac0790b5cb7","Type":"ContainerDied","Data":"190b6ca05712fa2197b51e2af517ec0faf92af31a2c8cde2edaa65efd79e3421"}
Sep 17 16:42:26.341986 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:26.341954    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfvrh"
Sep 17 16:42:26.342405 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:26.342052    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfvrh" podUID="fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1"
Sep 17 16:42:26.342405 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:26.342120    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd6rw"
Sep 17 16:42:26.342405 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:26.342194    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbdp6"
Sep 17 16:42:26.342405 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:26.342221    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd6rw" podUID="ddae5eb7-0626-4732-b8e7-92daf97e56a4"
Sep 17 16:42:26.342405 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:26.342257    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbdp6" podUID="2a0d6b45-1f66-4c14-988a-1ec74eac67ad"
Sep 17 16:42:26.505743 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:26.505705    2566 generic.go:356] "Generic (PLEG): container finished" podID="367c9d70-f0c6-44ae-b24e-2ac0790b5cb7" containerID="6e2dd716bfaa87c1394471f78ade8b7750a121c92c25f61af39cfe4ef072855a" exitCode=0
Sep 17 16:42:26.505891 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:26.505751    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pcq95" event={"ID":"367c9d70-f0c6-44ae-b24e-2ac0790b5cb7","Type":"ContainerDied","Data":"6e2dd716bfaa87c1394471f78ade8b7750a121c92c25f61af39cfe4ef072855a"}
Sep 17 16:42:28.340887 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:28.340851    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd6rw"
Sep 17 16:42:28.341333 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:28.340851    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbdp6"
Sep 17 16:42:28.341333 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:28.340983    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd6rw" podUID="ddae5eb7-0626-4732-b8e7-92daf97e56a4"
Sep 17 16:42:28.341333 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:28.341061    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbdp6" podUID="2a0d6b45-1f66-4c14-988a-1ec74eac67ad"
Sep 17 16:42:28.341333 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:28.340852    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfvrh"
Sep 17 16:42:28.341333 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:28.341183    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfvrh" podUID="fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1"
Sep 17 16:42:29.989149 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:29.988936    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1-metrics-certs\") pod \"network-metrics-daemon-pfvrh\" (UID: \"fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1\") " pod="openshift-multus/network-metrics-daemon-pfvrh"
Sep 17 16:42:29.989568 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:29.989198    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a0d6b45-1f66-4c14-988a-1ec74eac67ad-original-pull-secret\") pod \"global-pull-secret-syncer-xbdp6\" (UID: \"2a0d6b45-1f66-4c14-988a-1ec74eac67ad\") " pod="kube-system/global-pull-secret-syncer-xbdp6"
Sep 17 16:42:29.989568 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:29.989085    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:42:29.989568 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:29.989285    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 16:42:29.989568 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:29.989296    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1-metrics-certs podName:fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1 nodeName:}" failed. No retries permitted until 2026-09-17 16:43:01.989269958 +0000 UTC m=+67.883749093 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1-metrics-certs") pod "network-metrics-daemon-pfvrh" (UID: "fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:42:29.989568 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:29.989334    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a0d6b45-1f66-4c14-988a-1ec74eac67ad-original-pull-secret podName:2a0d6b45-1f66-4c14-988a-1ec74eac67ad nodeName:}" failed. No retries permitted until 2026-09-17 16:43:01.9893183 +0000 UTC m=+67.883797432 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a0d6b45-1f66-4c14-988a-1ec74eac67ad-original-pull-secret") pod "global-pull-secret-syncer-xbdp6" (UID: "2a0d6b45-1f66-4c14-988a-1ec74eac67ad") : object "kube-system"/"original-pull-secret" not registered
Sep 17 16:42:30.089527 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:30.089498    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfhsx\" (UniqueName: \"kubernetes.io/projected/ddae5eb7-0626-4732-b8e7-92daf97e56a4-kube-api-access-gfhsx\") pod \"network-check-target-cd6rw\" (UID: \"ddae5eb7-0626-4732-b8e7-92daf97e56a4\") " pod="openshift-network-diagnostics/network-check-target-cd6rw"
Sep 17 16:42:30.089698 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:30.089637    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 16:42:30.089698 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:30.089654    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 16:42:30.089698 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:30.089663    2566 projected.go:196] Error preparing data for projected volume kube-api-access-gfhsx for pod openshift-network-diagnostics/network-check-target-cd6rw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:42:30.089831 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:30.089719    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ddae5eb7-0626-4732-b8e7-92daf97e56a4-kube-api-access-gfhsx podName:ddae5eb7-0626-4732-b8e7-92daf97e56a4 nodeName:}" failed. No retries permitted until 2026-09-17 16:43:02.089701371 +0000 UTC m=+67.984180488 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gfhsx" (UniqueName: "kubernetes.io/projected/ddae5eb7-0626-4732-b8e7-92daf97e56a4-kube-api-access-gfhsx") pod "network-check-target-cd6rw" (UID: "ddae5eb7-0626-4732-b8e7-92daf97e56a4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:42:30.340791 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:30.340711    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfvrh"
Sep 17 16:42:30.340791 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:30.340737    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd6rw"
Sep 17 16:42:30.341022 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:30.340719    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbdp6"
Sep 17 16:42:30.341022 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:30.340857    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfvrh" podUID="fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1"
Sep 17 16:42:30.341022 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:30.340985    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbdp6" podUID="2a0d6b45-1f66-4c14-988a-1ec74eac67ad"
Sep 17 16:42:30.341161 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:30.341059    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd6rw" podUID="ddae5eb7-0626-4732-b8e7-92daf97e56a4"
Sep 17 16:42:30.789412 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:30.789385    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeReady"
Sep 17 16:42:30.789582 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:30.789519    2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 16:42:30.829088 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:30.829058    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-75b54c8748-tqz7p"]
Sep 17 16:42:30.850331 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:30.850305    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dmhng"]
Sep 17 16:42:30.850490 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:30.850471    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75b54c8748-tqz7p"
Sep 17 16:42:30.852739 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:30.852706    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 16:42:30.853149 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:30.853120    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 16:42:30.853250 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:30.853210    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 16:42:30.853738 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:30.853714    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qdwd2\""
Sep 17 16:42:30.860733 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:30.860716    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 16:42:30.873989 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:30.873971    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vbswf"]
Sep 17 16:42:30.874107 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:30.874090    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dmhng"
Sep 17 16:42:30.876666 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:30.876650    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-b2jmz\""
Sep 17 16:42:30.878559 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:30.878543    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 16:42:30.878652 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:30.878569    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 16:42:30.895204 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:30.895186    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75b54c8748-tqz7p"]
Sep 17 16:42:30.895279 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:30.895213    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dmhng"]
Sep 17 16:42:30.895279 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:30.895229    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vbswf"]
Sep 17 16:42:30.895346 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:30.895288    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vbswf"
Sep 17 16:42:30.897556 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:30.897538    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-h8q8g\""
Sep 17 16:42:30.897629 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:30.897573    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 16:42:30.897788 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:30.897772    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 16:42:30.898100 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:30.898085    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 16:42:30.994233 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:30.994203    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/310fde3a-70b0-4bc6-8dd8-021b49da2f53-cert\") pod \"ingress-canary-vbswf\" (UID: \"310fde3a-70b0-4bc6-8dd8-021b49da2f53\") " pod="openshift-ingress-canary/ingress-canary-vbswf"
Sep 17 16:42:30.994233 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:30.994236    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fee0259b-0d84-4ecc-96c0-16ab13596d57-ca-trust-extracted\") pod \"image-registry-75b54c8748-tqz7p\" (UID: \"fee0259b-0d84-4ecc-96c0-16ab13596d57\") " pod="openshift-image-registry/image-registry-75b54c8748-tqz7p"
Sep 17 16:42:30.994653 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:30.994259    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fee0259b-0d84-4ecc-96c0-16ab13596d57-bound-sa-token\") pod \"image-registry-75b54c8748-tqz7p\" (UID: \"fee0259b-0d84-4ecc-96c0-16ab13596d57\") " pod="openshift-image-registry/image-registry-75b54c8748-tqz7p"
Sep 17 16:42:30.994653 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:30.994279    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fee0259b-0d84-4ecc-96c0-16ab13596d57-registry-certificates\") pod \"image-registry-75b54c8748-tqz7p\" (UID: \"fee0259b-0d84-4ecc-96c0-16ab13596d57\") " pod="openshift-image-registry/image-registry-75b54c8748-tqz7p"
Sep 17 16:42:30.994653 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:30.994313    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fee0259b-0d84-4ecc-96c0-16ab13596d57-trusted-ca\") pod \"image-registry-75b54c8748-tqz7p\" (UID: \"fee0259b-0d84-4ecc-96c0-16ab13596d57\") " pod="openshift-image-registry/image-registry-75b54c8748-tqz7p"
Sep 17 16:42:30.994653 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:30.994336    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fee0259b-0d84-4ecc-96c0-16ab13596d57-installation-pull-secrets\") pod \"image-registry-75b54c8748-tqz7p\" (UID: \"fee0259b-0d84-4ecc-96c0-16ab13596d57\") " pod="openshift-image-registry/image-registry-75b54c8748-tqz7p"
Sep 17 16:42:30.994653 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:30.994363    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh8rs\" (UniqueName: \"kubernetes.io/projected/fee0259b-0d84-4ecc-96c0-16ab13596d57-kube-api-access-rh8rs\") pod \"image-registry-75b54c8748-tqz7p\" (UID: \"fee0259b-0d84-4ecc-96c0-16ab13596d57\") " pod="openshift-image-registry/image-registry-75b54c8748-tqz7p"
Sep 17 16:42:30.994653 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:30.994423    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fee0259b-0d84-4ecc-96c0-16ab13596d57-image-registry-private-configuration\") pod \"image-registry-75b54c8748-tqz7p\" (UID: \"fee0259b-0d84-4ecc-96c0-16ab13596d57\") " pod="openshift-image-registry/image-registry-75b54c8748-tqz7p"
Sep 17 16:42:30.994653 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:30.994477    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnl8m\" (UniqueName: \"kubernetes.io/projected/7ce87b87-3574-43de-9939-9be53e78b3e3-kube-api-access-bnl8m\") pod \"dns-default-dmhng\" (UID: \"7ce87b87-3574-43de-9939-9be53e78b3e3\") " pod="openshift-dns/dns-default-dmhng"
Sep 17 16:42:30.994653 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:30.994500    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7ce87b87-3574-43de-9939-9be53e78b3e3-config-volume\") pod \"dns-default-dmhng\" (UID: \"7ce87b87-3574-43de-9939-9be53e78b3e3\") " pod="openshift-dns/dns-default-dmhng"
Sep 17 16:42:30.994653 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:30.994560    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fee0259b-0d84-4ecc-96c0-16ab13596d57-registry-tls\") pod \"image-registry-75b54c8748-tqz7p\" (UID: \"fee0259b-0d84-4ecc-96c0-16ab13596d57\") " pod="openshift-image-registry/image-registry-75b54c8748-tqz7p"
Sep 17 16:42:30.994653 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:30.994585    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l4qv\" (UniqueName: \"kubernetes.io/projected/310fde3a-70b0-4bc6-8dd8-021b49da2f53-kube-api-access-4l4qv\") pod \"ingress-canary-vbswf\" (UID: \"310fde3a-70b0-4bc6-8dd8-021b49da2f53\") " pod="openshift-ingress-canary/ingress-canary-vbswf"
Sep 17 16:42:30.994653 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:30.994603    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ce87b87-3574-43de-9939-9be53e78b3e3-tmp-dir\") pod \"dns-default-dmhng\" (UID: \"7ce87b87-3574-43de-9939-9be53e78b3e3\") " pod="openshift-dns/dns-default-dmhng"
Sep 17 16:42:30.994653 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:30.994629    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7ce87b87-3574-43de-9939-9be53e78b3e3-metrics-tls\") pod \"dns-default-dmhng\" (UID: \"7ce87b87-3574-43de-9939-9be53e78b3e3\") " pod="openshift-dns/dns-default-dmhng"
Sep 17 16:42:31.095610 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:31.095586    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fee0259b-0d84-4ecc-96c0-16ab13596d57-ca-trust-extracted\") pod \"image-registry-75b54c8748-tqz7p\" (UID: \"fee0259b-0d84-4ecc-96c0-16ab13596d57\") " pod="openshift-image-registry/image-registry-75b54c8748-tqz7p"
Sep 17 16:42:31.095693 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:31.095612    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fee0259b-0d84-4ecc-96c0-16ab13596d57-bound-sa-token\") pod \"image-registry-75b54c8748-tqz7p\" (UID: \"fee0259b-0d84-4ecc-96c0-16ab13596d57\") " pod="openshift-image-registry/image-registry-75b54c8748-tqz7p"
Sep 17 16:42:31.095693 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:31.095628    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fee0259b-0d84-4ecc-96c0-16ab13596d57-registry-certificates\") pod \"image-registry-75b54c8748-tqz7p\" (UID: \"fee0259b-0d84-4ecc-96c0-16ab13596d57\") " pod="openshift-image-registry/image-registry-75b54c8748-tqz7p"
Sep 17 16:42:31.095693 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:31.095643    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fee0259b-0d84-4ecc-96c0-16ab13596d57-trusted-ca\") pod \"image-registry-75b54c8748-tqz7p\" (UID: \"fee0259b-0d84-4ecc-96c0-16ab13596d57\") " pod="openshift-image-registry/image-registry-75b54c8748-tqz7p"
Sep 17 16:42:31.095693 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:31.095660    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fee0259b-0d84-4ecc-96c0-16ab13596d57-installation-pull-secrets\") pod \"image-registry-75b54c8748-tqz7p\" (UID: \"fee0259b-0d84-4ecc-96c0-16ab13596d57\") " pod="openshift-image-registry/image-registry-75b54c8748-tqz7p"
Sep 17 16:42:31.095693 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:31.095674    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh8rs\" (UniqueName: \"kubernetes.io/projected/fee0259b-0d84-4ecc-96c0-16ab13596d57-kube-api-access-rh8rs\") pod \"image-registry-75b54c8748-tqz7p\" (UID: \"fee0259b-0d84-4ecc-96c0-16ab13596d57\") " pod="openshift-image-registry/image-registry-75b54c8748-tqz7p"
Sep 17 16:42:31.095925 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:31.095701    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fee0259b-0d84-4ecc-96c0-16ab13596d57-image-registry-private-configuration\") pod \"image-registry-75b54c8748-tqz7p\" (UID: \"fee0259b-0d84-4ecc-96c0-16ab13596d57\") " pod="openshift-image-registry/image-registry-75b54c8748-tqz7p"
Sep 17 16:42:31.095925 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:31.095730    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnl8m\" (UniqueName: \"kubernetes.io/projected/7ce87b87-3574-43de-9939-9be53e78b3e3-kube-api-access-bnl8m\") pod \"dns-default-dmhng\" (UID: \"7ce87b87-3574-43de-9939-9be53e78b3e3\") " pod="openshift-dns/dns-default-dmhng"
Sep 17 16:42:31.095925 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:31.095757    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7ce87b87-3574-43de-9939-9be53e78b3e3-config-volume\") pod \"dns-default-dmhng\" (UID: \"7ce87b87-3574-43de-9939-9be53e78b3e3\") " pod="openshift-dns/dns-default-dmhng"
Sep 17 16:42:31.095925 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:31.095788    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fee0259b-0d84-4ecc-96c0-16ab13596d57-registry-tls\") pod \"image-registry-75b54c8748-tqz7p\" (UID: \"fee0259b-0d84-4ecc-96c0-16ab13596d57\") " pod="openshift-image-registry/image-registry-75b54c8748-tqz7p"
Sep 17 16:42:31.095925 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:31.095815    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l4qv\" (UniqueName: \"kubernetes.io/projected/310fde3a-70b0-4bc6-8dd8-021b49da2f53-kube-api-access-4l4qv\") pod \"ingress-canary-vbswf\" (UID: \"310fde3a-70b0-4bc6-8dd8-021b49da2f53\") " pod="openshift-ingress-canary/ingress-canary-vbswf"
Sep 17 16:42:31.095925 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:31.095838    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ce87b87-3574-43de-9939-9be53e78b3e3-tmp-dir\") pod \"dns-default-dmhng\" (UID: \"7ce87b87-3574-43de-9939-9be53e78b3e3\") " pod="openshift-dns/dns-default-dmhng"
Sep 17 16:42:31.095925 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:31.095860    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7ce87b87-3574-43de-9939-9be53e78b3e3-metrics-tls\") pod \"dns-default-dmhng\" (UID: \"7ce87b87-3574-43de-9939-9be53e78b3e3\") " pod="openshift-dns/dns-default-dmhng"
Sep 17 16:42:31.095925 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:31.095917    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/310fde3a-70b0-4bc6-8dd8-021b49da2f53-cert\") pod \"ingress-canary-vbswf\" (UID: \"310fde3a-70b0-4bc6-8dd8-021b49da2f53\") " pod="openshift-ingress-canary/ingress-canary-vbswf"
Sep 17 16:42:31.096271 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:31.095963    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 16:42:31.096271 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:31.095978    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75b54c8748-tqz7p: secret "image-registry-tls" not found
Sep 17 16:42:31.096271 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:31.095998    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fee0259b-0d84-4ecc-96c0-16ab13596d57-ca-trust-extracted\") pod \"image-registry-75b54c8748-tqz7p\" (UID: \"fee0259b-0d84-4ecc-96c0-16ab13596d57\") " pod="openshift-image-registry/image-registry-75b54c8748-tqz7p"
Sep 17 16:42:31.096271 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:31.096023    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fee0259b-0d84-4ecc-96c0-16ab13596d57-registry-tls podName:fee0259b-0d84-4ecc-96c0-16ab13596d57 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:31.596006464 +0000 UTC m=+37.490485581 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fee0259b-0d84-4ecc-96c0-16ab13596d57-registry-tls") pod "image-registry-75b54c8748-tqz7p" (UID: "fee0259b-0d84-4ecc-96c0-16ab13596d57") : secret "image-registry-tls" not found
Sep 17 16:42:31.096506 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:31.096311    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 16:42:31.096506 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:31.096365    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ce87b87-3574-43de-9939-9be53e78b3e3-metrics-tls podName:7ce87b87-3574-43de-9939-9be53e78b3e3 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:31.596345099 +0000 UTC m=+37.490824252 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7ce87b87-3574-43de-9939-9be53e78b3e3-metrics-tls") pod "dns-default-dmhng" (UID: "7ce87b87-3574-43de-9939-9be53e78b3e3") : secret "dns-default-metrics-tls" not found
Sep 17 16:42:31.096506 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:31.096307    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ce87b87-3574-43de-9939-9be53e78b3e3-tmp-dir\") pod \"dns-default-dmhng\" (UID: \"7ce87b87-3574-43de-9939-9be53e78b3e3\") " pod="openshift-dns/dns-default-dmhng"
Sep 17 16:42:31.096650 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:31.096631    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7ce87b87-3574-43de-9939-9be53e78b3e3-config-volume\") pod \"dns-default-dmhng\" (UID: \"7ce87b87-3574-43de-9939-9be53e78b3e3\") " pod="openshift-dns/dns-default-dmhng"
Sep 17 16:42:31.096719 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:31.096701    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fee0259b-0d84-4ecc-96c0-16ab13596d57-trusted-ca\") pod \"image-registry-75b54c8748-tqz7p\" (UID: \"fee0259b-0d84-4ecc-96c0-16ab13596d57\") " pod="openshift-image-registry/image-registry-75b54c8748-tqz7p"
Sep 17 16:42:31.096806 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:31.096791    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fee0259b-0d84-4ecc-96c0-16ab13596d57-registry-certificates\") pod \"image-registry-75b54c8748-tqz7p\" (UID: \"fee0259b-0d84-4ecc-96c0-16ab13596d57\") " pod="openshift-image-registry/image-registry-75b54c8748-tqz7p"
Sep 17 16:42:31.100269 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:31.100245    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fee0259b-0d84-4ecc-96c0-16ab13596d57-installation-pull-secrets\") pod \"image-registry-75b54c8748-tqz7p\" (UID: \"fee0259b-0d84-4ecc-96c0-16ab13596d57\") " pod="openshift-image-registry/image-registry-75b54c8748-tqz7p"
Sep 17 16:42:31.100357 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:31.100268    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/310fde3a-70b0-4bc6-8dd8-021b49da2f53-cert\") pod \"ingress-canary-vbswf\" (UID: \"310fde3a-70b0-4bc6-8dd8-021b49da2f53\") " pod="openshift-ingress-canary/ingress-canary-vbswf"
Sep 17 16:42:31.100666 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:31.100649    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fee0259b-0d84-4ecc-96c0-16ab13596d57-image-registry-private-configuration\") pod \"image-registry-75b54c8748-tqz7p\" (UID: \"fee0259b-0d84-4ecc-96c0-16ab13596d57\") " pod="openshift-image-registry/image-registry-75b54c8748-tqz7p"
Sep 17 16:42:31.107925 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:31.107902    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l4qv\" (UniqueName: \"kubernetes.io/projected/310fde3a-70b0-4bc6-8dd8-021b49da2f53-kube-api-access-4l4qv\") pod \"ingress-canary-vbswf\" (UID: \"310fde3a-70b0-4bc6-8dd8-021b49da2f53\") " pod="openshift-ingress-canary/ingress-canary-vbswf"
Sep 17 16:42:31.108284 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:31.108232    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnl8m\" (UniqueName: \"kubernetes.io/projected/7ce87b87-3574-43de-9939-9be53e78b3e3-kube-api-access-bnl8m\") pod \"dns-default-dmhng\" (UID: \"7ce87b87-3574-43de-9939-9be53e78b3e3\") " pod="openshift-dns/dns-default-dmhng"
Sep 17 16:42:31.108349 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:31.108322    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fee0259b-0d84-4ecc-96c0-16ab13596d57-bound-sa-token\") pod \"image-registry-75b54c8748-tqz7p\" (UID: \"fee0259b-0d84-4ecc-96c0-16ab13596d57\") " pod="openshift-image-registry/image-registry-75b54c8748-tqz7p"
Sep 17 16:42:31.110033 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:31.109238    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh8rs\" (UniqueName: \"kubernetes.io/projected/fee0259b-0d84-4ecc-96c0-16ab13596d57-kube-api-access-rh8rs\") pod \"image-registry-75b54c8748-tqz7p\" (UID: \"fee0259b-0d84-4ecc-96c0-16ab13596d57\") " pod="openshift-image-registry/image-registry-75b54c8748-tqz7p"
Sep 17 16:42:31.202844 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:31.202819    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vbswf"
Sep 17 16:42:31.490900 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:31.490833    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vbswf"]
Sep 17 16:42:31.494420 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:42:31.494380    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod310fde3a_70b0_4bc6_8dd8_021b49da2f53.slice/crio-611711f3f50de7e19b75ca3303facbf9c1d96e3df384d00c01fb7c484dcff621 WatchSource:0}: Error finding container 611711f3f50de7e19b75ca3303facbf9c1d96e3df384d00c01fb7c484dcff621: Status 404 returned error can't find the container with id 611711f3f50de7e19b75ca3303facbf9c1d96e3df384d00c01fb7c484dcff621
Sep 17 16:42:31.521528 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:31.521496    2566 generic.go:356] "Generic (PLEG): container finished" podID="367c9d70-f0c6-44ae-b24e-2ac0790b5cb7" containerID="d534abd737568068e17e9ba0da44ba4602b6fd75a6fe4d23774fa37b9761e246" exitCode=0
Sep 17 16:42:31.521625 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:31.521585    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pcq95" event={"ID":"367c9d70-f0c6-44ae-b24e-2ac0790b5cb7","Type":"ContainerDied","Data":"d534abd737568068e17e9ba0da44ba4602b6fd75a6fe4d23774fa37b9761e246"}
Sep 17 16:42:31.522861 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:31.522736    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vbswf" event={"ID":"310fde3a-70b0-4bc6-8dd8-021b49da2f53","Type":"ContainerStarted","Data":"611711f3f50de7e19b75ca3303facbf9c1d96e3df384d00c01fb7c484dcff621"}
Sep 17 16:42:31.598953 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:31.598918    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fee0259b-0d84-4ecc-96c0-16ab13596d57-registry-tls\") pod \"image-registry-75b54c8748-tqz7p\" (UID: \"fee0259b-0d84-4ecc-96c0-16ab13596d57\") " pod="openshift-image-registry/image-registry-75b54c8748-tqz7p"
Sep 17 16:42:31.599043 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:31.598965    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7ce87b87-3574-43de-9939-9be53e78b3e3-metrics-tls\") pod \"dns-default-dmhng\" (UID: \"7ce87b87-3574-43de-9939-9be53e78b3e3\") " pod="openshift-dns/dns-default-dmhng"
Sep 17 16:42:31.599128 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:31.599095    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 16:42:31.599128 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:31.599110    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 16:42:31.599235 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:31.599130    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75b54c8748-tqz7p: secret "image-registry-tls" not found
Sep 17 16:42:31.599235 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:31.599161    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ce87b87-3574-43de-9939-9be53e78b3e3-metrics-tls podName:7ce87b87-3574-43de-9939-9be53e78b3e3 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:32.599141962 +0000 UTC m=+38.493621105 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7ce87b87-3574-43de-9939-9be53e78b3e3-metrics-tls") pod "dns-default-dmhng" (UID: "7ce87b87-3574-43de-9939-9be53e78b3e3") : secret "dns-default-metrics-tls" not found
Sep 17 16:42:31.599235 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:31.599192    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fee0259b-0d84-4ecc-96c0-16ab13596d57-registry-tls podName:fee0259b-0d84-4ecc-96c0-16ab13596d57 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:32.599171295 +0000 UTC m=+38.493650434 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fee0259b-0d84-4ecc-96c0-16ab13596d57-registry-tls") pod "image-registry-75b54c8748-tqz7p" (UID: "fee0259b-0d84-4ecc-96c0-16ab13596d57") : secret "image-registry-tls" not found
Sep 17 16:42:32.341459 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:32.341241    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbdp6"
Sep 17 16:42:32.342212 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:32.341245    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd6rw"
Sep 17 16:42:32.342212 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:32.341295    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfvrh"
Sep 17 16:42:32.345113 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:32.344856    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 16:42:32.345113 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:32.344892    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 16:42:32.345113 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:32.344928    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 16:42:32.345113 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:32.344943    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 16:42:32.345113 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:32.344951    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k782z\""
Sep 17 16:42:32.345113 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:32.344897    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hqd8r\""
Sep 17 16:42:32.529369 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:32.529338    2566 generic.go:356] "Generic (PLEG): container finished" podID="367c9d70-f0c6-44ae-b24e-2ac0790b5cb7" containerID="4d2a5359e7c70fea5be31f6ebbb61d70c0a727264dbd697ba660f804ab1c8fea" exitCode=0
Sep 17 16:42:32.529547 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:32.529382    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pcq95" event={"ID":"367c9d70-f0c6-44ae-b24e-2ac0790b5cb7","Type":"ContainerDied","Data":"4d2a5359e7c70fea5be31f6ebbb61d70c0a727264dbd697ba660f804ab1c8fea"}
Sep 17 16:42:32.608592 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:32.608515    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fee0259b-0d84-4ecc-96c0-16ab13596d57-registry-tls\") pod \"image-registry-75b54c8748-tqz7p\" (UID: \"fee0259b-0d84-4ecc-96c0-16ab13596d57\") " pod="openshift-image-registry/image-registry-75b54c8748-tqz7p"
Sep 17 16:42:32.608592 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:32.608553    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7ce87b87-3574-43de-9939-9be53e78b3e3-metrics-tls\") pod \"dns-default-dmhng\" (UID: \"7ce87b87-3574-43de-9939-9be53e78b3e3\") " pod="openshift-dns/dns-default-dmhng"
Sep 17 16:42:32.609164 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:32.609143    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 16:42:32.609164 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:32.609160    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 16:42:32.609338 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:32.609176    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75b54c8748-tqz7p: secret "image-registry-tls" not found
Sep 17 16:42:32.609338 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:32.609195    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ce87b87-3574-43de-9939-9be53e78b3e3-metrics-tls podName:7ce87b87-3574-43de-9939-9be53e78b3e3 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:34.609180973 +0000 UTC m=+40.503660091 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7ce87b87-3574-43de-9939-9be53e78b3e3-metrics-tls") pod "dns-default-dmhng" (UID: "7ce87b87-3574-43de-9939-9be53e78b3e3") : secret "dns-default-metrics-tls" not found
Sep 17 16:42:32.609338 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:32.609220    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fee0259b-0d84-4ecc-96c0-16ab13596d57-registry-tls podName:fee0259b-0d84-4ecc-96c0-16ab13596d57 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:34.609203984 +0000 UTC m=+40.503683102 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fee0259b-0d84-4ecc-96c0-16ab13596d57-registry-tls") pod "image-registry-75b54c8748-tqz7p" (UID: "fee0259b-0d84-4ecc-96c0-16ab13596d57") : secret "image-registry-tls" not found
Sep 17 16:42:33.534901 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:33.534868    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pcq95" event={"ID":"367c9d70-f0c6-44ae-b24e-2ac0790b5cb7","Type":"ContainerStarted","Data":"bcb2e05d71f863c18b299ff782a40f107fed95c2b669e3803ed4d42a767db3f9"}
Sep 17 16:42:33.559020 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:33.558969    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-pcq95" podStartSLOduration=8.403071498 podStartE2EDuration="37.558953562s" totalImagesPullingTime="29.155882064s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:41:56 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:33.558534719 +0000 UTC m=+39.453013858" watchObservedRunningTime="2026-09-17 16:42:33.558953562 +0000 UTC m=+39.453432700"
Sep 17 16:42:34.538711 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:34.538637    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vbswf" event={"ID":"310fde3a-70b0-4bc6-8dd8-021b49da2f53","Type":"ContainerStarted","Data":"7249d67b92ca716fa6fc64be2b7537e28c506e666cf3c793ef686a5c51da3033"}
Sep 17 16:42:34.556761 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:34.556719    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vbswf" podStartSLOduration=2.482063831 podStartE2EDuration="4.556704259s" totalImagesPullingTime="2.074640428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:42:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:34.556130898 +0000 UTC m=+40.450610035" watchObservedRunningTime="2026-09-17 16:42:34.556704259 +0000 UTC m=+40.451183397"
Sep 17 16:42:34.622396 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:34.622364    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fee0259b-0d84-4ecc-96c0-16ab13596d57-registry-tls\") pod \"image-registry-75b54c8748-tqz7p\" (UID: \"fee0259b-0d84-4ecc-96c0-16ab13596d57\") " pod="openshift-image-registry/image-registry-75b54c8748-tqz7p"
Sep 17 16:42:34.622573 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:34.622417    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7ce87b87-3574-43de-9939-9be53e78b3e3-metrics-tls\") pod \"dns-default-dmhng\" (UID: \"7ce87b87-3574-43de-9939-9be53e78b3e3\") " pod="openshift-dns/dns-default-dmhng"
Sep 17 16:42:34.622638 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:34.622579    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 16:42:34.622638 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:34.622588    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 16:42:34.622638 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:34.622610    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75b54c8748-tqz7p: secret "image-registry-tls" not found
Sep 17 16:42:34.622780 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:34.622665    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fee0259b-0d84-4ecc-96c0-16ab13596d57-registry-tls podName:fee0259b-0d84-4ecc-96c0-16ab13596d57 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:38.622647348 +0000 UTC m=+44.517126471 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fee0259b-0d84-4ecc-96c0-16ab13596d57-registry-tls") pod "image-registry-75b54c8748-tqz7p" (UID: "fee0259b-0d84-4ecc-96c0-16ab13596d57") : secret "image-registry-tls" not found
Sep 17 16:42:34.622848 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:34.622830    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ce87b87-3574-43de-9939-9be53e78b3e3-metrics-tls podName:7ce87b87-3574-43de-9939-9be53e78b3e3 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:38.622815775 +0000 UTC m=+44.517294892 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7ce87b87-3574-43de-9939-9be53e78b3e3-metrics-tls") pod "dns-default-dmhng" (UID: "7ce87b87-3574-43de-9939-9be53e78b3e3") : secret "dns-default-metrics-tls" not found
Sep 17 16:42:35.706229 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:35.706197    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nnsgj"]
Sep 17 16:42:35.710243 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:35.710228    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nnsgj"
Sep 17 16:42:35.714780 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:35.714756    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 16:42:35.715803 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:35.715785    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 16:42:35.715861 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:35.715829    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cp4pd\""
Sep 17 16:42:35.716096 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:35.716078    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 16:42:35.716207 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:35.716192    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 16:42:35.727785 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:35.727766    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nnsgj"]
Sep 17 16:42:35.830407 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:35.830378    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a654b952-b3e3-4827-8d9d-9e346f66857f-data-volume\") pod \"insights-runtime-extractor-nnsgj\" (UID: \"a654b952-b3e3-4827-8d9d-9e346f66857f\") " pod="openshift-insights/insights-runtime-extractor-nnsgj"
Sep 17 16:42:35.830407 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:35.830411    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a654b952-b3e3-4827-8d9d-9e346f66857f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nnsgj\" (UID: \"a654b952-b3e3-4827-8d9d-9e346f66857f\") " pod="openshift-insights/insights-runtime-extractor-nnsgj"
Sep 17 16:42:35.830638 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:35.830491    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a654b952-b3e3-4827-8d9d-9e346f66857f-crio-socket\") pod \"insights-runtime-extractor-nnsgj\" (UID: \"a654b952-b3e3-4827-8d9d-9e346f66857f\") " pod="openshift-insights/insights-runtime-extractor-nnsgj"
Sep 17 16:42:35.830638 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:35.830523    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a654b952-b3e3-4827-8d9d-9e346f66857f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nnsgj\" (UID: \"a654b952-b3e3-4827-8d9d-9e346f66857f\") " pod="openshift-insights/insights-runtime-extractor-nnsgj"
Sep 17 16:42:35.830638 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:35.830577    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8v7v\" (UniqueName: \"kubernetes.io/projected/a654b952-b3e3-4827-8d9d-9e346f66857f-kube-api-access-g8v7v\") pod \"insights-runtime-extractor-nnsgj\" (UID: \"a654b952-b3e3-4827-8d9d-9e346f66857f\") " pod="openshift-insights/insights-runtime-extractor-nnsgj"
Sep 17 16:42:35.931730 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:35.931689    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a654b952-b3e3-4827-8d9d-9e346f66857f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nnsgj\" (UID: \"a654b952-b3e3-4827-8d9d-9e346f66857f\") " pod="openshift-insights/insights-runtime-extractor-nnsgj"
Sep 17 16:42:35.931939 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:35.931751    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a654b952-b3e3-4827-8d9d-9e346f66857f-crio-socket\") pod \"insights-runtime-extractor-nnsgj\" (UID: \"a654b952-b3e3-4827-8d9d-9e346f66857f\") " pod="openshift-insights/insights-runtime-extractor-nnsgj"
Sep 17 16:42:35.931939 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:35.931776    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a654b952-b3e3-4827-8d9d-9e346f66857f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nnsgj\" (UID: \"a654b952-b3e3-4827-8d9d-9e346f66857f\") " pod="openshift-insights/insights-runtime-extractor-nnsgj"
Sep 17 16:42:35.931939 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:35.931799    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8v7v\" (UniqueName: \"kubernetes.io/projected/a654b952-b3e3-4827-8d9d-9e346f66857f-kube-api-access-g8v7v\") pod \"insights-runtime-extractor-nnsgj\" (UID: \"a654b952-b3e3-4827-8d9d-9e346f66857f\") " pod="openshift-insights/insights-runtime-extractor-nnsgj"
Sep 17 16:42:35.931939 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:35.931823    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 16:42:35.931939 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:35.931845    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a654b952-b3e3-4827-8d9d-9e346f66857f-data-volume\") pod \"insights-runtime-extractor-nnsgj\" (UID: \"a654b952-b3e3-4827-8d9d-9e346f66857f\") " pod="openshift-insights/insights-runtime-extractor-nnsgj"
Sep 17 16:42:35.931939 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:35.931914    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a654b952-b3e3-4827-8d9d-9e346f66857f-kube-rbac-proxy-cm podName:a654b952-b3e3-4827-8d9d-9e346f66857f nodeName:}" failed. No retries permitted until 2026-09-17 16:42:36.431896649 +0000 UTC m=+42.326375792 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a654b952-b3e3-4827-8d9d-9e346f66857f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nnsgj" (UID: "a654b952-b3e3-4827-8d9d-9e346f66857f") : configmap "kube-rbac-proxy" not found
Sep 17 16:42:35.932159 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:35.931943    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 16:42:35.932159 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:35.932007    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a654b952-b3e3-4827-8d9d-9e346f66857f-insights-runtime-extractor-tls podName:a654b952-b3e3-4827-8d9d-9e346f66857f nodeName:}" failed. No retries permitted until 2026-09-17 16:42:36.431991435 +0000 UTC m=+42.326470555 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a654b952-b3e3-4827-8d9d-9e346f66857f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nnsgj" (UID: "a654b952-b3e3-4827-8d9d-9e346f66857f") : secret "insights-runtime-extractor-tls" not found
Sep 17 16:42:35.932159 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:35.932044    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a654b952-b3e3-4827-8d9d-9e346f66857f-crio-socket\") pod \"insights-runtime-extractor-nnsgj\" (UID: \"a654b952-b3e3-4827-8d9d-9e346f66857f\") " pod="openshift-insights/insights-runtime-extractor-nnsgj"
Sep 17 16:42:35.932245 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:35.932218    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a654b952-b3e3-4827-8d9d-9e346f66857f-data-volume\") pod \"insights-runtime-extractor-nnsgj\" (UID: \"a654b952-b3e3-4827-8d9d-9e346f66857f\") " pod="openshift-insights/insights-runtime-extractor-nnsgj"
Sep 17 16:42:35.950882 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:35.950858    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8v7v\" (UniqueName: \"kubernetes.io/projected/a654b952-b3e3-4827-8d9d-9e346f66857f-kube-api-access-g8v7v\") pod \"insights-runtime-extractor-nnsgj\" (UID: \"a654b952-b3e3-4827-8d9d-9e346f66857f\") " pod="openshift-insights/insights-runtime-extractor-nnsgj"
Sep 17 16:42:36.435843 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:36.435801    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a654b952-b3e3-4827-8d9d-9e346f66857f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nnsgj\" (UID: \"a654b952-b3e3-4827-8d9d-9e346f66857f\") " pod="openshift-insights/insights-runtime-extractor-nnsgj"
Sep 17 16:42:36.436047 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:36.435863    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a654b952-b3e3-4827-8d9d-9e346f66857f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nnsgj\" (UID: \"a654b952-b3e3-4827-8d9d-9e346f66857f\") " pod="openshift-insights/insights-runtime-extractor-nnsgj"
Sep 17 16:42:36.436047 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:36.435938    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 16:42:36.436047 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:36.435953    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 16:42:36.436047 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:36.436001    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a654b952-b3e3-4827-8d9d-9e346f66857f-kube-rbac-proxy-cm podName:a654b952-b3e3-4827-8d9d-9e346f66857f nodeName:}" failed. No retries permitted until 2026-09-17 16:42:37.435983428 +0000 UTC m=+43.330462559 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a654b952-b3e3-4827-8d9d-9e346f66857f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nnsgj" (UID: "a654b952-b3e3-4827-8d9d-9e346f66857f") : configmap "kube-rbac-proxy" not found
Sep 17 16:42:36.436047 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:36.436017    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a654b952-b3e3-4827-8d9d-9e346f66857f-insights-runtime-extractor-tls podName:a654b952-b3e3-4827-8d9d-9e346f66857f nodeName:}" failed. No retries permitted until 2026-09-17 16:42:37.436009085 +0000 UTC m=+43.330488221 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a654b952-b3e3-4827-8d9d-9e346f66857f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nnsgj" (UID: "a654b952-b3e3-4827-8d9d-9e346f66857f") : secret "insights-runtime-extractor-tls" not found
Sep 17 16:42:37.047496 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:37.047467    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gnzg4_17d178d9-d43e-468a-9605-359baa45dc72/dns-node-resolver/0.log"
Sep 17 16:42:37.442664 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:37.442579    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a654b952-b3e3-4827-8d9d-9e346f66857f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nnsgj\" (UID: \"a654b952-b3e3-4827-8d9d-9e346f66857f\") " pod="openshift-insights/insights-runtime-extractor-nnsgj"
Sep 17 16:42:37.442664 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:37.442635    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a654b952-b3e3-4827-8d9d-9e346f66857f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nnsgj\" (UID: \"a654b952-b3e3-4827-8d9d-9e346f66857f\") " pod="openshift-insights/insights-runtime-extractor-nnsgj"
Sep 17 16:42:37.442838 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:37.442712    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 16:42:37.442838 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:37.442778    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a654b952-b3e3-4827-8d9d-9e346f66857f-kube-rbac-proxy-cm podName:a654b952-b3e3-4827-8d9d-9e346f66857f nodeName:}" failed. No retries permitted until 2026-09-17 16:42:39.442760108 +0000 UTC m=+45.337239239 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a654b952-b3e3-4827-8d9d-9e346f66857f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nnsgj" (UID: "a654b952-b3e3-4827-8d9d-9e346f66857f") : configmap "kube-rbac-proxy" not found
Sep 17 16:42:37.442838 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:37.442789    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 16:42:37.442838 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:37.442822    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a654b952-b3e3-4827-8d9d-9e346f66857f-insights-runtime-extractor-tls podName:a654b952-b3e3-4827-8d9d-9e346f66857f nodeName:}" failed. No retries permitted until 2026-09-17 16:42:39.442811472 +0000 UTC m=+45.337290591 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a654b952-b3e3-4827-8d9d-9e346f66857f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nnsgj" (UID: "a654b952-b3e3-4827-8d9d-9e346f66857f") : secret "insights-runtime-extractor-tls" not found
Sep 17 16:42:38.050169 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:38.050144    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-54cc4_96bf0c2f-8a48-491a-bb3d-73e417abfa26/node-ca/0.log"
Sep 17 16:42:38.652415 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:38.652369    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fee0259b-0d84-4ecc-96c0-16ab13596d57-registry-tls\") pod \"image-registry-75b54c8748-tqz7p\" (UID: \"fee0259b-0d84-4ecc-96c0-16ab13596d57\") " pod="openshift-image-registry/image-registry-75b54c8748-tqz7p"
Sep 17 16:42:38.652415 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:38.652415    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7ce87b87-3574-43de-9939-9be53e78b3e3-metrics-tls\") pod \"dns-default-dmhng\" (UID: \"7ce87b87-3574-43de-9939-9be53e78b3e3\") " pod="openshift-dns/dns-default-dmhng"
Sep 17 16:42:38.652701 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:38.652537    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 16:42:38.652701 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:38.652549    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 16:42:38.652701 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:38.652568    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75b54c8748-tqz7p: secret "image-registry-tls" not found
Sep 17 16:42:38.652701 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:38.652589    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ce87b87-3574-43de-9939-9be53e78b3e3-metrics-tls podName:7ce87b87-3574-43de-9939-9be53e78b3e3 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:46.652576092 +0000 UTC m=+52.547055208 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7ce87b87-3574-43de-9939-9be53e78b3e3-metrics-tls") pod "dns-default-dmhng" (UID: "7ce87b87-3574-43de-9939-9be53e78b3e3") : secret "dns-default-metrics-tls" not found
Sep 17 16:42:38.652701 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:38.652620    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fee0259b-0d84-4ecc-96c0-16ab13596d57-registry-tls podName:fee0259b-0d84-4ecc-96c0-16ab13596d57 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:46.652603496 +0000 UTC m=+52.547082619 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fee0259b-0d84-4ecc-96c0-16ab13596d57-registry-tls") pod "image-registry-75b54c8748-tqz7p" (UID: "fee0259b-0d84-4ecc-96c0-16ab13596d57") : secret "image-registry-tls" not found
Sep 17 16:42:39.052491 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:39.052467    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vbswf_310fde3a-70b0-4bc6-8dd8-021b49da2f53/serve-healthcheck-canary/0.log"
Sep 17 16:42:39.456832 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:39.456745    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a654b952-b3e3-4827-8d9d-9e346f66857f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nnsgj\" (UID: \"a654b952-b3e3-4827-8d9d-9e346f66857f\") " pod="openshift-insights/insights-runtime-extractor-nnsgj"
Sep 17 16:42:39.456832 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:39.456806    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a654b952-b3e3-4827-8d9d-9e346f66857f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nnsgj\" (UID: \"a654b952-b3e3-4827-8d9d-9e346f66857f\") " pod="openshift-insights/insights-runtime-extractor-nnsgj"
Sep 17 16:42:39.456990 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:39.456868    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 16:42:39.456990 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:39.456922    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 16:42:39.456990 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:39.456931    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a654b952-b3e3-4827-8d9d-9e346f66857f-kube-rbac-proxy-cm podName:a654b952-b3e3-4827-8d9d-9e346f66857f nodeName:}" failed. No retries permitted until 2026-09-17 16:42:43.456915425 +0000 UTC m=+49.351394560 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a654b952-b3e3-4827-8d9d-9e346f66857f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nnsgj" (UID: "a654b952-b3e3-4827-8d9d-9e346f66857f") : configmap "kube-rbac-proxy" not found
Sep 17 16:42:39.456990 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:39.456961    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a654b952-b3e3-4827-8d9d-9e346f66857f-insights-runtime-extractor-tls podName:a654b952-b3e3-4827-8d9d-9e346f66857f nodeName:}" failed. No retries permitted until 2026-09-17 16:42:43.456950209 +0000 UTC m=+49.351429326 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a654b952-b3e3-4827-8d9d-9e346f66857f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nnsgj" (UID: "a654b952-b3e3-4827-8d9d-9e346f66857f") : secret "insights-runtime-extractor-tls" not found
Sep 17 16:42:43.485310 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:43.485275    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a654b952-b3e3-4827-8d9d-9e346f66857f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nnsgj\" (UID: \"a654b952-b3e3-4827-8d9d-9e346f66857f\") " pod="openshift-insights/insights-runtime-extractor-nnsgj"
Sep 17 16:42:43.485722 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:43.485354    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a654b952-b3e3-4827-8d9d-9e346f66857f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nnsgj\" (UID: \"a654b952-b3e3-4827-8d9d-9e346f66857f\") " pod="openshift-insights/insights-runtime-extractor-nnsgj"
Sep 17 16:42:43.485722 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:43.485451    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 16:42:43.485722 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:43.485522    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a654b952-b3e3-4827-8d9d-9e346f66857f-insights-runtime-extractor-tls podName:a654b952-b3e3-4827-8d9d-9e346f66857f nodeName:}" failed. No retries permitted until 2026-09-17 16:42:51.485506372 +0000 UTC m=+57.379985489 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a654b952-b3e3-4827-8d9d-9e346f66857f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nnsgj" (UID: "a654b952-b3e3-4827-8d9d-9e346f66857f") : secret "insights-runtime-extractor-tls" not found
Sep 17 16:42:43.485863 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:43.485847    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a654b952-b3e3-4827-8d9d-9e346f66857f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nnsgj\" (UID: \"a654b952-b3e3-4827-8d9d-9e346f66857f\") " pod="openshift-insights/insights-runtime-extractor-nnsgj"
Sep 17 16:42:46.712247 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:46.712209    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fee0259b-0d84-4ecc-96c0-16ab13596d57-registry-tls\") pod \"image-registry-75b54c8748-tqz7p\" (UID: \"fee0259b-0d84-4ecc-96c0-16ab13596d57\") " pod="openshift-image-registry/image-registry-75b54c8748-tqz7p"
Sep 17 16:42:46.712247 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:46.712250    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7ce87b87-3574-43de-9939-9be53e78b3e3-metrics-tls\") pod \"dns-default-dmhng\" (UID: \"7ce87b87-3574-43de-9939-9be53e78b3e3\") " pod="openshift-dns/dns-default-dmhng"
Sep 17 16:42:46.712746 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:46.712361    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 16:42:46.712746 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:46.712380    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75b54c8748-tqz7p: secret "image-registry-tls" not found
Sep 17 16:42:46.712746 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:42:46.712446    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fee0259b-0d84-4ecc-96c0-16ab13596d57-registry-tls podName:fee0259b-0d84-4ecc-96c0-16ab13596d57 nodeName:}" failed. No retries permitted until 2026-09-17 16:43:02.712418958 +0000 UTC m=+68.606898075 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fee0259b-0d84-4ecc-96c0-16ab13596d57-registry-tls") pod "image-registry-75b54c8748-tqz7p" (UID: "fee0259b-0d84-4ecc-96c0-16ab13596d57") : secret "image-registry-tls" not found
Sep 17 16:42:46.714684 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:46.714665    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7ce87b87-3574-43de-9939-9be53e78b3e3-metrics-tls\") pod \"dns-default-dmhng\" (UID: \"7ce87b87-3574-43de-9939-9be53e78b3e3\") " pod="openshift-dns/dns-default-dmhng"
Sep 17 16:42:46.782255 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:46.782215    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dmhng"
Sep 17 16:42:47.095450 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:47.095412    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dmhng"]
Sep 17 16:42:47.097904 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:42:47.097877    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ce87b87_3574_43de_9939_9be53e78b3e3.slice/crio-3cbeb104d8af4d9e660f8fe8a74ffc3e53f751d395839d8c25aac463af38a7b2 WatchSource:0}: Error finding container 3cbeb104d8af4d9e660f8fe8a74ffc3e53f751d395839d8c25aac463af38a7b2: Status 404 returned error can't find the container with id 3cbeb104d8af4d9e660f8fe8a74ffc3e53f751d395839d8c25aac463af38a7b2
Sep 17 16:42:47.567425 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:47.567391    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dmhng" event={"ID":"7ce87b87-3574-43de-9939-9be53e78b3e3","Type":"ContainerStarted","Data":"3cbeb104d8af4d9e660f8fe8a74ffc3e53f751d395839d8c25aac463af38a7b2"}
Sep 17 16:42:49.574261 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:49.574224    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dmhng" event={"ID":"7ce87b87-3574-43de-9939-9be53e78b3e3","Type":"ContainerStarted","Data":"b746e653aa7c3ab6f1155daa4b1690e28b09029ce5bc7dc9cf1e150dbb8032c3"}
Sep 17 16:42:49.574630 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:49.574318    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dmhng" event={"ID":"7ce87b87-3574-43de-9939-9be53e78b3e3","Type":"ContainerStarted","Data":"d6f11f94174aebca29ad0fa26cbb4cd52d809a6aafc7ddc7f70af7d919c2e8be"}
Sep 17 16:42:49.574630 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:49.574456    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dmhng"
Sep 17 16:42:49.594078 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:49.594035    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-dmhng" podStartSLOduration=17.44561944 podStartE2EDuration="19.594018469s" totalImagesPullingTime="2.148399029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:42:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:49.593784485 +0000 UTC m=+55.488263625" watchObservedRunningTime="2026-09-17 16:42:49.594018469 +0000 UTC m=+55.488497608"
Sep 17 16:42:51.545986 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:51.545946    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a654b952-b3e3-4827-8d9d-9e346f66857f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nnsgj\" (UID: \"a654b952-b3e3-4827-8d9d-9e346f66857f\") " pod="openshift-insights/insights-runtime-extractor-nnsgj"
Sep 17 16:42:51.548400 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:51.548376    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a654b952-b3e3-4827-8d9d-9e346f66857f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nnsgj\" (UID: \"a654b952-b3e3-4827-8d9d-9e346f66857f\") " pod="openshift-insights/insights-runtime-extractor-nnsgj"
Sep 17 16:42:51.619486 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:51.619454    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nnsgj"
Sep 17 16:42:51.923614 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:51.923544    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nnsgj"]
Sep 17 16:42:51.926704 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:42:51.926675    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda654b952_b3e3_4827_8d9d_9e346f66857f.slice/crio-0512474e5f18f1b631cec118f87cb0fbe4bcc983f5c930ee58c3cf2457aa4814 WatchSource:0}: Error finding container 0512474e5f18f1b631cec118f87cb0fbe4bcc983f5c930ee58c3cf2457aa4814: Status 404 returned error can't find the container with id 0512474e5f18f1b631cec118f87cb0fbe4bcc983f5c930ee58c3cf2457aa4814
Sep 17 16:42:52.583350 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:52.583319    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nnsgj" event={"ID":"a654b952-b3e3-4827-8d9d-9e346f66857f","Type":"ContainerStarted","Data":"2886eba61ac98b082174642b64f451f0919e181c9f7cd6de5abfad1f325a47df"}
Sep 17 16:42:52.583705 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:52.583377    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nnsgj" event={"ID":"a654b952-b3e3-4827-8d9d-9e346f66857f","Type":"ContainerStarted","Data":"0512474e5f18f1b631cec118f87cb0fbe4bcc983f5c930ee58c3cf2457aa4814"}
Sep 17 16:42:53.587787 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:53.587748    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nnsgj" event={"ID":"a654b952-b3e3-4827-8d9d-9e346f66857f","Type":"ContainerStarted","Data":"fe228f3a1ee05784d89574005fc177d5a8e3fe6faf9ef98013ce149345098050"}
Sep 17 16:42:55.511797 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:55.511770    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gblhr"
Sep 17 16:42:55.596053 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:55.596022    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nnsgj" event={"ID":"a654b952-b3e3-4827-8d9d-9e346f66857f","Type":"ContainerStarted","Data":"48ad7a8edf18280a5d4fdefc59a750d91621fead8a45ce61ad520dfd23728d09"}
Sep 17 16:42:55.615600 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:55.615559    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-nnsgj" podStartSLOduration=18.050827274 podStartE2EDuration="20.615545785s" totalImagesPullingTime="2.564718511s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:42:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:55.615002951 +0000 UTC m=+61.509482112" watchObservedRunningTime="2026-09-17 16:42:55.615545785 +0000 UTC m=+61.510024916"
Sep 17 16:42:59.579529 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:42:59.579499    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dmhng"
Sep 17 16:43:02.017589 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.017555    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a0d6b45-1f66-4c14-988a-1ec74eac67ad-original-pull-secret\") pod \"global-pull-secret-syncer-xbdp6\" (UID: \"2a0d6b45-1f66-4c14-988a-1ec74eac67ad\") " pod="kube-system/global-pull-secret-syncer-xbdp6"
Sep 17 16:43:02.017589 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.017590    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1-metrics-certs\") pod \"network-metrics-daemon-pfvrh\" (UID: \"fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1\") " pod="openshift-multus/network-metrics-daemon-pfvrh"
Sep 17 16:43:02.020021 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.020002    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 16:43:02.020128 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.020005    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 16:43:02.030231 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.030203    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a0d6b45-1f66-4c14-988a-1ec74eac67ad-original-pull-secret\") pod \"global-pull-secret-syncer-xbdp6\" (UID: \"2a0d6b45-1f66-4c14-988a-1ec74eac67ad\") " pod="kube-system/global-pull-secret-syncer-xbdp6"
Sep 17 16:43:02.030231 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.030229    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1-metrics-certs\") pod \"network-metrics-daemon-pfvrh\" (UID: \"fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1\") " pod="openshift-multus/network-metrics-daemon-pfvrh"
Sep 17 16:43:02.052320 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.052295    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbdp6"
Sep 17 16:43:02.068210 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.068177    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k782z\""
Sep 17 16:43:02.075122 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.075100    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfvrh"
Sep 17 16:43:02.119052 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.118663    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfhsx\" (UniqueName: \"kubernetes.io/projected/ddae5eb7-0626-4732-b8e7-92daf97e56a4-kube-api-access-gfhsx\") pod \"network-check-target-cd6rw\" (UID: \"ddae5eb7-0626-4732-b8e7-92daf97e56a4\") " pod="openshift-network-diagnostics/network-check-target-cd6rw"
Sep 17 16:43:02.121465 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.121425    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 16:43:02.132139 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.132104    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 16:43:02.142753 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.142732    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfhsx\" (UniqueName: \"kubernetes.io/projected/ddae5eb7-0626-4732-b8e7-92daf97e56a4-kube-api-access-gfhsx\") pod \"network-check-target-cd6rw\" (UID: \"ddae5eb7-0626-4732-b8e7-92daf97e56a4\") " pod="openshift-network-diagnostics/network-check-target-cd6rw"
Sep 17 16:43:02.330076 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.330043    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xbdp6"]
Sep 17 16:43:02.334193 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:43:02.333734    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a0d6b45_1f66_4c14_988a_1ec74eac67ad.slice/crio-25b83d1c3ebefcb471aacd0e93d1cc692f0fbfc47c553dfc19b870852219f49a WatchSource:0}: Error finding container 25b83d1c3ebefcb471aacd0e93d1cc692f0fbfc47c553dfc19b870852219f49a: Status 404 returned error can't find the container with id 25b83d1c3ebefcb471aacd0e93d1cc692f0fbfc47c553dfc19b870852219f49a
Sep 17 16:43:02.361551 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.361528    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hqd8r\""
Sep 17 16:43:02.369460 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.369422    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd6rw"
Sep 17 16:43:02.370461 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.370426    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pfvrh"]
Sep 17 16:43:02.376826 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:43:02.376803    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc8d33fc_7b6c_47e2_b59d_5a27761bc0f1.slice/crio-5d4cacda96fe1d19996c5c302e4bda63bf708b0659e1862b5d0e4925f0851897 WatchSource:0}: Error finding container 5d4cacda96fe1d19996c5c302e4bda63bf708b0659e1862b5d0e4925f0851897: Status 404 returned error can't find the container with id 5d4cacda96fe1d19996c5c302e4bda63bf708b0659e1862b5d0e4925f0851897
Sep 17 16:43:02.458823 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.458795    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-868744fd8d-hwvmm"]
Sep 17 16:43:02.461412 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.461391    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-868744fd8d-hwvmm"
Sep 17 16:43:02.463553 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.463533    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 16:43:02.463660 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.463630    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-hsqxp\""
Sep 17 16:43:02.464849 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.464830    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 16:43:02.464955 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.464919    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 16:43:02.465194 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.465180    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 17 16:43:02.467232 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.467213    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d6d4666c-ddd5t"]
Sep 17 16:43:02.469413 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.469395    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d6d4666c-ddd5t"
Sep 17 16:43:02.478549 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.478528    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 17 16:43:02.478673 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.478626    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 17 16:43:02.478746 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.478729    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 17 16:43:02.478800 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.478732    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 17 16:43:02.486370 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.486343    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-868744fd8d-hwvmm"]
Sep 17 16:43:02.487120 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.487102    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d6d4666c-ddd5t"]
Sep 17 16:43:02.520749 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.520711    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3b7cb2b2-40e7-42d3-8ea6-01252a36b68e-hub\") pod \"cluster-proxy-proxy-agent-59d6d4666c-ddd5t\" (UID: \"3b7cb2b2-40e7-42d3-8ea6-01252a36b68e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d6d4666c-ddd5t"
Sep 17 16:43:02.520749 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.520743    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3b7cb2b2-40e7-42d3-8ea6-01252a36b68e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-59d6d4666c-ddd5t\" (UID: \"3b7cb2b2-40e7-42d3-8ea6-01252a36b68e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d6d4666c-ddd5t"
Sep 17 16:43:02.520933 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.520791    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b7cb2b2-40e7-42d3-8ea6-01252a36b68e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-59d6d4666c-ddd5t\" (UID: \"3b7cb2b2-40e7-42d3-8ea6-01252a36b68e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d6d4666c-ddd5t"
Sep 17 16:43:02.520933 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.520811    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/567524ca-393b-4466-929b-a8ca6c34bc98-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-868744fd8d-hwvmm\" (UID: \"567524ca-393b-4466-929b-a8ca6c34bc98\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-868744fd8d-hwvmm"
Sep 17 16:43:02.520933 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.520831    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3b7cb2b2-40e7-42d3-8ea6-01252a36b68e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-59d6d4666c-ddd5t\" (UID: \"3b7cb2b2-40e7-42d3-8ea6-01252a36b68e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d6d4666c-ddd5t"
Sep 17 16:43:02.520933 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.520865    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvj5f\" (UniqueName: \"kubernetes.io/projected/567524ca-393b-4466-929b-a8ca6c34bc98-kube-api-access-kvj5f\") pod \"managed-serviceaccount-addon-agent-868744fd8d-hwvmm\" (UID: \"567524ca-393b-4466-929b-a8ca6c34bc98\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-868744fd8d-hwvmm"
Sep 17 16:43:02.520933 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.520919    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3b7cb2b2-40e7-42d3-8ea6-01252a36b68e-ca\") pod \"cluster-proxy-proxy-agent-59d6d4666c-ddd5t\" (UID: \"3b7cb2b2-40e7-42d3-8ea6-01252a36b68e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d6d4666c-ddd5t"
Sep 17 16:43:02.521101 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.520948    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcgqr\" (UniqueName: \"kubernetes.io/projected/3b7cb2b2-40e7-42d3-8ea6-01252a36b68e-kube-api-access-kcgqr\") pod \"cluster-proxy-proxy-agent-59d6d4666c-ddd5t\" (UID: \"3b7cb2b2-40e7-42d3-8ea6-01252a36b68e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d6d4666c-ddd5t"
Sep 17 16:43:02.617545 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.617468    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pfvrh" event={"ID":"fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1","Type":"ContainerStarted","Data":"5d4cacda96fe1d19996c5c302e4bda63bf708b0659e1862b5d0e4925f0851897"}
Sep 17 16:43:02.618747 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.618707    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xbdp6" event={"ID":"2a0d6b45-1f66-4c14-988a-1ec74eac67ad","Type":"ContainerStarted","Data":"25b83d1c3ebefcb471aacd0e93d1cc692f0fbfc47c553dfc19b870852219f49a"}
Sep 17 16:43:02.622089 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.622065    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3b7cb2b2-40e7-42d3-8ea6-01252a36b68e-hub\") pod \"cluster-proxy-proxy-agent-59d6d4666c-ddd5t\" (UID: \"3b7cb2b2-40e7-42d3-8ea6-01252a36b68e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d6d4666c-ddd5t"
Sep 17 16:43:02.622167 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.622093    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3b7cb2b2-40e7-42d3-8ea6-01252a36b68e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-59d6d4666c-ddd5t\" (UID: \"3b7cb2b2-40e7-42d3-8ea6-01252a36b68e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d6d4666c-ddd5t"
Sep 17 16:43:02.622167 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.622122    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b7cb2b2-40e7-42d3-8ea6-01252a36b68e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-59d6d4666c-ddd5t\" (UID: \"3b7cb2b2-40e7-42d3-8ea6-01252a36b68e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d6d4666c-ddd5t"
Sep 17 16:43:02.622167 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.622139    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/567524ca-393b-4466-929b-a8ca6c34bc98-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-868744fd8d-hwvmm\" (UID: \"567524ca-393b-4466-929b-a8ca6c34bc98\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-868744fd8d-hwvmm"
Sep 17 16:43:02.622167 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.622157    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3b7cb2b2-40e7-42d3-8ea6-01252a36b68e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-59d6d4666c-ddd5t\" (UID: \"3b7cb2b2-40e7-42d3-8ea6-01252a36b68e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d6d4666c-ddd5t"
Sep 17 16:43:02.622339 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.622182    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvj5f\" (UniqueName: \"kubernetes.io/projected/567524ca-393b-4466-929b-a8ca6c34bc98-kube-api-access-kvj5f\") pod \"managed-serviceaccount-addon-agent-868744fd8d-hwvmm\" (UID: \"567524ca-393b-4466-929b-a8ca6c34bc98\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-868744fd8d-hwvmm"
Sep 17 16:43:02.622339 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.622200    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3b7cb2b2-40e7-42d3-8ea6-01252a36b68e-ca\") pod \"cluster-proxy-proxy-agent-59d6d4666c-ddd5t\" (UID: \"3b7cb2b2-40e7-42d3-8ea6-01252a36b68e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d6d4666c-ddd5t"
Sep 17 16:43:02.622339 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.622214    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcgqr\" (UniqueName: \"kubernetes.io/projected/3b7cb2b2-40e7-42d3-8ea6-01252a36b68e-kube-api-access-kcgqr\") pod \"cluster-proxy-proxy-agent-59d6d4666c-ddd5t\" (UID: \"3b7cb2b2-40e7-42d3-8ea6-01252a36b68e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d6d4666c-ddd5t"
Sep 17 16:43:02.622930 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.622873    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b7cb2b2-40e7-42d3-8ea6-01252a36b68e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-59d6d4666c-ddd5t\" (UID: \"3b7cb2b2-40e7-42d3-8ea6-01252a36b68e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d6d4666c-ddd5t"
Sep 17 16:43:02.624856 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.624827    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3b7cb2b2-40e7-42d3-8ea6-01252a36b68e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-59d6d4666c-ddd5t\" (UID: \"3b7cb2b2-40e7-42d3-8ea6-01252a36b68e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d6d4666c-ddd5t"
Sep 17 16:43:02.624927 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.624878    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3b7cb2b2-40e7-42d3-8ea6-01252a36b68e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-59d6d4666c-ddd5t\" (UID: \"3b7cb2b2-40e7-42d3-8ea6-01252a36b68e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d6d4666c-ddd5t"
Sep 17 16:43:02.624995 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.624973    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/567524ca-393b-4466-929b-a8ca6c34bc98-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-868744fd8d-hwvmm\" (UID: \"567524ca-393b-4466-929b-a8ca6c34bc98\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-868744fd8d-hwvmm"
Sep 17 16:43:02.625101 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.625086    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3b7cb2b2-40e7-42d3-8ea6-01252a36b68e-ca\") pod \"cluster-proxy-proxy-agent-59d6d4666c-ddd5t\" (UID: \"3b7cb2b2-40e7-42d3-8ea6-01252a36b68e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d6d4666c-ddd5t"
Sep 17 16:43:02.625136 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.625107    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3b7cb2b2-40e7-42d3-8ea6-01252a36b68e-hub\") pod \"cluster-proxy-proxy-agent-59d6d4666c-ddd5t\" (UID: \"3b7cb2b2-40e7-42d3-8ea6-01252a36b68e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d6d4666c-ddd5t"
Sep 17 16:43:02.648021 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.647995    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvj5f\" (UniqueName: \"kubernetes.io/projected/567524ca-393b-4466-929b-a8ca6c34bc98-kube-api-access-kvj5f\") pod \"managed-serviceaccount-addon-agent-868744fd8d-hwvmm\" (UID: \"567524ca-393b-4466-929b-a8ca6c34bc98\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-868744fd8d-hwvmm"
Sep 17 16:43:02.649075 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.649056    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcgqr\" (UniqueName: \"kubernetes.io/projected/3b7cb2b2-40e7-42d3-8ea6-01252a36b68e-kube-api-access-kcgqr\") pod \"cluster-proxy-proxy-agent-59d6d4666c-ddd5t\" (UID: \"3b7cb2b2-40e7-42d3-8ea6-01252a36b68e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d6d4666c-ddd5t"
Sep 17 16:43:02.659579 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.659558    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cd6rw"]
Sep 17 16:43:02.668543 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:43:02.668522    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddae5eb7_0626_4732_b8e7_92daf97e56a4.slice/crio-7d88f6b2953b563bb75212c8bbe6391305e4a54048a6d12749cfa5cee863cb8d WatchSource:0}: Error finding container 7d88f6b2953b563bb75212c8bbe6391305e4a54048a6d12749cfa5cee863cb8d: Status 404 returned error can't find the container with id 7d88f6b2953b563bb75212c8bbe6391305e4a54048a6d12749cfa5cee863cb8d
Sep 17 16:43:02.722979 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.722951    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fee0259b-0d84-4ecc-96c0-16ab13596d57-registry-tls\") pod \"image-registry-75b54c8748-tqz7p\" (UID: \"fee0259b-0d84-4ecc-96c0-16ab13596d57\") " pod="openshift-image-registry/image-registry-75b54c8748-tqz7p"
Sep 17 16:43:02.725348 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.725328    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fee0259b-0d84-4ecc-96c0-16ab13596d57-registry-tls\") pod \"image-registry-75b54c8748-tqz7p\" (UID: \"fee0259b-0d84-4ecc-96c0-16ab13596d57\") " pod="openshift-image-registry/image-registry-75b54c8748-tqz7p"
Sep 17 16:43:02.778835 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.778811    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-868744fd8d-hwvmm"
Sep 17 16:43:02.786429 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.786414    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d6d4666c-ddd5t"
Sep 17 16:43:02.964558 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.964465    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qdwd2\""
Sep 17 16:43:02.972445 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:02.972417    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75b54c8748-tqz7p"
Sep 17 16:43:03.057348 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:03.057282    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-868744fd8d-hwvmm"]
Sep 17 16:43:03.077846 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:03.077743    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d6d4666c-ddd5t"]
Sep 17 16:43:03.081449 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:43:03.081409    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b7cb2b2_40e7_42d3_8ea6_01252a36b68e.slice/crio-390a4b3722fb3fc4075ceba23c418b1f6eec063487cf6de9457f179e664d82ce WatchSource:0}: Error finding container 390a4b3722fb3fc4075ceba23c418b1f6eec063487cf6de9457f179e664d82ce: Status 404 returned error can't find the container with id 390a4b3722fb3fc4075ceba23c418b1f6eec063487cf6de9457f179e664d82ce
Sep 17 16:43:03.263490 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:03.263400    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75b54c8748-tqz7p"]
Sep 17 16:43:03.405461 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:43:03.402123    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfee0259b_0d84_4ecc_96c0_16ab13596d57.slice/crio-c7536a2dc8f8880785220cbeb07faa8ee3a40169e0d0fa79571682a8840dccf2 WatchSource:0}: Error finding container c7536a2dc8f8880785220cbeb07faa8ee3a40169e0d0fa79571682a8840dccf2: Status 404 returned error can't find the container with id c7536a2dc8f8880785220cbeb07faa8ee3a40169e0d0fa79571682a8840dccf2
Sep 17 16:43:03.627777 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:03.627631    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-868744fd8d-hwvmm" event={"ID":"567524ca-393b-4466-929b-a8ca6c34bc98","Type":"ContainerStarted","Data":"51e59d7377adb6f43212a276f1b438706d3257d3315d4cd05534704ddbb3e930"}
Sep 17 16:43:03.631774 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:03.631691    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cd6rw" event={"ID":"ddae5eb7-0626-4732-b8e7-92daf97e56a4","Type":"ContainerStarted","Data":"7d88f6b2953b563bb75212c8bbe6391305e4a54048a6d12749cfa5cee863cb8d"}
Sep 17 16:43:03.634257 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:03.634224    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d6d4666c-ddd5t" event={"ID":"3b7cb2b2-40e7-42d3-8ea6-01252a36b68e","Type":"ContainerStarted","Data":"390a4b3722fb3fc4075ceba23c418b1f6eec063487cf6de9457f179e664d82ce"}
Sep 17 16:43:03.636956 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:03.636924    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pfvrh" event={"ID":"fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1","Type":"ContainerStarted","Data":"a1b321f97d368814ebe7ed8fa06a7076d86b16a99aaaedad242b4a9c988311a8"}
Sep 17 16:43:03.641330 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:03.640573    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75b54c8748-tqz7p" event={"ID":"fee0259b-0d84-4ecc-96c0-16ab13596d57","Type":"ContainerStarted","Data":"3d34ce4bfd3090b80696dd9f1e0e77abac9833cc8a36807c6dc550422f54f6a1"}
Sep 17 16:43:03.641330 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:03.640617    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75b54c8748-tqz7p" event={"ID":"fee0259b-0d84-4ecc-96c0-16ab13596d57","Type":"ContainerStarted","Data":"c7536a2dc8f8880785220cbeb07faa8ee3a40169e0d0fa79571682a8840dccf2"}
Sep 17 16:43:03.641330 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:03.641293    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-75b54c8748-tqz7p"
Sep 17 16:43:03.667795 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:03.667737    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-75b54c8748-tqz7p" podStartSLOduration=65.66771566 podStartE2EDuration="1m5.66771566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:41:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:03.663280078 +0000 UTC m=+69.557759219" watchObservedRunningTime="2026-09-17 16:43:03.66771566 +0000 UTC m=+69.562194800"
Sep 17 16:43:04.651624 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:04.651487    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pfvrh" event={"ID":"fc8d33fc-7b6c-47e2-b59d-5a27761bc0f1","Type":"ContainerStarted","Data":"198cc7e96b2294bb6f5df092cae3c70d6bb36befcefe9c44d19c086610788942"}
Sep 17 16:43:04.675659 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:04.675458    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-pfvrh" podStartSLOduration=67.624862996 podStartE2EDuration="1m8.675419097s" totalImagesPullingTime="1.050556101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:41:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:04.670593871 +0000 UTC m=+70.565073012" watchObservedRunningTime="2026-09-17 16:43:04.675419097 +0000 UTC m=+70.569898237"
Sep 17 16:43:10.674648 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:10.674610    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d6d4666c-ddd5t" event={"ID":"3b7cb2b2-40e7-42d3-8ea6-01252a36b68e","Type":"ContainerStarted","Data":"bce2bdfc4328155abf709c283efb2ffc11d5a8b03ce2c4dc94adf2f611349214"}
Sep 17 16:43:10.675105 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:10.674665    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d6d4666c-ddd5t" event={"ID":"3b7cb2b2-40e7-42d3-8ea6-01252a36b68e","Type":"ContainerStarted","Data":"2fd7c71b77b9c9b4d28391b83f4284401c8185204ced9a674c1d9354a2e1e4c7"}
Sep 17 16:43:10.675105 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:10.674681    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d6d4666c-ddd5t" event={"ID":"3b7cb2b2-40e7-42d3-8ea6-01252a36b68e","Type":"ContainerStarted","Data":"b6c3d348fc95b050b6e3afaa2c2899d46ef3e7bdcc16959c24065bff1767f655"}
Sep 17 16:43:10.676201 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:10.676175    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xbdp6" event={"ID":"2a0d6b45-1f66-4c14-988a-1ec74eac67ad","Type":"ContainerStarted","Data":"1ca46782b6bcee6d0d6d70c92e7da738c11b4a18bc94d3ead3167572c12a81d3"}
Sep 17 16:43:10.677565 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:10.677544    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-868744fd8d-hwvmm" event={"ID":"567524ca-393b-4466-929b-a8ca6c34bc98","Type":"ContainerStarted","Data":"7960f46b9e3f0266a8a04138611dd9297ecd782c95a18d6e6a68047e69387f62"}
Sep 17 16:43:10.679095 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:10.679075    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cd6rw" event={"ID":"ddae5eb7-0626-4732-b8e7-92daf97e56a4","Type":"ContainerStarted","Data":"2fbdba6be8a299672b3b770f25394cc19b2ebb6f8efeef88a7757f79aef3b582"}
Sep 17 16:43:10.679214 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:10.679198    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cd6rw"
Sep 17 16:43:10.702076 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:10.702023    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d6d4666c-ddd5t" podStartSLOduration=1.731396174 podStartE2EDuration="8.702008797s" totalImagesPullingTime="6.970612623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:43:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:10.701634804 +0000 UTC m=+76.596113944" watchObservedRunningTime="2026-09-17 16:43:10.702008797 +0000 UTC m=+76.596487935"
Sep 17 16:43:10.722807 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:10.722764    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-868744fd8d-hwvmm" podStartSLOduration=1.752366065 podStartE2EDuration="8.722748641s" totalImagesPullingTime="6.970382576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:43:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:10.721469416 +0000 UTC m=+76.615948546" watchObservedRunningTime="2026-09-17 16:43:10.722748641 +0000 UTC m=+76.617227780"
Sep 17 16:43:10.750375 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:10.750320    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-cd6rw" podStartSLOduration=67.381787817 podStartE2EDuration="1m14.750304204s" totalImagesPullingTime="7.368516387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:41:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:10.749934294 +0000 UTC m=+76.644413434" watchObservedRunningTime="2026-09-17 16:43:10.750304204 +0000 UTC m=+76.644783345"
Sep 17 16:43:10.772809 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:10.772770    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xbdp6" podStartSLOduration=67.066949861 podStartE2EDuration="1m14.772757624s" totalImagesPullingTime="7.705807763s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:41:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:10.771548334 +0000 UTC m=+76.666027473" watchObservedRunningTime="2026-09-17 16:43:10.772757624 +0000 UTC m=+76.667236763"
Sep 17 16:43:16.463594 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.463565    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-pxb89"]
Sep 17 16:43:16.466394 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.466376    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pxb89"
Sep 17 16:43:16.469044 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.469014    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 16:43:16.469333 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.469312    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 16:43:16.469810 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.469793    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 17 16:43:16.469810 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.469800    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-892wv\""
Sep 17 16:43:16.470050 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.470038    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 16:43:16.470096 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.470052    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 17 16:43:16.479002 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.478979    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-l5ppz"]
Sep 17 16:43:16.481625 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.481606    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-pxb89"]
Sep 17 16:43:16.481746 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.481722    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l5ppz"
Sep 17 16:43:16.484540 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.484427    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 16:43:16.484645 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.484611    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 16:43:16.485026 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.485009    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 16:43:16.485110 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.485052    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bwswv\""
Sep 17 16:43:16.524309 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.524267    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7fkp\" (UniqueName: \"kubernetes.io/projected/63f1b2a4-fb40-43bf-9875-97364d40351e-kube-api-access-s7fkp\") pod \"openshift-state-metrics-86c5bc79d6-pxb89\" (UID: \"63f1b2a4-fb40-43bf-9875-97364d40351e\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pxb89"
Sep 17 16:43:16.524309 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.524309    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/63f1b2a4-fb40-43bf-9875-97364d40351e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-pxb89\" (UID: \"63f1b2a4-fb40-43bf-9875-97364d40351e\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pxb89"
Sep 17 16:43:16.524567 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.524327    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/406c1ffc-6544-4566-aac6-afd21bbe6071-node-exporter-tls\") pod \"node-exporter-l5ppz\" (UID: \"406c1ffc-6544-4566-aac6-afd21bbe6071\") " pod="openshift-monitoring/node-exporter-l5ppz"
Sep 17 16:43:16.524567 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.524346    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/406c1ffc-6544-4566-aac6-afd21bbe6071-root\") pod \"node-exporter-l5ppz\" (UID: \"406c1ffc-6544-4566-aac6-afd21bbe6071\") " pod="openshift-monitoring/node-exporter-l5ppz"
Sep 17 16:43:16.524567 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.524363    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/406c1ffc-6544-4566-aac6-afd21bbe6071-node-exporter-textfile\") pod \"node-exporter-l5ppz\" (UID: \"406c1ffc-6544-4566-aac6-afd21bbe6071\") " pod="openshift-monitoring/node-exporter-l5ppz"
Sep 17 16:43:16.524567 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.524464    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/406c1ffc-6544-4566-aac6-afd21bbe6071-metrics-client-ca\") pod \"node-exporter-l5ppz\" (UID: \"406c1ffc-6544-4566-aac6-afd21bbe6071\") " pod="openshift-monitoring/node-exporter-l5ppz"
Sep 17 16:43:16.524567 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.524498    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63f1b2a4-fb40-43bf-9875-97364d40351e-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-pxb89\" (UID: \"63f1b2a4-fb40-43bf-9875-97364d40351e\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pxb89"
Sep 17 16:43:16.524567 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.524551    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/406c1ffc-6544-4566-aac6-afd21bbe6071-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l5ppz\" (UID: \"406c1ffc-6544-4566-aac6-afd21bbe6071\") " pod="openshift-monitoring/node-exporter-l5ppz"
Sep 17 16:43:16.524850 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.524581    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/63f1b2a4-fb40-43bf-9875-97364d40351e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-pxb89\" (UID: \"63f1b2a4-fb40-43bf-9875-97364d40351e\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pxb89"
Sep 17 16:43:16.524850 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.524626    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxrw5\" (UniqueName: \"kubernetes.io/projected/406c1ffc-6544-4566-aac6-afd21bbe6071-kube-api-access-bxrw5\") pod \"node-exporter-l5ppz\" (UID: \"406c1ffc-6544-4566-aac6-afd21bbe6071\") " pod="openshift-monitoring/node-exporter-l5ppz"
Sep 17 16:43:16.524850 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.524669    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/406c1ffc-6544-4566-aac6-afd21bbe6071-node-exporter-accelerators-collector-config\") pod \"node-exporter-l5ppz\" (UID: \"406c1ffc-6544-4566-aac6-afd21bbe6071\") " pod="openshift-monitoring/node-exporter-l5ppz"
Sep 17 16:43:16.524850 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.524722    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/406c1ffc-6544-4566-aac6-afd21bbe6071-sys\") pod \"node-exporter-l5ppz\" (UID: \"406c1ffc-6544-4566-aac6-afd21bbe6071\") " pod="openshift-monitoring/node-exporter-l5ppz"
Sep 17 16:43:16.524850 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.524756    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/406c1ffc-6544-4566-aac6-afd21bbe6071-node-exporter-wtmp\") pod \"node-exporter-l5ppz\" (UID: \"406c1ffc-6544-4566-aac6-afd21bbe6071\") " pod="openshift-monitoring/node-exporter-l5ppz"
Sep 17 16:43:16.625778 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.625733    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/406c1ffc-6544-4566-aac6-afd21bbe6071-node-exporter-accelerators-collector-config\") pod \"node-exporter-l5ppz\" (UID: \"406c1ffc-6544-4566-aac6-afd21bbe6071\") " pod="openshift-monitoring/node-exporter-l5ppz"
Sep 17 16:43:16.625778 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.625780    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/406c1ffc-6544-4566-aac6-afd21bbe6071-sys\") pod \"node-exporter-l5ppz\" (UID: \"406c1ffc-6544-4566-aac6-afd21bbe6071\") " pod="openshift-monitoring/node-exporter-l5ppz"
Sep 17 16:43:16.626020 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.625805    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/406c1ffc-6544-4566-aac6-afd21bbe6071-node-exporter-wtmp\") pod \"node-exporter-l5ppz\" (UID: \"406c1ffc-6544-4566-aac6-afd21bbe6071\") " pod="openshift-monitoring/node-exporter-l5ppz"
Sep 17 16:43:16.626020 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.625829    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7fkp\" (UniqueName: \"kubernetes.io/projected/63f1b2a4-fb40-43bf-9875-97364d40351e-kube-api-access-s7fkp\") pod \"openshift-state-metrics-86c5bc79d6-pxb89\" (UID: \"63f1b2a4-fb40-43bf-9875-97364d40351e\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pxb89"
Sep 17 16:43:16.626020 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.625846    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/63f1b2a4-fb40-43bf-9875-97364d40351e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-pxb89\" (UID: \"63f1b2a4-fb40-43bf-9875-97364d40351e\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pxb89"
Sep 17 16:43:16.626020 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.625862    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/406c1ffc-6544-4566-aac6-afd21bbe6071-node-exporter-tls\") pod \"node-exporter-l5ppz\" (UID: \"406c1ffc-6544-4566-aac6-afd21bbe6071\") " pod="openshift-monitoring/node-exporter-l5ppz"
Sep 17 16:43:16.626020 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.625870    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/406c1ffc-6544-4566-aac6-afd21bbe6071-sys\") pod \"node-exporter-l5ppz\" (UID: \"406c1ffc-6544-4566-aac6-afd21bbe6071\") " pod="openshift-monitoring/node-exporter-l5ppz"
Sep 17 16:43:16.626020 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.625881    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/406c1ffc-6544-4566-aac6-afd21bbe6071-root\") pod \"node-exporter-l5ppz\" (UID: \"406c1ffc-6544-4566-aac6-afd21bbe6071\") " pod="openshift-monitoring/node-exporter-l5ppz"
Sep 17 16:43:16.626020 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.625913    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/406c1ffc-6544-4566-aac6-afd21bbe6071-root\") pod \"node-exporter-l5ppz\" (UID: \"406c1ffc-6544-4566-aac6-afd21bbe6071\") " pod="openshift-monitoring/node-exporter-l5ppz"
Sep 17 16:43:16.626020 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.625933    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/406c1ffc-6544-4566-aac6-afd21bbe6071-node-exporter-textfile\") pod \"node-exporter-l5ppz\" (UID: \"406c1ffc-6544-4566-aac6-afd21bbe6071\") " pod="openshift-monitoring/node-exporter-l5ppz"
Sep 17 16:43:16.626020 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.625968    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/406c1ffc-6544-4566-aac6-afd21bbe6071-metrics-client-ca\") pod \"node-exporter-l5ppz\" (UID: \"406c1ffc-6544-4566-aac6-afd21bbe6071\") " pod="openshift-monitoring/node-exporter-l5ppz"
Sep 17 16:43:16.626020 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:43:16.625994    2566 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 17 16:43:16.626521 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.626022    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/406c1ffc-6544-4566-aac6-afd21bbe6071-node-exporter-wtmp\") pod \"node-exporter-l5ppz\" (UID: \"406c1ffc-6544-4566-aac6-afd21bbe6071\") " pod="openshift-monitoring/node-exporter-l5ppz"
Sep 17 16:43:16.626521 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:43:16.626062    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63f1b2a4-fb40-43bf-9875-97364d40351e-openshift-state-metrics-tls podName:63f1b2a4-fb40-43bf-9875-97364d40351e nodeName:}" failed. No retries permitted until 2026-09-17 16:43:17.126041787 +0000 UTC m=+83.020520904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/63f1b2a4-fb40-43bf-9875-97364d40351e-openshift-state-metrics-tls") pod "openshift-state-metrics-86c5bc79d6-pxb89" (UID: "63f1b2a4-fb40-43bf-9875-97364d40351e") : secret "openshift-state-metrics-tls" not found
Sep 17 16:43:16.626521 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.625996    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63f1b2a4-fb40-43bf-9875-97364d40351e-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-pxb89\" (UID: \"63f1b2a4-fb40-43bf-9875-97364d40351e\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pxb89"
Sep 17 16:43:16.626521 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.626114    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/406c1ffc-6544-4566-aac6-afd21bbe6071-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l5ppz\" (UID: \"406c1ffc-6544-4566-aac6-afd21bbe6071\") " pod="openshift-monitoring/node-exporter-l5ppz"
Sep 17 16:43:16.626521 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.626148    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/63f1b2a4-fb40-43bf-9875-97364d40351e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-pxb89\" (UID: \"63f1b2a4-fb40-43bf-9875-97364d40351e\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pxb89"
Sep 17 16:43:16.626521 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.626179    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxrw5\" (UniqueName: \"kubernetes.io/projected/406c1ffc-6544-4566-aac6-afd21bbe6071-kube-api-access-bxrw5\") pod \"node-exporter-l5ppz\" (UID: \"406c1ffc-6544-4566-aac6-afd21bbe6071\") " pod="openshift-monitoring/node-exporter-l5ppz"
Sep 17 16:43:16.626521 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.626306    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/406c1ffc-6544-4566-aac6-afd21bbe6071-node-exporter-textfile\") pod \"node-exporter-l5ppz\" (UID: \"406c1ffc-6544-4566-aac6-afd21bbe6071\") " pod="openshift-monitoring/node-exporter-l5ppz"
Sep 17 16:43:16.626521 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.626476    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/406c1ffc-6544-4566-aac6-afd21bbe6071-node-exporter-accelerators-collector-config\") pod \"node-exporter-l5ppz\" (UID: \"406c1ffc-6544-4566-aac6-afd21bbe6071\") " pod="openshift-monitoring/node-exporter-l5ppz"
Sep 17 16:43:16.626902 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.626746    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/406c1ffc-6544-4566-aac6-afd21bbe6071-metrics-client-ca\") pod \"node-exporter-l5ppz\" (UID: \"406c1ffc-6544-4566-aac6-afd21bbe6071\") " pod="openshift-monitoring/node-exporter-l5ppz"
Sep 17 16:43:16.626902 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.626783    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63f1b2a4-fb40-43bf-9875-97364d40351e-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-pxb89\" (UID: \"63f1b2a4-fb40-43bf-9875-97364d40351e\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pxb89"
Sep 17 16:43:16.628428 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.628400    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/406c1ffc-6544-4566-aac6-afd21bbe6071-node-exporter-tls\") pod \"node-exporter-l5ppz\" (UID: \"406c1ffc-6544-4566-aac6-afd21bbe6071\") " pod="openshift-monitoring/node-exporter-l5ppz"
Sep 17 16:43:16.628562 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.628499    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/406c1ffc-6544-4566-aac6-afd21bbe6071-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l5ppz\" (UID: \"406c1ffc-6544-4566-aac6-afd21bbe6071\") " pod="openshift-monitoring/node-exporter-l5ppz"
Sep 17 16:43:16.628715 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.628696    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/63f1b2a4-fb40-43bf-9875-97364d40351e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-pxb89\" (UID: \"63f1b2a4-fb40-43bf-9875-97364d40351e\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pxb89"
Sep 17 16:43:16.635757 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.635732    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7fkp\" (UniqueName: \"kubernetes.io/projected/63f1b2a4-fb40-43bf-9875-97364d40351e-kube-api-access-s7fkp\") pod \"openshift-state-metrics-86c5bc79d6-pxb89\" (UID: \"63f1b2a4-fb40-43bf-9875-97364d40351e\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pxb89"
Sep 17 16:43:16.635857 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.635760    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxrw5\" (UniqueName: \"kubernetes.io/projected/406c1ffc-6544-4566-aac6-afd21bbe6071-kube-api-access-bxrw5\") pod \"node-exporter-l5ppz\" (UID: \"406c1ffc-6544-4566-aac6-afd21bbe6071\") " pod="openshift-monitoring/node-exporter-l5ppz"
Sep 17 16:43:16.789852 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:16.789826    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l5ppz"
Sep 17 16:43:16.802647 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:43:16.802621    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod406c1ffc_6544_4566_aac6_afd21bbe6071.slice/crio-8d7a9641311e57f2439d2705d824b5b6521ca23f912b23a71c5b9146ce0eba79 WatchSource:0}: Error finding container 8d7a9641311e57f2439d2705d824b5b6521ca23f912b23a71c5b9146ce0eba79: Status 404 returned error can't find the container with id 8d7a9641311e57f2439d2705d824b5b6521ca23f912b23a71c5b9146ce0eba79
Sep 17 16:43:17.129267 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:17.129188    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/63f1b2a4-fb40-43bf-9875-97364d40351e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-pxb89\" (UID: \"63f1b2a4-fb40-43bf-9875-97364d40351e\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pxb89"
Sep 17 16:43:17.131624 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:17.131604    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/63f1b2a4-fb40-43bf-9875-97364d40351e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-pxb89\" (UID: \"63f1b2a4-fb40-43bf-9875-97364d40351e\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pxb89"
Sep 17 16:43:17.375863 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:17.375831    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pxb89"
Sep 17 16:43:17.657008 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:17.656824    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-pxb89"]
Sep 17 16:43:17.703771 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:17.703735    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l5ppz" event={"ID":"406c1ffc-6544-4566-aac6-afd21bbe6071","Type":"ContainerStarted","Data":"8d7a9641311e57f2439d2705d824b5b6521ca23f912b23a71c5b9146ce0eba79"}
Sep 17 16:43:17.776771 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:43:17.776739    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63f1b2a4_fb40_43bf_9875_97364d40351e.slice/crio-a9aab96c6f63dd09534128daa080cdd03e2443864d09a6cbbc4632783e3d7a1f WatchSource:0}: Error finding container a9aab96c6f63dd09534128daa080cdd03e2443864d09a6cbbc4632783e3d7a1f: Status 404 returned error can't find the container with id a9aab96c6f63dd09534128daa080cdd03e2443864d09a6cbbc4632783e3d7a1f
Sep 17 16:43:18.710138 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:18.710097    2566 generic.go:356] "Generic (PLEG): container finished" podID="406c1ffc-6544-4566-aac6-afd21bbe6071" containerID="f782bac878216d34bea8000ba1e615cddd065de63b33a32d3711da99980b36d9" exitCode=0
Sep 17 16:43:18.710593 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:18.710193    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l5ppz" event={"ID":"406c1ffc-6544-4566-aac6-afd21bbe6071","Type":"ContainerDied","Data":"f782bac878216d34bea8000ba1e615cddd065de63b33a32d3711da99980b36d9"}
Sep 17 16:43:18.712390 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:18.712365    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pxb89" event={"ID":"63f1b2a4-fb40-43bf-9875-97364d40351e","Type":"ContainerStarted","Data":"8508d3059a864082312402fcc4b46c29c5327c851aa2dc08ab33815acb0368a3"}
Sep 17 16:43:18.712511 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:18.712417    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pxb89" event={"ID":"63f1b2a4-fb40-43bf-9875-97364d40351e","Type":"ContainerStarted","Data":"e0262942114095bd057641dd5ef88dac9f78673642cdfcc1b039d9da23744bdf"}
Sep 17 16:43:18.712511 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:18.712470    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pxb89" event={"ID":"63f1b2a4-fb40-43bf-9875-97364d40351e","Type":"ContainerStarted","Data":"a9aab96c6f63dd09534128daa080cdd03e2443864d09a6cbbc4632783e3d7a1f"}
Sep 17 16:43:19.718481 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:19.718430    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l5ppz" event={"ID":"406c1ffc-6544-4566-aac6-afd21bbe6071","Type":"ContainerStarted","Data":"c36f8a7ada55f32ac58256b363af0f48556ddfc315553d25011976f0a7d4b1a1"}
Sep 17 16:43:19.718911 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:19.718506    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l5ppz" event={"ID":"406c1ffc-6544-4566-aac6-afd21bbe6071","Type":"ContainerStarted","Data":"5d357b89c1284369cd5fac5ffe5ebf9dce8164fb979c94270ba07d9738b0b8a9"}
Sep 17 16:43:19.720630 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:19.720606    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pxb89" event={"ID":"63f1b2a4-fb40-43bf-9875-97364d40351e","Type":"ContainerStarted","Data":"3423fea8f9eab459d87fd7f7c660dc8381c21980d9b7cc7eb2c983de604fea2f"}
Sep 17 16:43:19.739722 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:19.739685    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-l5ppz" podStartSLOduration=2.649908353 podStartE2EDuration="3.739671495s" totalImagesPullingTime="1.01909815s" totalInitContainerRuntime="70.664992ms" isStatefulPod=true podCreationTimestamp="2026-09-17 16:43:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:19.738159216 +0000 UTC m=+85.632638356" watchObservedRunningTime="2026-09-17 16:43:19.739671495 +0000 UTC m=+85.634150652"
Sep 17 16:43:19.757408 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:19.757369    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pxb89" podStartSLOduration=2.787566804 podStartE2EDuration="3.757353235s" totalImagesPullingTime="969.786431ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:43:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:19.756557239 +0000 UTC m=+85.651036379" watchObservedRunningTime="2026-09-17 16:43:19.757353235 +0000 UTC m=+85.651832378"
Sep 17 16:43:25.660720 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:25.660693    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-75b54c8748-tqz7p"
Sep 17 16:43:26.286917 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:26.286881    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-75b54c8748-tqz7p"]
Sep 17 16:43:41.684758 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:41.684726    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-cd6rw"
Sep 17 16:43:51.306411 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:51.306366    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-75b54c8748-tqz7p" podUID="fee0259b-0d84-4ecc-96c0-16ab13596d57" containerName="registry" containerID="cri-o://3d34ce4bfd3090b80696dd9f1e0e77abac9833cc8a36807c6dc550422f54f6a1" gracePeriod=30
Sep 17 16:43:51.552928 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:51.552906    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75b54c8748-tqz7p"
Sep 17 16:43:51.674739 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:51.674633    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fee0259b-0d84-4ecc-96c0-16ab13596d57-registry-tls\") pod \"fee0259b-0d84-4ecc-96c0-16ab13596d57\" (UID: \"fee0259b-0d84-4ecc-96c0-16ab13596d57\") "
Sep 17 16:43:51.674739 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:51.674683    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fee0259b-0d84-4ecc-96c0-16ab13596d57-trusted-ca\") pod \"fee0259b-0d84-4ecc-96c0-16ab13596d57\" (UID: \"fee0259b-0d84-4ecc-96c0-16ab13596d57\") "
Sep 17 16:43:51.674739 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:51.674709    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fee0259b-0d84-4ecc-96c0-16ab13596d57-installation-pull-secrets\") pod \"fee0259b-0d84-4ecc-96c0-16ab13596d57\" (UID: \"fee0259b-0d84-4ecc-96c0-16ab13596d57\") "
Sep 17 16:43:51.674739 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:51.674726    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fee0259b-0d84-4ecc-96c0-16ab13596d57-image-registry-private-configuration\") pod \"fee0259b-0d84-4ecc-96c0-16ab13596d57\" (UID: \"fee0259b-0d84-4ecc-96c0-16ab13596d57\") "
Sep 17 16:43:51.675064 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:51.674924    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fee0259b-0d84-4ecc-96c0-16ab13596d57-registry-certificates\") pod \"fee0259b-0d84-4ecc-96c0-16ab13596d57\" (UID: \"fee0259b-0d84-4ecc-96c0-16ab13596d57\") "
Sep 17 16:43:51.675064 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:51.674997    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fee0259b-0d84-4ecc-96c0-16ab13596d57-ca-trust-extracted\") pod \"fee0259b-0d84-4ecc-96c0-16ab13596d57\" (UID: \"fee0259b-0d84-4ecc-96c0-16ab13596d57\") "
Sep 17 16:43:51.675064 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:51.675033    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fee0259b-0d84-4ecc-96c0-16ab13596d57-bound-sa-token\") pod \"fee0259b-0d84-4ecc-96c0-16ab13596d57\" (UID: \"fee0259b-0d84-4ecc-96c0-16ab13596d57\") "
Sep 17 16:43:51.675217 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:51.675065    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rh8rs\" (UniqueName: \"kubernetes.io/projected/fee0259b-0d84-4ecc-96c0-16ab13596d57-kube-api-access-rh8rs\") pod \"fee0259b-0d84-4ecc-96c0-16ab13596d57\" (UID: \"fee0259b-0d84-4ecc-96c0-16ab13596d57\") "
Sep 17 16:43:51.675217 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:51.675195    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fee0259b-0d84-4ecc-96c0-16ab13596d57-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "fee0259b-0d84-4ecc-96c0-16ab13596d57" (UID: "fee0259b-0d84-4ecc-96c0-16ab13596d57"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:43:51.675324 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:51.675314    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fee0259b-0d84-4ecc-96c0-16ab13596d57-trusted-ca\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:43:51.675480 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:51.675452    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fee0259b-0d84-4ecc-96c0-16ab13596d57-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "fee0259b-0d84-4ecc-96c0-16ab13596d57" (UID: "fee0259b-0d84-4ecc-96c0-16ab13596d57"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:43:51.677592 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:51.677478    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fee0259b-0d84-4ecc-96c0-16ab13596d57-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "fee0259b-0d84-4ecc-96c0-16ab13596d57" (UID: "fee0259b-0d84-4ecc-96c0-16ab13596d57"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:43:51.677706 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:51.677631    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fee0259b-0d84-4ecc-96c0-16ab13596d57-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "fee0259b-0d84-4ecc-96c0-16ab13596d57" (UID: "fee0259b-0d84-4ecc-96c0-16ab13596d57"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:43:51.677706 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:51.677677    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fee0259b-0d84-4ecc-96c0-16ab13596d57-kube-api-access-rh8rs" (OuterVolumeSpecName: "kube-api-access-rh8rs") pod "fee0259b-0d84-4ecc-96c0-16ab13596d57" (UID: "fee0259b-0d84-4ecc-96c0-16ab13596d57"). InnerVolumeSpecName "kube-api-access-rh8rs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:43:51.677802 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:51.677735    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fee0259b-0d84-4ecc-96c0-16ab13596d57-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "fee0259b-0d84-4ecc-96c0-16ab13596d57" (UID: "fee0259b-0d84-4ecc-96c0-16ab13596d57"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:43:51.677837 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:51.677813    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fee0259b-0d84-4ecc-96c0-16ab13596d57-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "fee0259b-0d84-4ecc-96c0-16ab13596d57" (UID: "fee0259b-0d84-4ecc-96c0-16ab13596d57"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:43:51.684342 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:51.684317    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fee0259b-0d84-4ecc-96c0-16ab13596d57-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "fee0259b-0d84-4ecc-96c0-16ab13596d57" (UID: "fee0259b-0d84-4ecc-96c0-16ab13596d57"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:43:51.776187 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:51.776149    2566 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fee0259b-0d84-4ecc-96c0-16ab13596d57-registry-certificates\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:43:51.776187 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:51.776181    2566 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fee0259b-0d84-4ecc-96c0-16ab13596d57-ca-trust-extracted\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:43:51.776187 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:51.776193    2566 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fee0259b-0d84-4ecc-96c0-16ab13596d57-bound-sa-token\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:43:51.776391 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:51.776202    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh8rs\" (UniqueName: \"kubernetes.io/projected/fee0259b-0d84-4ecc-96c0-16ab13596d57-kube-api-access-rh8rs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:43:51.776391 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:51.776210    2566 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fee0259b-0d84-4ecc-96c0-16ab13596d57-registry-tls\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:43:51.776391 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:51.776219    2566 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fee0259b-0d84-4ecc-96c0-16ab13596d57-installation-pull-secrets\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:43:51.776391 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:51.776229    2566 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fee0259b-0d84-4ecc-96c0-16ab13596d57-image-registry-private-configuration\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:43:51.832321 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:51.832292    2566 generic.go:356] "Generic (PLEG): container finished" podID="fee0259b-0d84-4ecc-96c0-16ab13596d57" containerID="3d34ce4bfd3090b80696dd9f1e0e77abac9833cc8a36807c6dc550422f54f6a1" exitCode=0
Sep 17 16:43:51.832468 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:51.832337    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75b54c8748-tqz7p"
Sep 17 16:43:51.832468 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:51.832354    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75b54c8748-tqz7p" event={"ID":"fee0259b-0d84-4ecc-96c0-16ab13596d57","Type":"ContainerDied","Data":"3d34ce4bfd3090b80696dd9f1e0e77abac9833cc8a36807c6dc550422f54f6a1"}
Sep 17 16:43:51.832468 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:51.832420    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75b54c8748-tqz7p" event={"ID":"fee0259b-0d84-4ecc-96c0-16ab13596d57","Type":"ContainerDied","Data":"c7536a2dc8f8880785220cbeb07faa8ee3a40169e0d0fa79571682a8840dccf2"}
Sep 17 16:43:51.832468 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:51.832462    2566 scope.go:117] "RemoveContainer" containerID="3d34ce4bfd3090b80696dd9f1e0e77abac9833cc8a36807c6dc550422f54f6a1"
Sep 17 16:43:51.841630 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:51.841614    2566 scope.go:117] "RemoveContainer" containerID="3d34ce4bfd3090b80696dd9f1e0e77abac9833cc8a36807c6dc550422f54f6a1"
Sep 17 16:43:51.841882 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:43:51.841854    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d34ce4bfd3090b80696dd9f1e0e77abac9833cc8a36807c6dc550422f54f6a1\": container with ID starting with 3d34ce4bfd3090b80696dd9f1e0e77abac9833cc8a36807c6dc550422f54f6a1 not found: ID does not exist" containerID="3d34ce4bfd3090b80696dd9f1e0e77abac9833cc8a36807c6dc550422f54f6a1"
Sep 17 16:43:51.841974 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:51.841887    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d34ce4bfd3090b80696dd9f1e0e77abac9833cc8a36807c6dc550422f54f6a1"} err="failed to get container status \"3d34ce4bfd3090b80696dd9f1e0e77abac9833cc8a36807c6dc550422f54f6a1\": rpc error: code = NotFound desc = could not find container \"3d34ce4bfd3090b80696dd9f1e0e77abac9833cc8a36807c6dc550422f54f6a1\": container with ID starting with 3d34ce4bfd3090b80696dd9f1e0e77abac9833cc8a36807c6dc550422f54f6a1 not found: ID does not exist"
Sep 17 16:43:51.856857 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:51.856827    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-75b54c8748-tqz7p"]
Sep 17 16:43:51.859305 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:51.859285    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-75b54c8748-tqz7p"]
Sep 17 16:43:52.345394 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:43:52.345354    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fee0259b-0d84-4ecc-96c0-16ab13596d57" path="/var/lib/kubelet/pods/fee0259b-0d84-4ecc-96c0-16ab13596d57/volumes"
Sep 17 16:46:56.244525 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:46:56.244496    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gblhr_91477834-6648-4e75-8530-e01bdee5e57e/ovn-acl-logging/0.log"
Sep 17 16:46:56.249900 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:46:56.249876    2566 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 16:46:56.250782 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:46:56.250765    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gblhr_91477834-6648-4e75-8530-e01bdee5e57e/ovn-acl-logging/0.log"
Sep 17 16:49:43.100611 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:49:43.100578    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1acdgfr"]
Sep 17 16:49:43.101062 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:49:43.100797    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fee0259b-0d84-4ecc-96c0-16ab13596d57" containerName="registry"
Sep 17 16:49:43.101062 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:49:43.100807    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fee0259b-0d84-4ecc-96c0-16ab13596d57" containerName="registry"
Sep 17 16:49:43.101062 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:49:43.100857    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fee0259b-0d84-4ecc-96c0-16ab13596d57" containerName="registry"
Sep 17 16:49:43.103711 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:49:43.103694    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1acdgfr"
Sep 17 16:49:43.105780 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:49:43.105761    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 16:49:43.106052 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:49:43.106035    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 16:49:43.106592 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:49:43.106572    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-gfmjh\""
Sep 17 16:49:43.118845 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:49:43.118825    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1acdgfr"]
Sep 17 16:49:43.250619 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:49:43.250578    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff1d23b7-d3d3-4fe8-a377-afc564e4caca-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1acdgfr\" (UID: \"ff1d23b7-d3d3-4fe8-a377-afc564e4caca\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1acdgfr"
Sep 17 16:49:43.250619 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:49:43.250628    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqgvb\" (UniqueName: \"kubernetes.io/projected/ff1d23b7-d3d3-4fe8-a377-afc564e4caca-kube-api-access-kqgvb\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1acdgfr\" (UID: \"ff1d23b7-d3d3-4fe8-a377-afc564e4caca\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1acdgfr"
Sep 17 16:49:43.250816 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:49:43.250648    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff1d23b7-d3d3-4fe8-a377-afc564e4caca-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1acdgfr\" (UID: \"ff1d23b7-d3d3-4fe8-a377-afc564e4caca\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1acdgfr"
Sep 17 16:49:43.351806 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:49:43.351744    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff1d23b7-d3d3-4fe8-a377-afc564e4caca-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1acdgfr\" (UID: \"ff1d23b7-d3d3-4fe8-a377-afc564e4caca\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1acdgfr"
Sep 17 16:49:43.351806 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:49:43.351777    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqgvb\" (UniqueName: \"kubernetes.io/projected/ff1d23b7-d3d3-4fe8-a377-afc564e4caca-kube-api-access-kqgvb\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1acdgfr\" (UID: \"ff1d23b7-d3d3-4fe8-a377-afc564e4caca\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1acdgfr"
Sep 17 16:49:43.351806 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:49:43.351797    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff1d23b7-d3d3-4fe8-a377-afc564e4caca-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1acdgfr\" (UID: \"ff1d23b7-d3d3-4fe8-a377-afc564e4caca\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1acdgfr"
Sep 17 16:49:43.352090 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:49:43.352077    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff1d23b7-d3d3-4fe8-a377-afc564e4caca-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1acdgfr\" (UID: \"ff1d23b7-d3d3-4fe8-a377-afc564e4caca\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1acdgfr"
Sep 17 16:49:43.352201 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:49:43.352177    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff1d23b7-d3d3-4fe8-a377-afc564e4caca-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1acdgfr\" (UID: \"ff1d23b7-d3d3-4fe8-a377-afc564e4caca\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1acdgfr"
Sep 17 16:49:43.364926 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:49:43.364900    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqgvb\" (UniqueName: \"kubernetes.io/projected/ff1d23b7-d3d3-4fe8-a377-afc564e4caca-kube-api-access-kqgvb\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1acdgfr\" (UID: \"ff1d23b7-d3d3-4fe8-a377-afc564e4caca\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1acdgfr"
Sep 17 16:49:43.412268 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:49:43.412242    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1acdgfr"
Sep 17 16:49:43.689833 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:49:43.689760    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1acdgfr"]
Sep 17 16:49:43.693359 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:49:43.693332    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff1d23b7_d3d3_4fe8_a377_afc564e4caca.slice/crio-34cd36f9e0bc03c68b34de5d5e9e4fb8e93af0b03691787bbb3a5b5915fd431c WatchSource:0}: Error finding container 34cd36f9e0bc03c68b34de5d5e9e4fb8e93af0b03691787bbb3a5b5915fd431c: Status 404 returned error can't find the container with id 34cd36f9e0bc03c68b34de5d5e9e4fb8e93af0b03691787bbb3a5b5915fd431c
Sep 17 16:49:43.695729 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:49:43.695714    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:49:43.961012 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:49:43.960931    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1acdgfr" event={"ID":"ff1d23b7-d3d3-4fe8-a377-afc564e4caca","Type":"ContainerStarted","Data":"34cd36f9e0bc03c68b34de5d5e9e4fb8e93af0b03691787bbb3a5b5915fd431c"}
Sep 17 16:49:49.982357 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:49:49.982255    2566 generic.go:356] "Generic (PLEG): container finished" podID="ff1d23b7-d3d3-4fe8-a377-afc564e4caca" containerID="3a11cb6f0d84c18291bc6aedac7e423f68efda793d36932c771f1ea290e8e8fd" exitCode=0
Sep 17 16:49:49.982357 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:49:49.982335    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1acdgfr" event={"ID":"ff1d23b7-d3d3-4fe8-a377-afc564e4caca","Type":"ContainerDied","Data":"3a11cb6f0d84c18291bc6aedac7e423f68efda793d36932c771f1ea290e8e8fd"}
Sep 17 16:49:52.994895 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:49:52.994862    2566 generic.go:356] "Generic (PLEG): container finished" podID="ff1d23b7-d3d3-4fe8-a377-afc564e4caca" containerID="6743e3a606c7526cf8d03482319f4727415f9205ed3eb4f647db7e180566596f" exitCode=0
Sep 17 16:49:52.995282 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:49:52.994947    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1acdgfr" event={"ID":"ff1d23b7-d3d3-4fe8-a377-afc564e4caca","Type":"ContainerDied","Data":"6743e3a606c7526cf8d03482319f4727415f9205ed3eb4f647db7e180566596f"}
Sep 17 16:50:00.021727 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:00.021701    2566 generic.go:356] "Generic (PLEG): container finished" podID="ff1d23b7-d3d3-4fe8-a377-afc564e4caca" containerID="202d878987aa511f6243add7672beeaea1bd800d638f8cf23b47c03365d2b366" exitCode=0
Sep 17 16:50:00.022020 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:00.021782    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1acdgfr" event={"ID":"ff1d23b7-d3d3-4fe8-a377-afc564e4caca","Type":"ContainerDied","Data":"202d878987aa511f6243add7672beeaea1bd800d638f8cf23b47c03365d2b366"}
Sep 17 16:50:01.149225 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:01.149203    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1acdgfr"
Sep 17 16:50:01.289429 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:01.289404    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff1d23b7-d3d3-4fe8-a377-afc564e4caca-util\") pod \"ff1d23b7-d3d3-4fe8-a377-afc564e4caca\" (UID: \"ff1d23b7-d3d3-4fe8-a377-afc564e4caca\") "
Sep 17 16:50:01.289592 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:01.289465    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqgvb\" (UniqueName: \"kubernetes.io/projected/ff1d23b7-d3d3-4fe8-a377-afc564e4caca-kube-api-access-kqgvb\") pod \"ff1d23b7-d3d3-4fe8-a377-afc564e4caca\" (UID: \"ff1d23b7-d3d3-4fe8-a377-afc564e4caca\") "
Sep 17 16:50:01.289592 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:01.289515    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff1d23b7-d3d3-4fe8-a377-afc564e4caca-bundle\") pod \"ff1d23b7-d3d3-4fe8-a377-afc564e4caca\" (UID: \"ff1d23b7-d3d3-4fe8-a377-afc564e4caca\") "
Sep 17 16:50:01.290124 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:01.290095    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff1d23b7-d3d3-4fe8-a377-afc564e4caca-bundle" (OuterVolumeSpecName: "bundle") pod "ff1d23b7-d3d3-4fe8-a377-afc564e4caca" (UID: "ff1d23b7-d3d3-4fe8-a377-afc564e4caca"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:50:01.291811 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:01.291789    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff1d23b7-d3d3-4fe8-a377-afc564e4caca-kube-api-access-kqgvb" (OuterVolumeSpecName: "kube-api-access-kqgvb") pod "ff1d23b7-d3d3-4fe8-a377-afc564e4caca" (UID: "ff1d23b7-d3d3-4fe8-a377-afc564e4caca"). InnerVolumeSpecName "kube-api-access-kqgvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:50:01.294290 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:01.294271    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff1d23b7-d3d3-4fe8-a377-afc564e4caca-util" (OuterVolumeSpecName: "util") pod "ff1d23b7-d3d3-4fe8-a377-afc564e4caca" (UID: "ff1d23b7-d3d3-4fe8-a377-afc564e4caca"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:50:01.390592 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:01.390562    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff1d23b7-d3d3-4fe8-a377-afc564e4caca-util\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:50:01.390592 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:01.390585    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqgvb\" (UniqueName: \"kubernetes.io/projected/ff1d23b7-d3d3-4fe8-a377-afc564e4caca-kube-api-access-kqgvb\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:50:01.390592 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:01.390595    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff1d23b7-d3d3-4fe8-a377-afc564e4caca-bundle\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:50:02.030107 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:02.030076    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1acdgfr"
Sep 17 16:50:02.030289 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:02.030071    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1acdgfr" event={"ID":"ff1d23b7-d3d3-4fe8-a377-afc564e4caca","Type":"ContainerDied","Data":"34cd36f9e0bc03c68b34de5d5e9e4fb8e93af0b03691787bbb3a5b5915fd431c"}
Sep 17 16:50:02.030289 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:02.030233    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34cd36f9e0bc03c68b34de5d5e9e4fb8e93af0b03691787bbb3a5b5915fd431c"
Sep 17 16:50:06.499509 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:06.499477    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jxw47"]
Sep 17 16:50:06.499868 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:06.499701    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff1d23b7-d3d3-4fe8-a377-afc564e4caca" containerName="pull"
Sep 17 16:50:06.499868 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:06.499710    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff1d23b7-d3d3-4fe8-a377-afc564e4caca" containerName="pull"
Sep 17 16:50:06.499868 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:06.499717    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff1d23b7-d3d3-4fe8-a377-afc564e4caca" containerName="extract"
Sep 17 16:50:06.499868 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:06.499723    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff1d23b7-d3d3-4fe8-a377-afc564e4caca" containerName="extract"
Sep 17 16:50:06.499868 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:06.499737    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff1d23b7-d3d3-4fe8-a377-afc564e4caca" containerName="util"
Sep 17 16:50:06.499868 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:06.499743    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff1d23b7-d3d3-4fe8-a377-afc564e4caca" containerName="util"
Sep 17 16:50:06.499868 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:06.499779    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff1d23b7-d3d3-4fe8-a377-afc564e4caca" containerName="extract"
Sep 17 16:50:06.502319 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:06.502304    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jxw47"
Sep 17 16:50:06.505959 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:06.505938    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 16:50:06.507173 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:06.507156    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 17 16:50:06.508459 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:06.508281    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 16:50:06.508459 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:06.508293    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-hq8jq\""
Sep 17 16:50:06.529376 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:06.529356    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgd5m\" (UniqueName: \"kubernetes.io/projected/69f90f49-9de5-49cf-a889-34969aecfa3f-kube-api-access-hgd5m\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-jxw47\" (UID: \"69f90f49-9de5-49cf-a889-34969aecfa3f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jxw47"
Sep 17 16:50:06.529481 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:06.529388    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/69f90f49-9de5-49cf-a889-34969aecfa3f-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-jxw47\" (UID: \"69f90f49-9de5-49cf-a889-34969aecfa3f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jxw47"
Sep 17 16:50:06.547792 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:06.547766    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jxw47"]
Sep 17 16:50:06.630256 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:06.630229    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgd5m\" (UniqueName: \"kubernetes.io/projected/69f90f49-9de5-49cf-a889-34969aecfa3f-kube-api-access-hgd5m\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-jxw47\" (UID: \"69f90f49-9de5-49cf-a889-34969aecfa3f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jxw47"
Sep 17 16:50:06.630372 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:06.630266    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/69f90f49-9de5-49cf-a889-34969aecfa3f-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-jxw47\" (UID: \"69f90f49-9de5-49cf-a889-34969aecfa3f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jxw47"
Sep 17 16:50:06.632629 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:06.632606    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/69f90f49-9de5-49cf-a889-34969aecfa3f-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-jxw47\" (UID: \"69f90f49-9de5-49cf-a889-34969aecfa3f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jxw47"
Sep 17 16:50:06.645751 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:06.645729    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgd5m\" (UniqueName: \"kubernetes.io/projected/69f90f49-9de5-49cf-a889-34969aecfa3f-kube-api-access-hgd5m\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-jxw47\" (UID: \"69f90f49-9de5-49cf-a889-34969aecfa3f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jxw47"
Sep 17 16:50:06.811322 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:06.811286    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jxw47"
Sep 17 16:50:07.098169 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:07.098103    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jxw47"]
Sep 17 16:50:07.101450 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:50:07.101405    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69f90f49_9de5_49cf_a889_34969aecfa3f.slice/crio-b73af0980eaa3f5128b8c036a5300bdabe812a49e03db90aea26503fcfcc74a5 WatchSource:0}: Error finding container b73af0980eaa3f5128b8c036a5300bdabe812a49e03db90aea26503fcfcc74a5: Status 404 returned error can't find the container with id b73af0980eaa3f5128b8c036a5300bdabe812a49e03db90aea26503fcfcc74a5
Sep 17 16:50:08.055194 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:08.055156    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jxw47" event={"ID":"69f90f49-9de5-49cf-a889-34969aecfa3f","Type":"ContainerStarted","Data":"b73af0980eaa3f5128b8c036a5300bdabe812a49e03db90aea26503fcfcc74a5"}
Sep 17 16:50:12.072168 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:12.072134    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jxw47" event={"ID":"69f90f49-9de5-49cf-a889-34969aecfa3f","Type":"ContainerStarted","Data":"a43963e0f7fa89003c273d5403806fc2c7ae807505699e443be704da67ef1c45"}
Sep 17 16:50:12.072562 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:12.072262    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jxw47"
Sep 17 16:50:12.149153 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:12.149103    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jxw47" podStartSLOduration=1.7297049759999998 podStartE2EDuration="6.149089751s" totalImagesPullingTime="4.419384775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:50:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:50:12.146773625 +0000 UTC m=+498.041252764" watchObservedRunningTime="2026-09-17 16:50:12.149089751 +0000 UTC m=+498.043568893"
Sep 17 16:50:12.580113 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:12.580082    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-zrjf9"]
Sep 17 16:50:12.583121 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:12.583107    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-zrjf9"
Sep 17 16:50:12.585525 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:12.585498    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 16:50:12.585525 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:12.585513    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 17 16:50:12.585681 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:12.585635    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-nh2lq\""
Sep 17 16:50:12.602620 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:12.602596    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-zrjf9"]
Sep 17 16:50:12.671011 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:12.670977    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/79b0e787-c6b7-4d04-b5f9-9965bfeb24aa-cabundle0\") pod \"keda-operator-579c8958f7-zrjf9\" (UID: \"79b0e787-c6b7-4d04-b5f9-9965bfeb24aa\") " pod="openshift-keda/keda-operator-579c8958f7-zrjf9"
Sep 17 16:50:12.671152 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:12.671026    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxpb4\" (UniqueName: \"kubernetes.io/projected/79b0e787-c6b7-4d04-b5f9-9965bfeb24aa-kube-api-access-dxpb4\") pod \"keda-operator-579c8958f7-zrjf9\" (UID: \"79b0e787-c6b7-4d04-b5f9-9965bfeb24aa\") " pod="openshift-keda/keda-operator-579c8958f7-zrjf9"
Sep 17 16:50:12.671152 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:12.671060    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/79b0e787-c6b7-4d04-b5f9-9965bfeb24aa-certificates\") pod \"keda-operator-579c8958f7-zrjf9\" (UID: \"79b0e787-c6b7-4d04-b5f9-9965bfeb24aa\") " pod="openshift-keda/keda-operator-579c8958f7-zrjf9"
Sep 17 16:50:12.771742 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:12.771712    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/79b0e787-c6b7-4d04-b5f9-9965bfeb24aa-certificates\") pod \"keda-operator-579c8958f7-zrjf9\" (UID: \"79b0e787-c6b7-4d04-b5f9-9965bfeb24aa\") " pod="openshift-keda/keda-operator-579c8958f7-zrjf9"
Sep 17 16:50:12.771742 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:12.771740    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/79b0e787-c6b7-4d04-b5f9-9965bfeb24aa-cabundle0\") pod \"keda-operator-579c8958f7-zrjf9\" (UID: \"79b0e787-c6b7-4d04-b5f9-9965bfeb24aa\") " pod="openshift-keda/keda-operator-579c8958f7-zrjf9"
Sep 17 16:50:12.771951 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:12.771774    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxpb4\" (UniqueName: \"kubernetes.io/projected/79b0e787-c6b7-4d04-b5f9-9965bfeb24aa-kube-api-access-dxpb4\") pod \"keda-operator-579c8958f7-zrjf9\" (UID: \"79b0e787-c6b7-4d04-b5f9-9965bfeb24aa\") " pod="openshift-keda/keda-operator-579c8958f7-zrjf9"
Sep 17 16:50:12.771951 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:50:12.771874    2566 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Sep 17 16:50:12.771951 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:50:12.771894    2566 secret.go:281] references non-existent secret key: ca.crt
Sep 17 16:50:12.771951 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:50:12.771905    2566 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 17 16:50:12.771951 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:50:12.771920    2566 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-zrjf9: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 17 16:50:12.772179 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:50:12.771982    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79b0e787-c6b7-4d04-b5f9-9965bfeb24aa-certificates podName:79b0e787-c6b7-4d04-b5f9-9965bfeb24aa nodeName:}" failed. No retries permitted until 2026-09-17 16:50:13.271963189 +0000 UTC m=+499.166442326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/79b0e787-c6b7-4d04-b5f9-9965bfeb24aa-certificates") pod "keda-operator-579c8958f7-zrjf9" (UID: "79b0e787-c6b7-4d04-b5f9-9965bfeb24aa") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 17 16:50:12.772397 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:12.772379    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/79b0e787-c6b7-4d04-b5f9-9965bfeb24aa-cabundle0\") pod \"keda-operator-579c8958f7-zrjf9\" (UID: \"79b0e787-c6b7-4d04-b5f9-9965bfeb24aa\") " pod="openshift-keda/keda-operator-579c8958f7-zrjf9"
Sep 17 16:50:12.788731 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:12.788709    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxpb4\" (UniqueName: \"kubernetes.io/projected/79b0e787-c6b7-4d04-b5f9-9965bfeb24aa-kube-api-access-dxpb4\") pod \"keda-operator-579c8958f7-zrjf9\" (UID: \"79b0e787-c6b7-4d04-b5f9-9965bfeb24aa\") " pod="openshift-keda/keda-operator-579c8958f7-zrjf9"
Sep 17 16:50:13.274513 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:13.274478    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/79b0e787-c6b7-4d04-b5f9-9965bfeb24aa-certificates\") pod \"keda-operator-579c8958f7-zrjf9\" (UID: \"79b0e787-c6b7-4d04-b5f9-9965bfeb24aa\") " pod="openshift-keda/keda-operator-579c8958f7-zrjf9"
Sep 17 16:50:13.276946 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:13.276927    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/79b0e787-c6b7-4d04-b5f9-9965bfeb24aa-certificates\") pod \"keda-operator-579c8958f7-zrjf9\" (UID: \"79b0e787-c6b7-4d04-b5f9-9965bfeb24aa\") " pod="openshift-keda/keda-operator-579c8958f7-zrjf9"
Sep 17 16:50:13.492664 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:13.492629    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-zrjf9"
Sep 17 16:50:13.595877 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:13.595839    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-ctg25"]
Sep 17 16:50:13.599061 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:13.599041    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-ctg25"
Sep 17 16:50:13.607536 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:13.607521    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 17 16:50:13.676385 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:13.676356    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9b58\" (UniqueName: \"kubernetes.io/projected/5f10b39c-95b3-48d9-bc4e-8eee69860358-kube-api-access-f9b58\") pod \"keda-admission-6c4d864b86-ctg25\" (UID: \"5f10b39c-95b3-48d9-bc4e-8eee69860358\") " pod="openshift-keda/keda-admission-6c4d864b86-ctg25"
Sep 17 16:50:13.676516 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:13.676450    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5f10b39c-95b3-48d9-bc4e-8eee69860358-certificates\") pod \"keda-admission-6c4d864b86-ctg25\" (UID: \"5f10b39c-95b3-48d9-bc4e-8eee69860358\") " pod="openshift-keda/keda-admission-6c4d864b86-ctg25"
Sep 17 16:50:13.715628 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:13.715598    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-ctg25"]
Sep 17 16:50:13.774202 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:13.774179    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-zrjf9"]
Sep 17 16:50:13.776832 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:50:13.776803    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79b0e787_c6b7_4d04_b5f9_9965bfeb24aa.slice/crio-051cd5ec3fb08fb44925ae7b9c367ff49dce206f5cc3262c491293c2bd51d12f WatchSource:0}: Error finding container 051cd5ec3fb08fb44925ae7b9c367ff49dce206f5cc3262c491293c2bd51d12f: Status 404 returned error can't find the container with id 051cd5ec3fb08fb44925ae7b9c367ff49dce206f5cc3262c491293c2bd51d12f
Sep 17 16:50:13.776832 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:13.776831    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5f10b39c-95b3-48d9-bc4e-8eee69860358-certificates\") pod \"keda-admission-6c4d864b86-ctg25\" (UID: \"5f10b39c-95b3-48d9-bc4e-8eee69860358\") " pod="openshift-keda/keda-admission-6c4d864b86-ctg25"
Sep 17 16:50:13.777001 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:13.776921    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9b58\" (UniqueName: \"kubernetes.io/projected/5f10b39c-95b3-48d9-bc4e-8eee69860358-kube-api-access-f9b58\") pod \"keda-admission-6c4d864b86-ctg25\" (UID: \"5f10b39c-95b3-48d9-bc4e-8eee69860358\") " pod="openshift-keda/keda-admission-6c4d864b86-ctg25"
Sep 17 16:50:13.780091 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:13.780071    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5f10b39c-95b3-48d9-bc4e-8eee69860358-certificates\") pod \"keda-admission-6c4d864b86-ctg25\" (UID: \"5f10b39c-95b3-48d9-bc4e-8eee69860358\") " pod="openshift-keda/keda-admission-6c4d864b86-ctg25"
Sep 17 16:50:13.799755 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:13.799696    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9b58\" (UniqueName: \"kubernetes.io/projected/5f10b39c-95b3-48d9-bc4e-8eee69860358-kube-api-access-f9b58\") pod \"keda-admission-6c4d864b86-ctg25\" (UID: \"5f10b39c-95b3-48d9-bc4e-8eee69860358\") " pod="openshift-keda/keda-admission-6c4d864b86-ctg25"
Sep 17 16:50:13.908276 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:13.908242    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-ctg25"
Sep 17 16:50:14.079638 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:14.079561    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-zrjf9" event={"ID":"79b0e787-c6b7-4d04-b5f9-9965bfeb24aa","Type":"ContainerStarted","Data":"051cd5ec3fb08fb44925ae7b9c367ff49dce206f5cc3262c491293c2bd51d12f"}
Sep 17 16:50:14.219971 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:14.219937    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-ctg25"]
Sep 17 16:50:14.225469 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:50:14.225424    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f10b39c_95b3_48d9_bc4e_8eee69860358.slice/crio-8393bc52f1da90dd12c3ef951221577fc6101f0d404f8d2933ba003183cd76dc WatchSource:0}: Error finding container 8393bc52f1da90dd12c3ef951221577fc6101f0d404f8d2933ba003183cd76dc: Status 404 returned error can't find the container with id 8393bc52f1da90dd12c3ef951221577fc6101f0d404f8d2933ba003183cd76dc
Sep 17 16:50:15.084257 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:15.084226    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-ctg25" event={"ID":"5f10b39c-95b3-48d9-bc4e-8eee69860358","Type":"ContainerStarted","Data":"8393bc52f1da90dd12c3ef951221577fc6101f0d404f8d2933ba003183cd76dc"}
Sep 17 16:50:17.091998 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:17.091963    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-ctg25" event={"ID":"5f10b39c-95b3-48d9-bc4e-8eee69860358","Type":"ContainerStarted","Data":"33d9e5810f8fe72094e4aeffd9b367d515086d585ade95afc5c438bb518a0f59"}
Sep 17 16:50:17.092363 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:17.092095    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-ctg25"
Sep 17 16:50:17.121075 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:17.121011    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-ctg25" podStartSLOduration=1.773521868 podStartE2EDuration="4.120997242s" totalImagesPullingTime="2.347475374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:50:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:50:17.120589423 +0000 UTC m=+503.015068563" watchObservedRunningTime="2026-09-17 16:50:17.120997242 +0000 UTC m=+503.015476442"
Sep 17 16:50:24.121023 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:24.120986    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-zrjf9" event={"ID":"79b0e787-c6b7-4d04-b5f9-9965bfeb24aa","Type":"ContainerStarted","Data":"ca8f57e51782c0f7ac71f99af7a13dc06c301921a1f60ba22a02a40d1e2de719"}
Sep 17 16:50:24.121403 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:24.121073    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-zrjf9"
Sep 17 16:50:24.145310 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:24.145264    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-zrjf9" podStartSLOduration=2.256858357 podStartE2EDuration="12.145251031s" totalImagesPullingTime="9.888392674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:50:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:50:24.143322948 +0000 UTC m=+510.037802099" watchObservedRunningTime="2026-09-17 16:50:24.145251031 +0000 UTC m=+510.039730171"
Sep 17 16:50:33.077482 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:33.077452    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jxw47"
Sep 17 16:50:38.097472 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:38.097365    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-ctg25"
Sep 17 16:50:45.126933 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:50:45.126904    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-zrjf9"
Sep 17 16:51:03.787092 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:03.787054    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404mmwf"]
Sep 17 16:51:03.792387 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:03.792361    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404mmwf"
Sep 17 16:51:03.794667 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:03.794638    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 16:51:03.795291 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:03.795272    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-gfmjh\""
Sep 17 16:51:03.795381 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:03.795272    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 16:51:03.801716 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:03.801686    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404mmwf"]
Sep 17 16:51:03.922831 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:03.922783    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28020565-7b1d-471f-9835-99da1e7b54e6-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404mmwf\" (UID: \"28020565-7b1d-471f-9835-99da1e7b54e6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404mmwf"
Sep 17 16:51:03.923015 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:03.922862    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28020565-7b1d-471f-9835-99da1e7b54e6-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404mmwf\" (UID: \"28020565-7b1d-471f-9835-99da1e7b54e6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404mmwf"
Sep 17 16:51:03.923015 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:03.922913    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfk9c\" (UniqueName: \"kubernetes.io/projected/28020565-7b1d-471f-9835-99da1e7b54e6-kube-api-access-wfk9c\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404mmwf\" (UID: \"28020565-7b1d-471f-9835-99da1e7b54e6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404mmwf"
Sep 17 16:51:04.023856 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:04.023824    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28020565-7b1d-471f-9835-99da1e7b54e6-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404mmwf\" (UID: \"28020565-7b1d-471f-9835-99da1e7b54e6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404mmwf"
Sep 17 16:51:04.024009 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:04.023865    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfk9c\" (UniqueName: \"kubernetes.io/projected/28020565-7b1d-471f-9835-99da1e7b54e6-kube-api-access-wfk9c\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404mmwf\" (UID: \"28020565-7b1d-471f-9835-99da1e7b54e6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404mmwf"
Sep 17 16:51:04.024009 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:04.023993    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28020565-7b1d-471f-9835-99da1e7b54e6-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404mmwf\" (UID: \"28020565-7b1d-471f-9835-99da1e7b54e6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404mmwf"
Sep 17 16:51:04.024304 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:04.024283    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28020565-7b1d-471f-9835-99da1e7b54e6-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404mmwf\" (UID: \"28020565-7b1d-471f-9835-99da1e7b54e6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404mmwf"
Sep 17 16:51:04.024344 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:04.024307    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28020565-7b1d-471f-9835-99da1e7b54e6-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404mmwf\" (UID: \"28020565-7b1d-471f-9835-99da1e7b54e6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404mmwf"
Sep 17 16:51:04.037785 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:04.037715    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfk9c\" (UniqueName: \"kubernetes.io/projected/28020565-7b1d-471f-9835-99da1e7b54e6-kube-api-access-wfk9c\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404mmwf\" (UID: \"28020565-7b1d-471f-9835-99da1e7b54e6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404mmwf"
Sep 17 16:51:04.103388 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:04.103344    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404mmwf"
Sep 17 16:51:04.376595 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:04.376564    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404mmwf"]
Sep 17 16:51:04.379921 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:51:04.379885    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28020565_7b1d_471f_9835_99da1e7b54e6.slice/crio-4436f6a743de030e8284c9623e6a1abb567281dae91ea4b8f5336bdf092a3b7a WatchSource:0}: Error finding container 4436f6a743de030e8284c9623e6a1abb567281dae91ea4b8f5336bdf092a3b7a: Status 404 returned error can't find the container with id 4436f6a743de030e8284c9623e6a1abb567281dae91ea4b8f5336bdf092a3b7a
Sep 17 16:51:05.268272 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:05.268241    2566 generic.go:356] "Generic (PLEG): container finished" podID="28020565-7b1d-471f-9835-99da1e7b54e6" containerID="95b19f3ec29eec5960c2ff47993cf3ea4dc1c526b6d7a0ae042fea8bd54e8578" exitCode=0
Sep 17 16:51:05.268672 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:05.268300    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404mmwf" event={"ID":"28020565-7b1d-471f-9835-99da1e7b54e6","Type":"ContainerDied","Data":"95b19f3ec29eec5960c2ff47993cf3ea4dc1c526b6d7a0ae042fea8bd54e8578"}
Sep 17 16:51:05.268672 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:05.268342    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404mmwf" event={"ID":"28020565-7b1d-471f-9835-99da1e7b54e6","Type":"ContainerStarted","Data":"4436f6a743de030e8284c9623e6a1abb567281dae91ea4b8f5336bdf092a3b7a"}
Sep 17 16:51:08.282373 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:08.282342    2566 generic.go:356] "Generic (PLEG): container finished" podID="28020565-7b1d-471f-9835-99da1e7b54e6" containerID="0fb2dcb88ceb8511dfb5ef17a0c2f69f08544c0d52a26d0bf6e4c81cdf84adf1" exitCode=0
Sep 17 16:51:08.282781 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:08.282411    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404mmwf" event={"ID":"28020565-7b1d-471f-9835-99da1e7b54e6","Type":"ContainerDied","Data":"0fb2dcb88ceb8511dfb5ef17a0c2f69f08544c0d52a26d0bf6e4c81cdf84adf1"}
Sep 17 16:51:09.288098 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:09.288062    2566 generic.go:356] "Generic (PLEG): container finished" podID="28020565-7b1d-471f-9835-99da1e7b54e6" containerID="0f7a0207aa57c44ca1edc1b70723c2a2e6f65011646f8b7ef5cfaab4a8963042" exitCode=0
Sep 17 16:51:09.288495 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:09.288114    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404mmwf" event={"ID":"28020565-7b1d-471f-9835-99da1e7b54e6","Type":"ContainerDied","Data":"0f7a0207aa57c44ca1edc1b70723c2a2e6f65011646f8b7ef5cfaab4a8963042"}
Sep 17 16:51:10.418554 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:10.418531    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404mmwf"
Sep 17 16:51:10.470183 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:10.470155    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfk9c\" (UniqueName: \"kubernetes.io/projected/28020565-7b1d-471f-9835-99da1e7b54e6-kube-api-access-wfk9c\") pod \"28020565-7b1d-471f-9835-99da1e7b54e6\" (UID: \"28020565-7b1d-471f-9835-99da1e7b54e6\") "
Sep 17 16:51:10.470309 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:10.470205    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28020565-7b1d-471f-9835-99da1e7b54e6-bundle\") pod \"28020565-7b1d-471f-9835-99da1e7b54e6\" (UID: \"28020565-7b1d-471f-9835-99da1e7b54e6\") "
Sep 17 16:51:10.470309 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:10.470255    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28020565-7b1d-471f-9835-99da1e7b54e6-util\") pod \"28020565-7b1d-471f-9835-99da1e7b54e6\" (UID: \"28020565-7b1d-471f-9835-99da1e7b54e6\") "
Sep 17 16:51:10.470997 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:10.470915    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28020565-7b1d-471f-9835-99da1e7b54e6-bundle" (OuterVolumeSpecName: "bundle") pod "28020565-7b1d-471f-9835-99da1e7b54e6" (UID: "28020565-7b1d-471f-9835-99da1e7b54e6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:51:10.472384 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:10.472355    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28020565-7b1d-471f-9835-99da1e7b54e6-kube-api-access-wfk9c" (OuterVolumeSpecName: "kube-api-access-wfk9c") pod "28020565-7b1d-471f-9835-99da1e7b54e6" (UID: "28020565-7b1d-471f-9835-99da1e7b54e6"). InnerVolumeSpecName "kube-api-access-wfk9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:51:10.475102 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:10.475070    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28020565-7b1d-471f-9835-99da1e7b54e6-util" (OuterVolumeSpecName: "util") pod "28020565-7b1d-471f-9835-99da1e7b54e6" (UID: "28020565-7b1d-471f-9835-99da1e7b54e6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:51:10.570894 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:10.570810    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfk9c\" (UniqueName: \"kubernetes.io/projected/28020565-7b1d-471f-9835-99da1e7b54e6-kube-api-access-wfk9c\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:51:10.570894 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:10.570839    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28020565-7b1d-471f-9835-99da1e7b54e6-bundle\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:51:10.570894 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:10.570849    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28020565-7b1d-471f-9835-99da1e7b54e6-util\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:51:11.296674 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:11.296644    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404mmwf" event={"ID":"28020565-7b1d-471f-9835-99da1e7b54e6","Type":"ContainerDied","Data":"4436f6a743de030e8284c9623e6a1abb567281dae91ea4b8f5336bdf092a3b7a"}
Sep 17 16:51:11.296830 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:11.296693    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4436f6a743de030e8284c9623e6a1abb567281dae91ea4b8f5336bdf092a3b7a"
Sep 17 16:51:11.296830 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:11.296655    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404mmwf"
Sep 17 16:51:16.723761 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:16.723731    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-4cwhc"]
Sep 17 16:51:16.724260 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:16.723962    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28020565-7b1d-471f-9835-99da1e7b54e6" containerName="extract"
Sep 17 16:51:16.724260 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:16.723974    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="28020565-7b1d-471f-9835-99da1e7b54e6" containerName="extract"
Sep 17 16:51:16.724260 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:16.724014    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28020565-7b1d-471f-9835-99da1e7b54e6" containerName="util"
Sep 17 16:51:16.724260 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:16.724023    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="28020565-7b1d-471f-9835-99da1e7b54e6" containerName="util"
Sep 17 16:51:16.724260 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:16.724031    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28020565-7b1d-471f-9835-99da1e7b54e6" containerName="pull"
Sep 17 16:51:16.724260 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:16.724039    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="28020565-7b1d-471f-9835-99da1e7b54e6" containerName="pull"
Sep 17 16:51:16.724260 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:16.724100    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="28020565-7b1d-471f-9835-99da1e7b54e6" containerName="extract"
Sep 17 16:51:16.727140 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:16.727124    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-4cwhc"
Sep 17 16:51:16.731406 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:16.731381    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 17 16:51:16.731546 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:16.731425    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 17 16:51:16.731546 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:16.731430    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-rvkn9\""
Sep 17 16:51:16.796489 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:16.796451    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-4cwhc"]
Sep 17 16:51:16.814827 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:16.814793    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9647f934-f796-4808-b3c8-ea24fc2cc284-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-4cwhc\" (UID: \"9647f934-f796-4808-b3c8-ea24fc2cc284\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-4cwhc"
Sep 17 16:51:16.814827 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:16.814828    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndfw7\" (UniqueName: \"kubernetes.io/projected/9647f934-f796-4808-b3c8-ea24fc2cc284-kube-api-access-ndfw7\") pod \"cert-manager-operator-controller-manager-684f79c9b9-4cwhc\" (UID: \"9647f934-f796-4808-b3c8-ea24fc2cc284\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-4cwhc"
Sep 17 16:51:16.916153 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:16.916120    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9647f934-f796-4808-b3c8-ea24fc2cc284-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-4cwhc\" (UID: \"9647f934-f796-4808-b3c8-ea24fc2cc284\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-4cwhc"
Sep 17 16:51:16.916321 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:16.916158    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndfw7\" (UniqueName: \"kubernetes.io/projected/9647f934-f796-4808-b3c8-ea24fc2cc284-kube-api-access-ndfw7\") pod \"cert-manager-operator-controller-manager-684f79c9b9-4cwhc\" (UID: \"9647f934-f796-4808-b3c8-ea24fc2cc284\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-4cwhc"
Sep 17 16:51:16.916601 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:16.916580    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9647f934-f796-4808-b3c8-ea24fc2cc284-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-4cwhc\" (UID: \"9647f934-f796-4808-b3c8-ea24fc2cc284\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-4cwhc"
Sep 17 16:51:16.927371 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:16.927345    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndfw7\" (UniqueName: \"kubernetes.io/projected/9647f934-f796-4808-b3c8-ea24fc2cc284-kube-api-access-ndfw7\") pod \"cert-manager-operator-controller-manager-684f79c9b9-4cwhc\" (UID: \"9647f934-f796-4808-b3c8-ea24fc2cc284\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-4cwhc"
Sep 17 16:51:17.035731 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:17.035649    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-4cwhc"
Sep 17 16:51:17.324730 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:17.324707    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-4cwhc"]
Sep 17 16:51:17.327241 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:51:17.327211    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9647f934_f796_4808_b3c8_ea24fc2cc284.slice/crio-e16be1e1169472f59ac629c1c6dcace2a6dbda226b900dda093e630464658b64 WatchSource:0}: Error finding container e16be1e1169472f59ac629c1c6dcace2a6dbda226b900dda093e630464658b64: Status 404 returned error can't find the container with id e16be1e1169472f59ac629c1c6dcace2a6dbda226b900dda093e630464658b64
Sep 17 16:51:18.322322 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:18.322283    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-4cwhc" event={"ID":"9647f934-f796-4808-b3c8-ea24fc2cc284","Type":"ContainerStarted","Data":"e16be1e1169472f59ac629c1c6dcace2a6dbda226b900dda093e630464658b64"}
Sep 17 16:51:20.333349 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:20.333316    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-4cwhc" event={"ID":"9647f934-f796-4808-b3c8-ea24fc2cc284","Type":"ContainerStarted","Data":"726d157aad24418b27a81d977932731176263c190e9ded84cb7525af32a0c009"}
Sep 17 16:51:20.367250 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:20.367187    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-4cwhc" podStartSLOduration=2.392249354 podStartE2EDuration="4.367168352s" totalImagesPullingTime="1.974918998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:51:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:51:20.364195142 +0000 UTC m=+566.258674275" watchObservedRunningTime="2026-09-17 16:51:20.367168352 +0000 UTC m=+566.261647496"
Sep 17 16:51:25.277253 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:25.277220    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qdtv"]
Sep 17 16:51:25.280592 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:25.280574    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qdtv"
Sep 17 16:51:25.283033 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:25.283011    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 16:51:25.283858 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:25.283838    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-gfmjh\""
Sep 17 16:51:25.286426 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:25.286407    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 16:51:25.295295 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:25.295275    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qdtv"]
Sep 17 16:51:25.373824 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:25.373791    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dcf0c436-2de3-46f2-88ba-40c7b0b5214b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qdtv\" (UID: \"dcf0c436-2de3-46f2-88ba-40c7b0b5214b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qdtv"
Sep 17 16:51:25.373824 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:25.373821    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dcf0c436-2de3-46f2-88ba-40c7b0b5214b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qdtv\" (UID: \"dcf0c436-2de3-46f2-88ba-40c7b0b5214b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qdtv"
Sep 17 16:51:25.374062 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:25.373844    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm7bl\" (UniqueName: \"kubernetes.io/projected/dcf0c436-2de3-46f2-88ba-40c7b0b5214b-kube-api-access-dm7bl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qdtv\" (UID: \"dcf0c436-2de3-46f2-88ba-40c7b0b5214b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qdtv"
Sep 17 16:51:25.475137 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:25.475103    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dcf0c436-2de3-46f2-88ba-40c7b0b5214b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qdtv\" (UID: \"dcf0c436-2de3-46f2-88ba-40c7b0b5214b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qdtv"
Sep 17 16:51:25.475137 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:25.475135    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dcf0c436-2de3-46f2-88ba-40c7b0b5214b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qdtv\" (UID: \"dcf0c436-2de3-46f2-88ba-40c7b0b5214b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qdtv"
Sep 17 16:51:25.475345 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:25.475157    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm7bl\" (UniqueName: \"kubernetes.io/projected/dcf0c436-2de3-46f2-88ba-40c7b0b5214b-kube-api-access-dm7bl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qdtv\" (UID: \"dcf0c436-2de3-46f2-88ba-40c7b0b5214b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qdtv"
Sep 17 16:51:25.475527 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:25.475509    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dcf0c436-2de3-46f2-88ba-40c7b0b5214b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qdtv\" (UID: \"dcf0c436-2de3-46f2-88ba-40c7b0b5214b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qdtv"
Sep 17 16:51:25.475568 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:25.475531    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dcf0c436-2de3-46f2-88ba-40c7b0b5214b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qdtv\" (UID: \"dcf0c436-2de3-46f2-88ba-40c7b0b5214b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qdtv"
Sep 17 16:51:25.484625 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:25.484600    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm7bl\" (UniqueName: \"kubernetes.io/projected/dcf0c436-2de3-46f2-88ba-40c7b0b5214b-kube-api-access-dm7bl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qdtv\" (UID: \"dcf0c436-2de3-46f2-88ba-40c7b0b5214b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qdtv"
Sep 17 16:51:25.590300 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:25.590266    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qdtv"
Sep 17 16:51:25.860493 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:25.860470    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qdtv"]
Sep 17 16:51:25.862795 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:51:25.862767    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcf0c436_2de3_46f2_88ba_40c7b0b5214b.slice/crio-a0f91f3cc32b46ab03f17985d89c145183521a77bf4fc9e04b5c0a4eeac571b7 WatchSource:0}: Error finding container a0f91f3cc32b46ab03f17985d89c145183521a77bf4fc9e04b5c0a4eeac571b7: Status 404 returned error can't find the container with id a0f91f3cc32b46ab03f17985d89c145183521a77bf4fc9e04b5c0a4eeac571b7
Sep 17 16:51:26.359355 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:26.359321    2566 generic.go:356] "Generic (PLEG): container finished" podID="dcf0c436-2de3-46f2-88ba-40c7b0b5214b" containerID="bcf81b8f1101a48371a439686e7fe7dee027f30e7080c3a3640831e1868201c2" exitCode=0
Sep 17 16:51:26.359883 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:26.359413    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qdtv" event={"ID":"dcf0c436-2de3-46f2-88ba-40c7b0b5214b","Type":"ContainerDied","Data":"bcf81b8f1101a48371a439686e7fe7dee027f30e7080c3a3640831e1868201c2"}
Sep 17 16:51:26.359883 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:26.359475    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qdtv" event={"ID":"dcf0c436-2de3-46f2-88ba-40c7b0b5214b","Type":"ContainerStarted","Data":"a0f91f3cc32b46ab03f17985d89c145183521a77bf4fc9e04b5c0a4eeac571b7"}
Sep 17 16:51:29.374225 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:29.374187    2566 generic.go:356] "Generic (PLEG): container finished" podID="dcf0c436-2de3-46f2-88ba-40c7b0b5214b" containerID="5357c0b4a4a4669aa140a15f6d4bbaf7534484039143a67ad425f4a395b3fa05" exitCode=0
Sep 17 16:51:29.374624 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:29.374252    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qdtv" event={"ID":"dcf0c436-2de3-46f2-88ba-40c7b0b5214b","Type":"ContainerDied","Data":"5357c0b4a4a4669aa140a15f6d4bbaf7534484039143a67ad425f4a395b3fa05"}
Sep 17 16:51:30.380682 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:30.380651    2566 generic.go:356] "Generic (PLEG): container finished" podID="dcf0c436-2de3-46f2-88ba-40c7b0b5214b" containerID="53125dc9f7344ad6efb4aef8f57fecee92d85cccd8dd5be06f34f4ae7cb6ffe8" exitCode=0
Sep 17 16:51:30.381061 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:30.380709    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qdtv" event={"ID":"dcf0c436-2de3-46f2-88ba-40c7b0b5214b","Type":"ContainerDied","Data":"53125dc9f7344ad6efb4aef8f57fecee92d85cccd8dd5be06f34f4ae7cb6ffe8"}
Sep 17 16:51:31.505769 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:31.505744    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qdtv"
Sep 17 16:51:31.618162 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:31.618130    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dcf0c436-2de3-46f2-88ba-40c7b0b5214b-util\") pod \"dcf0c436-2de3-46f2-88ba-40c7b0b5214b\" (UID: \"dcf0c436-2de3-46f2-88ba-40c7b0b5214b\") "
Sep 17 16:51:31.618162 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:31.618164    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dcf0c436-2de3-46f2-88ba-40c7b0b5214b-bundle\") pod \"dcf0c436-2de3-46f2-88ba-40c7b0b5214b\" (UID: \"dcf0c436-2de3-46f2-88ba-40c7b0b5214b\") "
Sep 17 16:51:31.618340 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:31.618225    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dm7bl\" (UniqueName: \"kubernetes.io/projected/dcf0c436-2de3-46f2-88ba-40c7b0b5214b-kube-api-access-dm7bl\") pod \"dcf0c436-2de3-46f2-88ba-40c7b0b5214b\" (UID: \"dcf0c436-2de3-46f2-88ba-40c7b0b5214b\") "
Sep 17 16:51:31.618642 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:31.618592    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf0c436-2de3-46f2-88ba-40c7b0b5214b-bundle" (OuterVolumeSpecName: "bundle") pod "dcf0c436-2de3-46f2-88ba-40c7b0b5214b" (UID: "dcf0c436-2de3-46f2-88ba-40c7b0b5214b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:51:31.620371 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:31.620347    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcf0c436-2de3-46f2-88ba-40c7b0b5214b-kube-api-access-dm7bl" (OuterVolumeSpecName: "kube-api-access-dm7bl") pod "dcf0c436-2de3-46f2-88ba-40c7b0b5214b" (UID: "dcf0c436-2de3-46f2-88ba-40c7b0b5214b"). InnerVolumeSpecName "kube-api-access-dm7bl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:51:31.623094 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:31.623074    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf0c436-2de3-46f2-88ba-40c7b0b5214b-util" (OuterVolumeSpecName: "util") pod "dcf0c436-2de3-46f2-88ba-40c7b0b5214b" (UID: "dcf0c436-2de3-46f2-88ba-40c7b0b5214b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:51:31.718851 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:31.718774    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm7bl\" (UniqueName: \"kubernetes.io/projected/dcf0c436-2de3-46f2-88ba-40c7b0b5214b-kube-api-access-dm7bl\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:51:31.718851 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:31.718799    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dcf0c436-2de3-46f2-88ba-40c7b0b5214b-util\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:51:31.718851 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:31.718809    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dcf0c436-2de3-46f2-88ba-40c7b0b5214b-bundle\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:51:32.390070 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:32.390041    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qdtv"
Sep 17 16:51:32.390192 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:32.390048    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4qdtv" event={"ID":"dcf0c436-2de3-46f2-88ba-40c7b0b5214b","Type":"ContainerDied","Data":"a0f91f3cc32b46ab03f17985d89c145183521a77bf4fc9e04b5c0a4eeac571b7"}
Sep 17 16:51:32.390192 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:32.390156    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0f91f3cc32b46ab03f17985d89c145183521a77bf4fc9e04b5c0a4eeac571b7"
Sep 17 16:51:36.858134 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:36.858101    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gkcdn"]
Sep 17 16:51:36.858506 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:36.858343    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcf0c436-2de3-46f2-88ba-40c7b0b5214b" containerName="pull"
Sep 17 16:51:36.858506 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:36.858353    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcf0c436-2de3-46f2-88ba-40c7b0b5214b" containerName="pull"
Sep 17 16:51:36.858506 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:36.858365    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcf0c436-2de3-46f2-88ba-40c7b0b5214b" containerName="util"
Sep 17 16:51:36.858506 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:36.858370    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcf0c436-2de3-46f2-88ba-40c7b0b5214b" containerName="util"
Sep 17 16:51:36.858506 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:36.858376    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcf0c436-2de3-46f2-88ba-40c7b0b5214b" containerName="extract"
Sep 17 16:51:36.858506 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:36.858382    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcf0c436-2de3-46f2-88ba-40c7b0b5214b" containerName="extract"
Sep 17 16:51:36.858506 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:36.858445    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcf0c436-2de3-46f2-88ba-40c7b0b5214b" containerName="extract"
Sep 17 16:51:36.860918 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:36.860904    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gkcdn"
Sep 17 16:51:36.868147 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:36.868120    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 17 16:51:36.868290 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:36.868261    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-zg6jw\""
Sep 17 16:51:36.868596 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:36.868580    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 17 16:51:36.882007 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:36.881984    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gkcdn"]
Sep 17 16:51:36.956965 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:36.956937    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c19fe3f1-8cde-4773-875b-c125d01d964a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gkcdn\" (UID: \"c19fe3f1-8cde-4773-875b-c125d01d964a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gkcdn"
Sep 17 16:51:36.957134 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:36.956985    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc7f5\" (UniqueName: \"kubernetes.io/projected/c19fe3f1-8cde-4773-875b-c125d01d964a-kube-api-access-zc7f5\") pod \"openshift-lws-operator-fd8ccff4c-gkcdn\" (UID: \"c19fe3f1-8cde-4773-875b-c125d01d964a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gkcdn"
Sep 17 16:51:37.058190 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:37.058153    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c19fe3f1-8cde-4773-875b-c125d01d964a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gkcdn\" (UID: \"c19fe3f1-8cde-4773-875b-c125d01d964a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gkcdn"
Sep 17 16:51:37.058340 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:37.058203    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc7f5\" (UniqueName: \"kubernetes.io/projected/c19fe3f1-8cde-4773-875b-c125d01d964a-kube-api-access-zc7f5\") pod \"openshift-lws-operator-fd8ccff4c-gkcdn\" (UID: \"c19fe3f1-8cde-4773-875b-c125d01d964a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gkcdn"
Sep 17 16:51:37.058578 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:37.058558    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c19fe3f1-8cde-4773-875b-c125d01d964a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gkcdn\" (UID: \"c19fe3f1-8cde-4773-875b-c125d01d964a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gkcdn"
Sep 17 16:51:37.071215 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:37.071184    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc7f5\" (UniqueName: \"kubernetes.io/projected/c19fe3f1-8cde-4773-875b-c125d01d964a-kube-api-access-zc7f5\") pod \"openshift-lws-operator-fd8ccff4c-gkcdn\" (UID: \"c19fe3f1-8cde-4773-875b-c125d01d964a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gkcdn"
Sep 17 16:51:37.173365 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:37.173293    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gkcdn"
Sep 17 16:51:37.442633 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:37.442606    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gkcdn"]
Sep 17 16:51:37.445180 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:51:37.445153    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc19fe3f1_8cde_4773_875b_c125d01d964a.slice/crio-943abf117bec596d9ce432e5cb43cc54c968bca00bbc0ef09ea9bbde5d4ebdf8 WatchSource:0}: Error finding container 943abf117bec596d9ce432e5cb43cc54c968bca00bbc0ef09ea9bbde5d4ebdf8: Status 404 returned error can't find the container with id 943abf117bec596d9ce432e5cb43cc54c968bca00bbc0ef09ea9bbde5d4ebdf8
Sep 17 16:51:38.416253 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:38.416214    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gkcdn" event={"ID":"c19fe3f1-8cde-4773-875b-c125d01d964a","Type":"ContainerStarted","Data":"943abf117bec596d9ce432e5cb43cc54c968bca00bbc0ef09ea9bbde5d4ebdf8"}
Sep 17 16:51:39.422844 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:39.422805    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gkcdn" event={"ID":"c19fe3f1-8cde-4773-875b-c125d01d964a","Type":"ContainerStarted","Data":"92304beeacfe981323e9eed06e6ec5d365c547840cf8c21ce3d0ce9539e74869"}
Sep 17 16:51:39.446066 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:39.446016    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gkcdn" podStartSLOduration=1.630549116 podStartE2EDuration="3.446000672s" totalImagesPullingTime="1.815451556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:51:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:51:39.445185853 +0000 UTC m=+585.339664992" watchObservedRunningTime="2026-09-17 16:51:39.446000672 +0000 UTC m=+585.340479810"
Sep 17 16:51:48.593795 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:48.593763    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w4b7c"]
Sep 17 16:51:48.601444 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:48.601414    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w4b7c"
Sep 17 16:51:48.603783 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:48.603758    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 17 16:51:48.603926 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:48.603765    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 17 16:51:48.603926 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:48.603798    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 16:51:48.604011 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:48.603950    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 17 16:51:48.604011 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:48.603975    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-6bb4c\""
Sep 17 16:51:48.604283 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:48.604266    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 17 16:51:48.604370 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:48.604289    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 16:51:48.619903 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:48.619875    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w4b7c"]
Sep 17 16:51:48.744533 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:48.744497    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/28ebe838-968c-4986-b80d-f5654eb1e2c7-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-w4b7c\" (UID: \"28ebe838-968c-4986-b80d-f5654eb1e2c7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w4b7c"
Sep 17 16:51:48.744533 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:48.744537    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/28ebe838-968c-4986-b80d-f5654eb1e2c7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-w4b7c\" (UID: \"28ebe838-968c-4986-b80d-f5654eb1e2c7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w4b7c"
Sep 17 16:51:48.744766 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:48.744580    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/28ebe838-968c-4986-b80d-f5654eb1e2c7-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-w4b7c\" (UID: \"28ebe838-968c-4986-b80d-f5654eb1e2c7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w4b7c"
Sep 17 16:51:48.744766 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:48.744617    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28ebe838-968c-4986-b80d-f5654eb1e2c7-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-w4b7c\" (UID: \"28ebe838-968c-4986-b80d-f5654eb1e2c7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w4b7c"
Sep 17 16:51:48.744766 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:48.744705    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfmrj\" (UniqueName: \"kubernetes.io/projected/28ebe838-968c-4986-b80d-f5654eb1e2c7-kube-api-access-kfmrj\") pod \"istiod-openshift-gateway-7b7995d5cd-w4b7c\" (UID: \"28ebe838-968c-4986-b80d-f5654eb1e2c7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w4b7c"
Sep 17 16:51:48.744766 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:48.744733    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/28ebe838-968c-4986-b80d-f5654eb1e2c7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-w4b7c\" (UID: \"28ebe838-968c-4986-b80d-f5654eb1e2c7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w4b7c"
Sep 17 16:51:48.744766 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:48.744749    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/28ebe838-968c-4986-b80d-f5654eb1e2c7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-w4b7c\" (UID: \"28ebe838-968c-4986-b80d-f5654eb1e2c7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w4b7c"
Sep 17 16:51:48.850477 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:48.850371    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/28ebe838-968c-4986-b80d-f5654eb1e2c7-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-w4b7c\" (UID: \"28ebe838-968c-4986-b80d-f5654eb1e2c7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w4b7c"
Sep 17 16:51:48.850477 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:48.850423    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28ebe838-968c-4986-b80d-f5654eb1e2c7-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-w4b7c\" (UID: \"28ebe838-968c-4986-b80d-f5654eb1e2c7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w4b7c"
Sep 17 16:51:48.850477 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:48.850474    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfmrj\" (UniqueName: \"kubernetes.io/projected/28ebe838-968c-4986-b80d-f5654eb1e2c7-kube-api-access-kfmrj\") pod \"istiod-openshift-gateway-7b7995d5cd-w4b7c\" (UID: \"28ebe838-968c-4986-b80d-f5654eb1e2c7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w4b7c"
Sep 17 16:51:48.850717 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:48.850509    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/28ebe838-968c-4986-b80d-f5654eb1e2c7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-w4b7c\" (UID: \"28ebe838-968c-4986-b80d-f5654eb1e2c7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w4b7c"
Sep 17 16:51:48.850717 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:48.850534    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/28ebe838-968c-4986-b80d-f5654eb1e2c7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-w4b7c\" (UID: \"28ebe838-968c-4986-b80d-f5654eb1e2c7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w4b7c"
Sep 17 16:51:48.850717 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:48.850575    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/28ebe838-968c-4986-b80d-f5654eb1e2c7-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-w4b7c\" (UID: \"28ebe838-968c-4986-b80d-f5654eb1e2c7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w4b7c"
Sep 17 16:51:48.850717 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:48.850606    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/28ebe838-968c-4986-b80d-f5654eb1e2c7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-w4b7c\" (UID: \"28ebe838-968c-4986-b80d-f5654eb1e2c7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w4b7c"
Sep 17 16:51:48.851192 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:48.851162    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/28ebe838-968c-4986-b80d-f5654eb1e2c7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-w4b7c\" (UID: \"28ebe838-968c-4986-b80d-f5654eb1e2c7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w4b7c"
Sep 17 16:51:48.852874 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:48.852843    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/28ebe838-968c-4986-b80d-f5654eb1e2c7-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-w4b7c\" (UID: \"28ebe838-968c-4986-b80d-f5654eb1e2c7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w4b7c"
Sep 17 16:51:48.852984 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:48.852967    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/28ebe838-968c-4986-b80d-f5654eb1e2c7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-w4b7c\" (UID: \"28ebe838-968c-4986-b80d-f5654eb1e2c7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w4b7c"
Sep 17 16:51:48.853529 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:48.853509    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/28ebe838-968c-4986-b80d-f5654eb1e2c7-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-w4b7c\" (UID: \"28ebe838-968c-4986-b80d-f5654eb1e2c7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w4b7c"
Sep 17 16:51:48.853606 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:48.853589    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/28ebe838-968c-4986-b80d-f5654eb1e2c7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-w4b7c\" (UID: \"28ebe838-968c-4986-b80d-f5654eb1e2c7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w4b7c"
Sep 17 16:51:48.864485 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:48.864459    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28ebe838-968c-4986-b80d-f5654eb1e2c7-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-w4b7c\" (UID: \"28ebe838-968c-4986-b80d-f5654eb1e2c7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w4b7c"
Sep 17 16:51:48.902191 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:48.902160    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfmrj\" (UniqueName: \"kubernetes.io/projected/28ebe838-968c-4986-b80d-f5654eb1e2c7-kube-api-access-kfmrj\") pod \"istiod-openshift-gateway-7b7995d5cd-w4b7c\" (UID: \"28ebe838-968c-4986-b80d-f5654eb1e2c7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w4b7c"
Sep 17 16:51:48.910005 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:48.909976    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w4b7c"
Sep 17 16:51:49.195034 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:49.194950    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w4b7c"]
Sep 17 16:51:49.199491 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:51:49.198669    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28ebe838_968c_4986_b80d_f5654eb1e2c7.slice/crio-095cdbb2b1f63db170346a3153a72a6e269197d5dda74eaf112d436b3ad314f4 WatchSource:0}: Error finding container 095cdbb2b1f63db170346a3153a72a6e269197d5dda74eaf112d436b3ad314f4: Status 404 returned error can't find the container with id 095cdbb2b1f63db170346a3153a72a6e269197d5dda74eaf112d436b3ad314f4
Sep 17 16:51:49.467648 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:49.467554    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w4b7c" event={"ID":"28ebe838-968c-4986-b80d-f5654eb1e2c7","Type":"ContainerStarted","Data":"095cdbb2b1f63db170346a3153a72a6e269197d5dda74eaf112d436b3ad314f4"}
Sep 17 16:51:56.272966 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:56.272936    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gblhr_91477834-6648-4e75-8530-e01bdee5e57e/ovn-acl-logging/0.log"
Sep 17 16:51:56.274394 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:51:56.274371    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gblhr_91477834-6648-4e75-8530-e01bdee5e57e/ovn-acl-logging/0.log"
Sep 17 16:52:08.197169 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:08.197130    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 16:52:08.197459 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:08.197197    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 16:52:08.549588 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:08.549557    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w4b7c" event={"ID":"28ebe838-968c-4986-b80d-f5654eb1e2c7","Type":"ContainerStarted","Data":"572f63259d108bc7b99f8bbc9960101f2de580cb0d3889d56ddb33bf42a4244a"}
Sep 17 16:52:08.549757 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:08.549661    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w4b7c"
Sep 17 16:52:08.577870 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:08.577816    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w4b7c" podStartSLOduration=1.582867213 podStartE2EDuration="20.577798721s" totalImagesPullingTime="18.994931508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:51:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:52:08.576336589 +0000 UTC m=+614.470815739" watchObservedRunningTime="2026-09-17 16:52:08.577798721 +0000 UTC m=+614.472277862"
Sep 17 16:52:09.556987 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:09.556946    2566 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-w4b7c container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 16:52:09.557415 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:09.557016    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w4b7c" podUID="28ebe838-968c-4986-b80d-f5654eb1e2c7" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:52:12.556482 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:12.556422    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w4b7c"
Sep 17 16:52:21.680853 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:21.680816    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6px74h"]
Sep 17 16:52:21.684161 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:21.684144    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6px74h"
Sep 17 16:52:21.686419 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:21.686393    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 16:52:21.686552 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:21.686463    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 16:52:21.686552 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:21.686465    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-gfmjh\""
Sep 17 16:52:21.694704 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:21.694682    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6px74h"]
Sep 17 16:52:21.775798 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:21.775764    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsvrf"]
Sep 17 16:52:21.779153 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:21.779130    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsvrf"
Sep 17 16:52:21.781395 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:21.781370    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfv8d\" (UniqueName: \"kubernetes.io/projected/fe91cb91-4790-479f-adf7-dd96a854e3a4-kube-api-access-jfv8d\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6px74h\" (UID: \"fe91cb91-4790-479f-adf7-dd96a854e3a4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6px74h"
Sep 17 16:52:21.781528 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:21.781422    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe91cb91-4790-479f-adf7-dd96a854e3a4-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6px74h\" (UID: \"fe91cb91-4790-479f-adf7-dd96a854e3a4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6px74h"
Sep 17 16:52:21.781528 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:21.781468    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe91cb91-4790-479f-adf7-dd96a854e3a4-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6px74h\" (UID: \"fe91cb91-4790-479f-adf7-dd96a854e3a4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6px74h"
Sep 17 16:52:21.788289 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:21.788264    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsvrf"]
Sep 17 16:52:21.876616 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:21.876585    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl65r5"]
Sep 17 16:52:21.879883 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:21.879869    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl65r5"
Sep 17 16:52:21.881878 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:21.881853    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe91cb91-4790-479f-adf7-dd96a854e3a4-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6px74h\" (UID: \"fe91cb91-4790-479f-adf7-dd96a854e3a4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6px74h"
Sep 17 16:52:21.881972 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:21.881895    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/72c3347f-4b65-49ab-8878-71374a5eecaa-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsvrf\" (UID: \"72c3347f-4b65-49ab-8878-71374a5eecaa\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsvrf"
Sep 17 16:52:21.881972 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:21.881942    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlbb9\" (UniqueName: \"kubernetes.io/projected/72c3347f-4b65-49ab-8878-71374a5eecaa-kube-api-access-qlbb9\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsvrf\" (UID: \"72c3347f-4b65-49ab-8878-71374a5eecaa\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsvrf"
Sep 17 16:52:21.882082 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:21.881974    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe91cb91-4790-479f-adf7-dd96a854e3a4-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6px74h\" (UID: \"fe91cb91-4790-479f-adf7-dd96a854e3a4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6px74h"
Sep 17 16:52:21.882082 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:21.882006    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/72c3347f-4b65-49ab-8878-71374a5eecaa-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsvrf\" (UID: \"72c3347f-4b65-49ab-8878-71374a5eecaa\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsvrf"
Sep 17 16:52:21.882082 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:21.882045    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfv8d\" (UniqueName: \"kubernetes.io/projected/fe91cb91-4790-479f-adf7-dd96a854e3a4-kube-api-access-jfv8d\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6px74h\" (UID: \"fe91cb91-4790-479f-adf7-dd96a854e3a4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6px74h"
Sep 17 16:52:21.882271 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:21.882249    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe91cb91-4790-479f-adf7-dd96a854e3a4-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6px74h\" (UID: \"fe91cb91-4790-479f-adf7-dd96a854e3a4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6px74h"
Sep 17 16:52:21.882331 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:21.882258    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe91cb91-4790-479f-adf7-dd96a854e3a4-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6px74h\" (UID: \"fe91cb91-4790-479f-adf7-dd96a854e3a4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6px74h"
Sep 17 16:52:21.889032 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:21.889011    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl65r5"]
Sep 17 16:52:21.892798 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:21.892775    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfv8d\" (UniqueName: \"kubernetes.io/projected/fe91cb91-4790-479f-adf7-dd96a854e3a4-kube-api-access-jfv8d\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6px74h\" (UID: \"fe91cb91-4790-479f-adf7-dd96a854e3a4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6px74h"
Sep 17 16:52:21.982912 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:21.982825    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcnjf\" (UniqueName: \"kubernetes.io/projected/d9de42b9-3216-4968-b607-805fb7776e93-kube-api-access-dcnjf\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl65r5\" (UID: \"d9de42b9-3216-4968-b607-805fb7776e93\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl65r5"
Sep 17 16:52:21.982912 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:21.982863    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/72c3347f-4b65-49ab-8878-71374a5eecaa-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsvrf\" (UID: \"72c3347f-4b65-49ab-8878-71374a5eecaa\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsvrf"
Sep 17 16:52:21.982912 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:21.982891    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlbb9\" (UniqueName: \"kubernetes.io/projected/72c3347f-4b65-49ab-8878-71374a5eecaa-kube-api-access-qlbb9\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsvrf\" (UID: \"72c3347f-4b65-49ab-8878-71374a5eecaa\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsvrf"
Sep 17 16:52:21.983191 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:21.982922    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/72c3347f-4b65-49ab-8878-71374a5eecaa-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsvrf\" (UID: \"72c3347f-4b65-49ab-8878-71374a5eecaa\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsvrf"
Sep 17 16:52:21.983191 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:21.982941    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9de42b9-3216-4968-b607-805fb7776e93-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl65r5\" (UID: \"d9de42b9-3216-4968-b607-805fb7776e93\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl65r5"
Sep 17 16:52:21.983191 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:21.982958    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9de42b9-3216-4968-b607-805fb7776e93-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl65r5\" (UID: \"d9de42b9-3216-4968-b607-805fb7776e93\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl65r5"
Sep 17 16:52:21.983341 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:21.983279    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/72c3347f-4b65-49ab-8878-71374a5eecaa-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsvrf\" (UID: \"72c3347f-4b65-49ab-8878-71374a5eecaa\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsvrf"
Sep 17 16:52:21.983395 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:21.983321    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/72c3347f-4b65-49ab-8878-71374a5eecaa-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsvrf\" (UID: \"72c3347f-4b65-49ab-8878-71374a5eecaa\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsvrf"
Sep 17 16:52:21.985166 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:21.985141    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2grmx"]
Sep 17 16:52:21.988772 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:21.988754    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2grmx"
Sep 17 16:52:21.992156 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:21.992136    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlbb9\" (UniqueName: \"kubernetes.io/projected/72c3347f-4b65-49ab-8878-71374a5eecaa-kube-api-access-qlbb9\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsvrf\" (UID: \"72c3347f-4b65-49ab-8878-71374a5eecaa\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsvrf"
Sep 17 16:52:21.993021 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:21.993007    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6px74h"
Sep 17 16:52:22.001026 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:22.001007    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2grmx"]
Sep 17 16:52:22.084001 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:22.083968    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43a42869-aebc-4946-bc3a-70161575845f-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2grmx\" (UID: \"43a42869-aebc-4946-bc3a-70161575845f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2grmx"
Sep 17 16:52:22.084158 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:22.084007    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcnjf\" (UniqueName: \"kubernetes.io/projected/d9de42b9-3216-4968-b607-805fb7776e93-kube-api-access-dcnjf\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl65r5\" (UID: \"d9de42b9-3216-4968-b607-805fb7776e93\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl65r5"
Sep 17 16:52:22.084158 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:22.084036    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43a42869-aebc-4946-bc3a-70161575845f-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2grmx\" (UID: \"43a42869-aebc-4946-bc3a-70161575845f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2grmx"
Sep 17 16:52:22.084158 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:22.084090    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9de42b9-3216-4968-b607-805fb7776e93-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl65r5\" (UID: \"d9de42b9-3216-4968-b607-805fb7776e93\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl65r5"
Sep 17 16:52:22.084158 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:22.084116    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9de42b9-3216-4968-b607-805fb7776e93-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl65r5\" (UID: \"d9de42b9-3216-4968-b607-805fb7776e93\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl65r5"
Sep 17 16:52:22.084158 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:22.084140    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lgr4\" (UniqueName: \"kubernetes.io/projected/43a42869-aebc-4946-bc3a-70161575845f-kube-api-access-8lgr4\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2grmx\" (UID: \"43a42869-aebc-4946-bc3a-70161575845f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2grmx"
Sep 17 16:52:22.084501 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:22.084480    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9de42b9-3216-4968-b607-805fb7776e93-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl65r5\" (UID: \"d9de42b9-3216-4968-b607-805fb7776e93\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl65r5"
Sep 17 16:52:22.084573 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:22.084528    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9de42b9-3216-4968-b607-805fb7776e93-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl65r5\" (UID: \"d9de42b9-3216-4968-b607-805fb7776e93\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl65r5"
Sep 17 16:52:22.087946 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:22.087925    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsvrf"
Sep 17 16:52:22.094846 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:22.094821    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcnjf\" (UniqueName: \"kubernetes.io/projected/d9de42b9-3216-4968-b607-805fb7776e93-kube-api-access-dcnjf\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl65r5\" (UID: \"d9de42b9-3216-4968-b607-805fb7776e93\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl65r5"
Sep 17 16:52:22.185112 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:22.185083    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43a42869-aebc-4946-bc3a-70161575845f-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2grmx\" (UID: \"43a42869-aebc-4946-bc3a-70161575845f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2grmx"
Sep 17 16:52:22.185247 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:22.185138    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lgr4\" (UniqueName: \"kubernetes.io/projected/43a42869-aebc-4946-bc3a-70161575845f-kube-api-access-8lgr4\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2grmx\" (UID: \"43a42869-aebc-4946-bc3a-70161575845f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2grmx"
Sep 17 16:52:22.185247 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:22.185176    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43a42869-aebc-4946-bc3a-70161575845f-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2grmx\" (UID: \"43a42869-aebc-4946-bc3a-70161575845f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2grmx"
Sep 17 16:52:22.185549 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:22.185535    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43a42869-aebc-4946-bc3a-70161575845f-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2grmx\" (UID: \"43a42869-aebc-4946-bc3a-70161575845f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2grmx"
Sep 17 16:52:22.185597 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:22.185532    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43a42869-aebc-4946-bc3a-70161575845f-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2grmx\" (UID: \"43a42869-aebc-4946-bc3a-70161575845f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2grmx"
Sep 17 16:52:22.189058 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:22.189041    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl65r5"
Sep 17 16:52:22.195442 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:22.195415    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lgr4\" (UniqueName: \"kubernetes.io/projected/43a42869-aebc-4946-bc3a-70161575845f-kube-api-access-8lgr4\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2grmx\" (UID: \"43a42869-aebc-4946-bc3a-70161575845f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2grmx"
Sep 17 16:52:22.269136 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:22.269112    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6px74h"]
Sep 17 16:52:22.271160 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:52:22.271126    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe91cb91_4790_479f_adf7_dd96a854e3a4.slice/crio-ffc7bcc397a61f6e6c77bb2c9321c7e679dd04c3c6883ea7595b000cf5d5dd95 WatchSource:0}: Error finding container ffc7bcc397a61f6e6c77bb2c9321c7e679dd04c3c6883ea7595b000cf5d5dd95: Status 404 returned error can't find the container with id ffc7bcc397a61f6e6c77bb2c9321c7e679dd04c3c6883ea7595b000cf5d5dd95
Sep 17 16:52:22.321539 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:22.321518    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2grmx"
Sep 17 16:52:22.363715 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:22.363681    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsvrf"]
Sep 17 16:52:22.366246 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:52:22.366218    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72c3347f_4b65_49ab_8878_71374a5eecaa.slice/crio-ca33bfe398ba090fdcdeea540ecaa387c6dddbf17994ac7f9af09dd7b30d0c14 WatchSource:0}: Error finding container ca33bfe398ba090fdcdeea540ecaa387c6dddbf17994ac7f9af09dd7b30d0c14: Status 404 returned error can't find the container with id ca33bfe398ba090fdcdeea540ecaa387c6dddbf17994ac7f9af09dd7b30d0c14
Sep 17 16:52:22.460110 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:22.460083    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl65r5"]
Sep 17 16:52:22.461686 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:52:22.461660    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9de42b9_3216_4968_b607_805fb7776e93.slice/crio-96f7e3a8e38947b2fc0981864bc70739dc477aaa32544ffeaa609558e7496310 WatchSource:0}: Error finding container 96f7e3a8e38947b2fc0981864bc70739dc477aaa32544ffeaa609558e7496310: Status 404 returned error can't find the container with id 96f7e3a8e38947b2fc0981864bc70739dc477aaa32544ffeaa609558e7496310
Sep 17 16:52:22.605104 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:22.605075    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2grmx"]
Sep 17 16:52:22.614911 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:22.614889    2566 generic.go:356] "Generic (PLEG): container finished" podID="d9de42b9-3216-4968-b607-805fb7776e93" containerID="f18646bd7b74446924637c884ac0e6ed1be096204dde6408eb156fd8c0d2cfb5" exitCode=0
Sep 17 16:52:22.615033 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:22.614968    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl65r5" event={"ID":"d9de42b9-3216-4968-b607-805fb7776e93","Type":"ContainerDied","Data":"f18646bd7b74446924637c884ac0e6ed1be096204dde6408eb156fd8c0d2cfb5"}
Sep 17 16:52:22.615033 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:22.615007    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl65r5" event={"ID":"d9de42b9-3216-4968-b607-805fb7776e93","Type":"ContainerStarted","Data":"96f7e3a8e38947b2fc0981864bc70739dc477aaa32544ffeaa609558e7496310"}
Sep 17 16:52:22.619485 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:22.619456    2566 generic.go:356] "Generic (PLEG): container finished" podID="72c3347f-4b65-49ab-8878-71374a5eecaa" containerID="10982a1a74764970fd381237c985b9ec026e94c5ad921b4255249bedb052e5bb" exitCode=0
Sep 17 16:52:22.619614 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:22.619549    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsvrf" event={"ID":"72c3347f-4b65-49ab-8878-71374a5eecaa","Type":"ContainerDied","Data":"10982a1a74764970fd381237c985b9ec026e94c5ad921b4255249bedb052e5bb"}
Sep 17 16:52:22.619614 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:22.619594    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsvrf" event={"ID":"72c3347f-4b65-49ab-8878-71374a5eecaa","Type":"ContainerStarted","Data":"ca33bfe398ba090fdcdeea540ecaa387c6dddbf17994ac7f9af09dd7b30d0c14"}
Sep 17 16:52:22.621454 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:22.621411    2566 generic.go:356] "Generic (PLEG): container finished" podID="fe91cb91-4790-479f-adf7-dd96a854e3a4" containerID="b83c54eb1daf14499304574c258d5ab81e2fffb5a0a9e827dc74f55b559e4814" exitCode=0
Sep 17 16:52:22.621518 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:22.621466    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6px74h" event={"ID":"fe91cb91-4790-479f-adf7-dd96a854e3a4","Type":"ContainerDied","Data":"b83c54eb1daf14499304574c258d5ab81e2fffb5a0a9e827dc74f55b559e4814"}
Sep 17 16:52:22.621567 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:22.621551    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6px74h" event={"ID":"fe91cb91-4790-479f-adf7-dd96a854e3a4","Type":"ContainerStarted","Data":"ffc7bcc397a61f6e6c77bb2c9321c7e679dd04c3c6883ea7595b000cf5d5dd95"}
Sep 17 16:52:22.632450 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:52:22.632412    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43a42869_aebc_4946_bc3a_70161575845f.slice/crio-df4a3b53abaee5e64129327c2b217bab5852c4dd6eab557aad0f3f6633b803e1 WatchSource:0}: Error finding container df4a3b53abaee5e64129327c2b217bab5852c4dd6eab557aad0f3f6633b803e1: Status 404 returned error can't find the container with id df4a3b53abaee5e64129327c2b217bab5852c4dd6eab557aad0f3f6633b803e1
Sep 17 16:52:23.629785 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:23.629752    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6px74h" event={"ID":"fe91cb91-4790-479f-adf7-dd96a854e3a4","Type":"ContainerStarted","Data":"484e165a2b05a37bdccdf14f6364c4d297a2b1d8cebac2cea5922f6cb7b055d1"}
Sep 17 16:52:23.631757 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:23.631734    2566 generic.go:356] "Generic (PLEG): container finished" podID="d9de42b9-3216-4968-b607-805fb7776e93" containerID="fa6a13b28f6f7b3d9fd992bba3252ab8a609206dcee5594ef45960d162905c2f" exitCode=0
Sep 17 16:52:23.631852 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:23.631812    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl65r5" event={"ID":"d9de42b9-3216-4968-b607-805fb7776e93","Type":"ContainerDied","Data":"fa6a13b28f6f7b3d9fd992bba3252ab8a609206dcee5594ef45960d162905c2f"}
Sep 17 16:52:23.633664 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:23.633631    2566 generic.go:356] "Generic (PLEG): container finished" podID="43a42869-aebc-4946-bc3a-70161575845f" containerID="7eff234cd1145a1e1cfa135d303455e9f2057733d0d76ccefc41a0e2ae7bbb42" exitCode=0
Sep 17 16:52:23.633762 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:23.633661    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2grmx" event={"ID":"43a42869-aebc-4946-bc3a-70161575845f","Type":"ContainerDied","Data":"7eff234cd1145a1e1cfa135d303455e9f2057733d0d76ccefc41a0e2ae7bbb42"}
Sep 17 16:52:23.633762 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:23.633707    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2grmx" event={"ID":"43a42869-aebc-4946-bc3a-70161575845f","Type":"ContainerStarted","Data":"df4a3b53abaee5e64129327c2b217bab5852c4dd6eab557aad0f3f6633b803e1"}
Sep 17 16:52:24.640447 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:24.640392    2566 generic.go:356] "Generic (PLEG): container finished" podID="d9de42b9-3216-4968-b607-805fb7776e93" containerID="3af30a85a564c461bf82196dd04ffcbe15b00c6ee852ba1b22b41245f908ef1e" exitCode=0
Sep 17 16:52:24.640868 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:24.640478    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl65r5" event={"ID":"d9de42b9-3216-4968-b607-805fb7776e93","Type":"ContainerDied","Data":"3af30a85a564c461bf82196dd04ffcbe15b00c6ee852ba1b22b41245f908ef1e"}
Sep 17 16:52:24.642504 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:24.642480    2566 generic.go:356] "Generic (PLEG): container finished" podID="72c3347f-4b65-49ab-8878-71374a5eecaa" containerID="13df8cefa1c7348c52f7ba8c26489ca3d164d013fc9e6de922c8357018228059" exitCode=0
Sep 17 16:52:24.642600 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:24.642559    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsvrf" event={"ID":"72c3347f-4b65-49ab-8878-71374a5eecaa","Type":"ContainerDied","Data":"13df8cefa1c7348c52f7ba8c26489ca3d164d013fc9e6de922c8357018228059"}
Sep 17 16:52:24.644603 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:24.644581    2566 generic.go:356] "Generic (PLEG): container finished" podID="fe91cb91-4790-479f-adf7-dd96a854e3a4" containerID="484e165a2b05a37bdccdf14f6364c4d297a2b1d8cebac2cea5922f6cb7b055d1" exitCode=0
Sep 17 16:52:24.644737 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:24.644644    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6px74h" event={"ID":"fe91cb91-4790-479f-adf7-dd96a854e3a4","Type":"ContainerDied","Data":"484e165a2b05a37bdccdf14f6364c4d297a2b1d8cebac2cea5922f6cb7b055d1"}
Sep 17 16:52:25.651564 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:25.651534    2566 generic.go:356] "Generic (PLEG): container finished" podID="fe91cb91-4790-479f-adf7-dd96a854e3a4" containerID="ccc57e972de9a9960b3820d09f99c03ee8c5a8c17a3d6db11b1f674a50989e29" exitCode=0
Sep 17 16:52:25.652012 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:25.651620    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6px74h" event={"ID":"fe91cb91-4790-479f-adf7-dd96a854e3a4","Type":"ContainerDied","Data":"ccc57e972de9a9960b3820d09f99c03ee8c5a8c17a3d6db11b1f674a50989e29"}
Sep 17 16:52:25.653478 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:25.653425    2566 generic.go:356] "Generic (PLEG): container finished" podID="43a42869-aebc-4946-bc3a-70161575845f" containerID="b7c8b76cc285f1820ea76586f50c3c85da5db99308cf7e3256a322c66e852353" exitCode=0
Sep 17 16:52:25.653478 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:25.653466    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2grmx" event={"ID":"43a42869-aebc-4946-bc3a-70161575845f","Type":"ContainerDied","Data":"b7c8b76cc285f1820ea76586f50c3c85da5db99308cf7e3256a322c66e852353"}
Sep 17 16:52:25.655741 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:25.655718    2566 generic.go:356] "Generic (PLEG): container finished" podID="72c3347f-4b65-49ab-8878-71374a5eecaa" containerID="2faf446bc8cbf53f733915bf13f5280b60eb4da21f32475f6cd327ee5595db09" exitCode=0
Sep 17 16:52:25.655988 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:25.655800    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsvrf" event={"ID":"72c3347f-4b65-49ab-8878-71374a5eecaa","Type":"ContainerDied","Data":"2faf446bc8cbf53f733915bf13f5280b60eb4da21f32475f6cd327ee5595db09"}
Sep 17 16:52:25.788523 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:25.788496    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl65r5"
Sep 17 16:52:25.812430 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:25.812380    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcnjf\" (UniqueName: \"kubernetes.io/projected/d9de42b9-3216-4968-b607-805fb7776e93-kube-api-access-dcnjf\") pod \"d9de42b9-3216-4968-b607-805fb7776e93\" (UID: \"d9de42b9-3216-4968-b607-805fb7776e93\") "
Sep 17 16:52:25.812660 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:25.812484    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9de42b9-3216-4968-b607-805fb7776e93-bundle\") pod \"d9de42b9-3216-4968-b607-805fb7776e93\" (UID: \"d9de42b9-3216-4968-b607-805fb7776e93\") "
Sep 17 16:52:25.812660 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:25.812532    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9de42b9-3216-4968-b607-805fb7776e93-util\") pod \"d9de42b9-3216-4968-b607-805fb7776e93\" (UID: \"d9de42b9-3216-4968-b607-805fb7776e93\") "
Sep 17 16:52:25.813030 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:25.813000    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9de42b9-3216-4968-b607-805fb7776e93-bundle" (OuterVolumeSpecName: "bundle") pod "d9de42b9-3216-4968-b607-805fb7776e93" (UID: "d9de42b9-3216-4968-b607-805fb7776e93"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:52:25.815516 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:25.815483    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9de42b9-3216-4968-b607-805fb7776e93-kube-api-access-dcnjf" (OuterVolumeSpecName: "kube-api-access-dcnjf") pod "d9de42b9-3216-4968-b607-805fb7776e93" (UID: "d9de42b9-3216-4968-b607-805fb7776e93"). InnerVolumeSpecName "kube-api-access-dcnjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:52:25.819599 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:25.819370    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9de42b9-3216-4968-b607-805fb7776e93-util" (OuterVolumeSpecName: "util") pod "d9de42b9-3216-4968-b607-805fb7776e93" (UID: "d9de42b9-3216-4968-b607-805fb7776e93"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:52:25.913585 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:25.913562    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcnjf\" (UniqueName: \"kubernetes.io/projected/d9de42b9-3216-4968-b607-805fb7776e93-kube-api-access-dcnjf\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:52:25.913585 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:25.913585    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9de42b9-3216-4968-b607-805fb7776e93-bundle\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:52:25.913726 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:25.913596    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9de42b9-3216-4968-b607-805fb7776e93-util\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:52:26.662107 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:26.662019    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl65r5" event={"ID":"d9de42b9-3216-4968-b607-805fb7776e93","Type":"ContainerDied","Data":"96f7e3a8e38947b2fc0981864bc70739dc477aaa32544ffeaa609558e7496310"}
Sep 17 16:52:26.662107 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:26.662072    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96f7e3a8e38947b2fc0981864bc70739dc477aaa32544ffeaa609558e7496310"
Sep 17 16:52:26.662107 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:26.662031    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl65r5"
Sep 17 16:52:26.664512 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:26.664488    2566 generic.go:356] "Generic (PLEG): container finished" podID="43a42869-aebc-4946-bc3a-70161575845f" containerID="a8a4ce21377bb3450b84592ab146c8251e6e88e0cb64aa2f2253568c33cd2a5a" exitCode=0
Sep 17 16:52:26.664632 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:26.664517    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2grmx" event={"ID":"43a42869-aebc-4946-bc3a-70161575845f","Type":"ContainerDied","Data":"a8a4ce21377bb3450b84592ab146c8251e6e88e0cb64aa2f2253568c33cd2a5a"}
Sep 17 16:52:26.813959 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:26.813933    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsvrf"
Sep 17 16:52:26.816642 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:26.816624    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6px74h"
Sep 17 16:52:26.920925 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:26.920842    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfv8d\" (UniqueName: \"kubernetes.io/projected/fe91cb91-4790-479f-adf7-dd96a854e3a4-kube-api-access-jfv8d\") pod \"fe91cb91-4790-479f-adf7-dd96a854e3a4\" (UID: \"fe91cb91-4790-479f-adf7-dd96a854e3a4\") "
Sep 17 16:52:26.920925 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:26.920880    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/72c3347f-4b65-49ab-8878-71374a5eecaa-util\") pod \"72c3347f-4b65-49ab-8878-71374a5eecaa\" (UID: \"72c3347f-4b65-49ab-8878-71374a5eecaa\") "
Sep 17 16:52:26.920925 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:26.920909    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/72c3347f-4b65-49ab-8878-71374a5eecaa-bundle\") pod \"72c3347f-4b65-49ab-8878-71374a5eecaa\" (UID: \"72c3347f-4b65-49ab-8878-71374a5eecaa\") "
Sep 17 16:52:26.921169 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:26.920949    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe91cb91-4790-479f-adf7-dd96a854e3a4-bundle\") pod \"fe91cb91-4790-479f-adf7-dd96a854e3a4\" (UID: \"fe91cb91-4790-479f-adf7-dd96a854e3a4\") "
Sep 17 16:52:26.921169 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:26.920972    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlbb9\" (UniqueName: \"kubernetes.io/projected/72c3347f-4b65-49ab-8878-71374a5eecaa-kube-api-access-qlbb9\") pod \"72c3347f-4b65-49ab-8878-71374a5eecaa\" (UID: \"72c3347f-4b65-49ab-8878-71374a5eecaa\") "
Sep 17 16:52:26.921169 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:26.921026    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe91cb91-4790-479f-adf7-dd96a854e3a4-util\") pod \"fe91cb91-4790-479f-adf7-dd96a854e3a4\" (UID: \"fe91cb91-4790-479f-adf7-dd96a854e3a4\") "
Sep 17 16:52:26.921565 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:26.921531    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72c3347f-4b65-49ab-8878-71374a5eecaa-bundle" (OuterVolumeSpecName: "bundle") pod "72c3347f-4b65-49ab-8878-71374a5eecaa" (UID: "72c3347f-4b65-49ab-8878-71374a5eecaa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:52:26.921565 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:26.921539    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe91cb91-4790-479f-adf7-dd96a854e3a4-bundle" (OuterVolumeSpecName: "bundle") pod "fe91cb91-4790-479f-adf7-dd96a854e3a4" (UID: "fe91cb91-4790-479f-adf7-dd96a854e3a4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:52:26.923164 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:26.923134    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe91cb91-4790-479f-adf7-dd96a854e3a4-kube-api-access-jfv8d" (OuterVolumeSpecName: "kube-api-access-jfv8d") pod "fe91cb91-4790-479f-adf7-dd96a854e3a4" (UID: "fe91cb91-4790-479f-adf7-dd96a854e3a4"). InnerVolumeSpecName "kube-api-access-jfv8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:52:26.923262 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:26.923232    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72c3347f-4b65-49ab-8878-71374a5eecaa-kube-api-access-qlbb9" (OuterVolumeSpecName: "kube-api-access-qlbb9") pod "72c3347f-4b65-49ab-8878-71374a5eecaa" (UID: "72c3347f-4b65-49ab-8878-71374a5eecaa"). InnerVolumeSpecName "kube-api-access-qlbb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:52:26.926411 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:26.926370    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72c3347f-4b65-49ab-8878-71374a5eecaa-util" (OuterVolumeSpecName: "util") pod "72c3347f-4b65-49ab-8878-71374a5eecaa" (UID: "72c3347f-4b65-49ab-8878-71374a5eecaa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:52:26.927108 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:26.926936    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe91cb91-4790-479f-adf7-dd96a854e3a4-util" (OuterVolumeSpecName: "util") pod "fe91cb91-4790-479f-adf7-dd96a854e3a4" (UID: "fe91cb91-4790-479f-adf7-dd96a854e3a4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:52:27.022240 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:27.022197    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfv8d\" (UniqueName: \"kubernetes.io/projected/fe91cb91-4790-479f-adf7-dd96a854e3a4-kube-api-access-jfv8d\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:52:27.022240 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:27.022236    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/72c3347f-4b65-49ab-8878-71374a5eecaa-util\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:52:27.022240 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:27.022246    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/72c3347f-4b65-49ab-8878-71374a5eecaa-bundle\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:52:27.022240 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:27.022254    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe91cb91-4790-479f-adf7-dd96a854e3a4-bundle\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:52:27.022520 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:27.022262    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlbb9\" (UniqueName: \"kubernetes.io/projected/72c3347f-4b65-49ab-8878-71374a5eecaa-kube-api-access-qlbb9\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:52:27.022520 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:27.022270    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe91cb91-4790-479f-adf7-dd96a854e3a4-util\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:52:27.670985 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:27.670951    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsvrf"
Sep 17 16:52:27.670985 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:27.670965    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsvrf" event={"ID":"72c3347f-4b65-49ab-8878-71374a5eecaa","Type":"ContainerDied","Data":"ca33bfe398ba090fdcdeea540ecaa387c6dddbf17994ac7f9af09dd7b30d0c14"}
Sep 17 16:52:27.671409 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:27.671009    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca33bfe398ba090fdcdeea540ecaa387c6dddbf17994ac7f9af09dd7b30d0c14"
Sep 17 16:52:27.672911 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:27.672886    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6px74h" event={"ID":"fe91cb91-4790-479f-adf7-dd96a854e3a4","Type":"ContainerDied","Data":"ffc7bcc397a61f6e6c77bb2c9321c7e679dd04c3c6883ea7595b000cf5d5dd95"}
Sep 17 16:52:27.673020 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:27.672931    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffc7bcc397a61f6e6c77bb2c9321c7e679dd04c3c6883ea7595b000cf5d5dd95"
Sep 17 16:52:27.673079 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:27.673064    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6px74h"
Sep 17 16:52:27.795076 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:27.795048    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2grmx"
Sep 17 16:52:27.828512 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:27.828485    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43a42869-aebc-4946-bc3a-70161575845f-bundle\") pod \"43a42869-aebc-4946-bc3a-70161575845f\" (UID: \"43a42869-aebc-4946-bc3a-70161575845f\") "
Sep 17 16:52:27.828633 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:27.828526    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43a42869-aebc-4946-bc3a-70161575845f-util\") pod \"43a42869-aebc-4946-bc3a-70161575845f\" (UID: \"43a42869-aebc-4946-bc3a-70161575845f\") "
Sep 17 16:52:27.828633 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:27.828575    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lgr4\" (UniqueName: \"kubernetes.io/projected/43a42869-aebc-4946-bc3a-70161575845f-kube-api-access-8lgr4\") pod \"43a42869-aebc-4946-bc3a-70161575845f\" (UID: \"43a42869-aebc-4946-bc3a-70161575845f\") "
Sep 17 16:52:27.829313 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:27.829285    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a42869-aebc-4946-bc3a-70161575845f-bundle" (OuterVolumeSpecName: "bundle") pod "43a42869-aebc-4946-bc3a-70161575845f" (UID: "43a42869-aebc-4946-bc3a-70161575845f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:52:27.830859 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:27.830836    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43a42869-aebc-4946-bc3a-70161575845f-kube-api-access-8lgr4" (OuterVolumeSpecName: "kube-api-access-8lgr4") pod "43a42869-aebc-4946-bc3a-70161575845f" (UID: "43a42869-aebc-4946-bc3a-70161575845f"). InnerVolumeSpecName "kube-api-access-8lgr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:52:27.833937 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:27.833912    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a42869-aebc-4946-bc3a-70161575845f-util" (OuterVolumeSpecName: "util") pod "43a42869-aebc-4946-bc3a-70161575845f" (UID: "43a42869-aebc-4946-bc3a-70161575845f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:52:27.929752 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:27.929670    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43a42869-aebc-4946-bc3a-70161575845f-bundle\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:52:27.929752 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:27.929705    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43a42869-aebc-4946-bc3a-70161575845f-util\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:52:27.929752 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:27.929715    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lgr4\" (UniqueName: \"kubernetes.io/projected/43a42869-aebc-4946-bc3a-70161575845f-kube-api-access-8lgr4\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:52:28.678783 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:28.678710    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2grmx"
Sep 17 16:52:28.678783 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:28.678737    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2grmx" event={"ID":"43a42869-aebc-4946-bc3a-70161575845f","Type":"ContainerDied","Data":"df4a3b53abaee5e64129327c2b217bab5852c4dd6eab557aad0f3f6633b803e1"}
Sep 17 16:52:28.678783 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:28.678780    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df4a3b53abaee5e64129327c2b217bab5852c4dd6eab557aad0f3f6633b803e1"
Sep 17 16:52:34.502934 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:34.502896    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-5bhzb"]
Sep 17 16:52:34.505082 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:34.503208    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe91cb91-4790-479f-adf7-dd96a854e3a4" containerName="extract"
Sep 17 16:52:34.505082 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:34.503223    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe91cb91-4790-479f-adf7-dd96a854e3a4" containerName="extract"
Sep 17 16:52:34.505082 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:34.503232    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43a42869-aebc-4946-bc3a-70161575845f" containerName="util"
Sep 17 16:52:34.505082 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:34.503238    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="43a42869-aebc-4946-bc3a-70161575845f" containerName="util"
Sep 17 16:52:34.505082 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:34.503244    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9de42b9-3216-4968-b607-805fb7776e93" containerName="extract"
Sep 17 16:52:34.505082 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:34.503249    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9de42b9-3216-4968-b607-805fb7776e93" containerName="extract"
Sep 17 16:52:34.505082 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:34.503259    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72c3347f-4b65-49ab-8878-71374a5eecaa" containerName="util"
Sep 17 16:52:34.505082 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:34.503263    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="72c3347f-4b65-49ab-8878-71374a5eecaa" containerName="util"
Sep 17 16:52:34.505082 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:34.503269    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9de42b9-3216-4968-b607-805fb7776e93" containerName="pull"
Sep 17 16:52:34.505082 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:34.503274    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9de42b9-3216-4968-b607-805fb7776e93" containerName="pull"
Sep 17 16:52:34.505082 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:34.503279    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72c3347f-4b65-49ab-8878-71374a5eecaa" containerName="pull"
Sep 17 16:52:34.505082 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:34.503284    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="72c3347f-4b65-49ab-8878-71374a5eecaa" containerName="pull"
Sep 17 16:52:34.505082 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:34.503294    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72c3347f-4b65-49ab-8878-71374a5eecaa" containerName="extract"
Sep 17 16:52:34.505082 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:34.503299    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="72c3347f-4b65-49ab-8878-71374a5eecaa" containerName="extract"
Sep 17 16:52:34.505082 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:34.503307    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43a42869-aebc-4946-bc3a-70161575845f" containerName="extract"
Sep 17 16:52:34.505082 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:34.503312    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="43a42869-aebc-4946-bc3a-70161575845f" containerName="extract"
Sep 17 16:52:34.505082 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:34.503319    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe91cb91-4790-479f-adf7-dd96a854e3a4" containerName="util"
Sep 17 16:52:34.505082 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:34.503324    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe91cb91-4790-479f-adf7-dd96a854e3a4" containerName="util"
Sep 17 16:52:34.505082 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:34.503329    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43a42869-aebc-4946-bc3a-70161575845f" containerName="pull"
Sep 17 16:52:34.505082 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:34.503334    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="43a42869-aebc-4946-bc3a-70161575845f" containerName="pull"
Sep 17 16:52:34.505082 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:34.503342    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe91cb91-4790-479f-adf7-dd96a854e3a4" containerName="pull"
Sep 17 16:52:34.505082 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:34.503346    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe91cb91-4790-479f-adf7-dd96a854e3a4" containerName="pull"
Sep 17 16:52:34.505082 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:34.503351    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9de42b9-3216-4968-b607-805fb7776e93" containerName="util"
Sep 17 16:52:34.505082 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:34.503356    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9de42b9-3216-4968-b607-805fb7776e93" containerName="util"
Sep 17 16:52:34.505082 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:34.503393    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9de42b9-3216-4968-b607-805fb7776e93" containerName="extract"
Sep 17 16:52:34.505082 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:34.503402    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="72c3347f-4b65-49ab-8878-71374a5eecaa" containerName="extract"
Sep 17 16:52:34.505082 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:34.503408    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="43a42869-aebc-4946-bc3a-70161575845f" containerName="extract"
Sep 17 16:52:34.505082 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:34.503414    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe91cb91-4790-479f-adf7-dd96a854e3a4" containerName="extract"
Sep 17 16:52:34.505979 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:34.505965    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-5bhzb"
Sep 17 16:52:34.512245 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:34.512027    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 16:52:34.512834 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:34.512819    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 16:52:34.512899 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:34.512839    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-nlh8v\""
Sep 17 16:52:34.523894 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:34.523867    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-5bhzb"]
Sep 17 16:52:34.574349 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:34.574315    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwcjg\" (UniqueName: \"kubernetes.io/projected/d7125f90-b9d9-48ba-a300-1678738e5ee8-kube-api-access-hwcjg\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-5bhzb\" (UID: \"d7125f90-b9d9-48ba-a300-1678738e5ee8\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-5bhzb"
Sep 17 16:52:34.674696 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:34.674663    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwcjg\" (UniqueName: \"kubernetes.io/projected/d7125f90-b9d9-48ba-a300-1678738e5ee8-kube-api-access-hwcjg\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-5bhzb\" (UID: \"d7125f90-b9d9-48ba-a300-1678738e5ee8\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-5bhzb"
Sep 17 16:52:34.690067 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:34.690035    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwcjg\" (UniqueName: \"kubernetes.io/projected/d7125f90-b9d9-48ba-a300-1678738e5ee8-kube-api-access-hwcjg\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-5bhzb\" (UID: \"d7125f90-b9d9-48ba-a300-1678738e5ee8\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-5bhzb"
Sep 17 16:52:34.815606 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:34.815577    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-5bhzb"
Sep 17 16:52:35.095384 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:35.095351    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-5bhzb"]
Sep 17 16:52:35.097846 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:52:35.097819    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7125f90_b9d9_48ba_a300_1678738e5ee8.slice/crio-02d0e36264598ca1be315121a844e646012711ef9dbd92407792a5b0b3e1a264 WatchSource:0}: Error finding container 02d0e36264598ca1be315121a844e646012711ef9dbd92407792a5b0b3e1a264: Status 404 returned error can't find the container with id 02d0e36264598ca1be315121a844e646012711ef9dbd92407792a5b0b3e1a264
Sep 17 16:52:35.710030 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:35.709996    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-5bhzb" event={"ID":"d7125f90-b9d9-48ba-a300-1678738e5ee8","Type":"ContainerStarted","Data":"02d0e36264598ca1be315121a844e646012711ef9dbd92407792a5b0b3e1a264"}
Sep 17 16:52:36.716214 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:36.716182    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-5bhzb" event={"ID":"d7125f90-b9d9-48ba-a300-1678738e5ee8","Type":"ContainerStarted","Data":"3f4b66c913d468bd0573a8e779cea593fedb5576b0b5d2a94841ea77df62b0dc"}
Sep 17 16:52:36.716691 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:36.716316    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-5bhzb"
Sep 17 16:52:36.747770 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:36.747711    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-5bhzb" podStartSLOduration=1.261551737 podStartE2EDuration="2.747696413s" totalImagesPullingTime="1.486144676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:52:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:52:36.7460254 +0000 UTC m=+642.640504538" watchObservedRunningTime="2026-09-17 16:52:36.747696413 +0000 UTC m=+642.642175577"
Sep 17 16:52:39.593832 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:39.593801    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-qnzll"]
Sep 17 16:52:39.603402 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:39.603380    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-qnzll"
Sep 17 16:52:39.610837 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:39.610818    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-lqcwz\""
Sep 17 16:52:39.626614 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:39.626590    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-qnzll"]
Sep 17 16:52:39.717740 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:39.717707    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h7hh\" (UniqueName: \"kubernetes.io/projected/fbc29aee-02c6-4de1-ba3f-69063895858d-kube-api-access-7h7hh\") pod \"authorino-operator-6fc8f864f4-qnzll\" (UID: \"fbc29aee-02c6-4de1-ba3f-69063895858d\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-qnzll"
Sep 17 16:52:39.818643 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:39.818612    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h7hh\" (UniqueName: \"kubernetes.io/projected/fbc29aee-02c6-4de1-ba3f-69063895858d-kube-api-access-7h7hh\") pod \"authorino-operator-6fc8f864f4-qnzll\" (UID: \"fbc29aee-02c6-4de1-ba3f-69063895858d\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-qnzll"
Sep 17 16:52:39.835906 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:39.835882    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h7hh\" (UniqueName: \"kubernetes.io/projected/fbc29aee-02c6-4de1-ba3f-69063895858d-kube-api-access-7h7hh\") pod \"authorino-operator-6fc8f864f4-qnzll\" (UID: \"fbc29aee-02c6-4de1-ba3f-69063895858d\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-qnzll"
Sep 17 16:52:39.912825 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:39.912746    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-qnzll"
Sep 17 16:52:40.192409 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:40.192386    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-qnzll"]
Sep 17 16:52:40.194008 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:52:40.193977    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbc29aee_02c6_4de1_ba3f_69063895858d.slice/crio-fbdbd3a9f3e4f7ad33d3d34721c7f927428497562460e618c2b98c61e9b5f642 WatchSource:0}: Error finding container fbdbd3a9f3e4f7ad33d3d34721c7f927428497562460e618c2b98c61e9b5f642: Status 404 returned error can't find the container with id fbdbd3a9f3e4f7ad33d3d34721c7f927428497562460e618c2b98c61e9b5f642
Sep 17 16:52:40.736480 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:40.736421    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-qnzll" event={"ID":"fbc29aee-02c6-4de1-ba3f-69063895858d","Type":"ContainerStarted","Data":"fbdbd3a9f3e4f7ad33d3d34721c7f927428497562460e618c2b98c61e9b5f642"}
Sep 17 16:52:47.723575 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:47.723539    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-5bhzb"
Sep 17 16:52:48.775358 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:48.775319    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-qnzll" event={"ID":"fbc29aee-02c6-4de1-ba3f-69063895858d","Type":"ContainerStarted","Data":"3083d7bd1457744fec9c117e8a49bc4364780ddebadb47f91dbdf7b2eb19a43b"}
Sep 17 16:52:48.775823 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:48.775391    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-qnzll"
Sep 17 16:52:48.821028 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:48.820977    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-6fc8f864f4-qnzll" podStartSLOduration=2.17828325 podStartE2EDuration="9.820961825s" totalImagesPullingTime="7.642678575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:52:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:52:48.819960991 +0000 UTC m=+654.714440129" watchObservedRunningTime="2026-09-17 16:52:48.820961825 +0000 UTC m=+654.715440964"
Sep 17 16:52:59.781495 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:52:59.781459    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-qnzll"
Sep 17 16:53:31.778755 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:53:31.778714    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sh727"]
Sep 17 16:53:31.783472 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:53:31.783423    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-sh727"
Sep 17 16:53:31.785569 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:53:31.785548    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 17 16:53:31.785825 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:53:31.785811    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-n9bzr\""
Sep 17 16:53:31.795088 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:53:31.795065    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sh727"]
Sep 17 16:53:31.801537 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:53:31.801512    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c18cf2f8-66d1-4492-bf48-0defd23dc1c7-config-file\") pod \"limitador-limitador-84994f7ffc-sh727\" (UID: \"c18cf2f8-66d1-4492-bf48-0defd23dc1c7\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-sh727"
Sep 17 16:53:31.801634 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:53:31.801612    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbzmp\" (UniqueName: \"kubernetes.io/projected/c18cf2f8-66d1-4492-bf48-0defd23dc1c7-kube-api-access-rbzmp\") pod \"limitador-limitador-84994f7ffc-sh727\" (UID: \"c18cf2f8-66d1-4492-bf48-0defd23dc1c7\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-sh727"
Sep 17 16:53:31.837092 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:53:31.837067    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sh727"]
Sep 17 16:53:31.902611 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:53:31.902580    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c18cf2f8-66d1-4492-bf48-0defd23dc1c7-config-file\") pod \"limitador-limitador-84994f7ffc-sh727\" (UID: \"c18cf2f8-66d1-4492-bf48-0defd23dc1c7\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-sh727"
Sep 17 16:53:31.902750 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:53:31.902647    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbzmp\" (UniqueName: \"kubernetes.io/projected/c18cf2f8-66d1-4492-bf48-0defd23dc1c7-kube-api-access-rbzmp\") pod \"limitador-limitador-84994f7ffc-sh727\" (UID: \"c18cf2f8-66d1-4492-bf48-0defd23dc1c7\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-sh727"
Sep 17 16:53:31.903211 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:53:31.903187    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c18cf2f8-66d1-4492-bf48-0defd23dc1c7-config-file\") pod \"limitador-limitador-84994f7ffc-sh727\" (UID: \"c18cf2f8-66d1-4492-bf48-0defd23dc1c7\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-sh727"
Sep 17 16:53:31.912371 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:53:31.912342    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbzmp\" (UniqueName: \"kubernetes.io/projected/c18cf2f8-66d1-4492-bf48-0defd23dc1c7-kube-api-access-rbzmp\") pod \"limitador-limitador-84994f7ffc-sh727\" (UID: \"c18cf2f8-66d1-4492-bf48-0defd23dc1c7\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-sh727"
Sep 17 16:53:32.093398 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:53:32.093367    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-sh727"
Sep 17 16:53:32.377344 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:53:32.377150    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sh727"]
Sep 17 16:53:32.379799 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:53:32.379775    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc18cf2f8_66d1_4492_bf48_0defd23dc1c7.slice/crio-fb91d6e6db47973f4eaac349d85cb9f0d6ffb220989672fb20fc269aed0447b1 WatchSource:0}: Error finding container fb91d6e6db47973f4eaac349d85cb9f0d6ffb220989672fb20fc269aed0447b1: Status 404 returned error can't find the container with id fb91d6e6db47973f4eaac349d85cb9f0d6ffb220989672fb20fc269aed0447b1
Sep 17 16:53:32.985741 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:53:32.985708    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-sh727" event={"ID":"c18cf2f8-66d1-4492-bf48-0defd23dc1c7","Type":"ContainerStarted","Data":"fb91d6e6db47973f4eaac349d85cb9f0d6ffb220989672fb20fc269aed0447b1"}
Sep 17 16:53:33.991789 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:53:33.991759    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-sh727" event={"ID":"c18cf2f8-66d1-4492-bf48-0defd23dc1c7","Type":"ContainerStarted","Data":"9eb146a7acb258b36656d486b58f7ace180627adfda9dd4f086486d7277d17b2"}
Sep 17 16:53:33.992174 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:53:33.991911    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-sh727"
Sep 17 16:53:44.997174 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:53:44.997093    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-sh727"
Sep 17 16:53:45.025302 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:53:45.025253    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-sh727" podStartSLOduration=13.001411798 podStartE2EDuration="14.025237193s" totalImagesPullingTime="1.023825395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:53:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:53:34.011241967 +0000 UTC m=+699.905721107" watchObservedRunningTime="2026-09-17 16:53:45.025237193 +0000 UTC m=+710.919716332"
Sep 17 16:54:19.869665 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:19.869626    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w4b7c"]
Sep 17 16:54:19.870193 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:19.869859    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w4b7c" podUID="28ebe838-968c-4986-b80d-f5654eb1e2c7" containerName="discovery" containerID="cri-o://572f63259d108bc7b99f8bbc9960101f2de580cb0d3889d56ddb33bf42a4244a" gracePeriod=30
Sep 17 16:54:20.115009 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:20.114984    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w4b7c"
Sep 17 16:54:20.157035 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:20.156970    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/28ebe838-968c-4986-b80d-f5654eb1e2c7-istio-csr-dns-cert\") pod \"28ebe838-968c-4986-b80d-f5654eb1e2c7\" (UID: \"28ebe838-968c-4986-b80d-f5654eb1e2c7\") "
Sep 17 16:54:20.157035 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:20.157005    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfmrj\" (UniqueName: \"kubernetes.io/projected/28ebe838-968c-4986-b80d-f5654eb1e2c7-kube-api-access-kfmrj\") pod \"28ebe838-968c-4986-b80d-f5654eb1e2c7\" (UID: \"28ebe838-968c-4986-b80d-f5654eb1e2c7\") "
Sep 17 16:54:20.157035 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:20.157026    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/28ebe838-968c-4986-b80d-f5654eb1e2c7-istio-kubeconfig\") pod \"28ebe838-968c-4986-b80d-f5654eb1e2c7\" (UID: \"28ebe838-968c-4986-b80d-f5654eb1e2c7\") "
Sep 17 16:54:20.157291 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:20.157066    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/28ebe838-968c-4986-b80d-f5654eb1e2c7-istio-csr-ca-configmap\") pod \"28ebe838-968c-4986-b80d-f5654eb1e2c7\" (UID: \"28ebe838-968c-4986-b80d-f5654eb1e2c7\") "
Sep 17 16:54:20.157291 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:20.157088    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/28ebe838-968c-4986-b80d-f5654eb1e2c7-cacerts\") pod \"28ebe838-968c-4986-b80d-f5654eb1e2c7\" (UID: \"28ebe838-968c-4986-b80d-f5654eb1e2c7\") "
Sep 17 16:54:20.157291 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:20.157113    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/28ebe838-968c-4986-b80d-f5654eb1e2c7-local-certs\") pod \"28ebe838-968c-4986-b80d-f5654eb1e2c7\" (UID: \"28ebe838-968c-4986-b80d-f5654eb1e2c7\") "
Sep 17 16:54:20.157291 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:20.157147    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28ebe838-968c-4986-b80d-f5654eb1e2c7-istio-token\") pod \"28ebe838-968c-4986-b80d-f5654eb1e2c7\" (UID: \"28ebe838-968c-4986-b80d-f5654eb1e2c7\") "
Sep 17 16:54:20.157534 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:20.157421    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28ebe838-968c-4986-b80d-f5654eb1e2c7-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "28ebe838-968c-4986-b80d-f5654eb1e2c7" (UID: "28ebe838-968c-4986-b80d-f5654eb1e2c7"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:54:20.159708 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:20.159674    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28ebe838-968c-4986-b80d-f5654eb1e2c7-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "28ebe838-968c-4986-b80d-f5654eb1e2c7" (UID: "28ebe838-968c-4986-b80d-f5654eb1e2c7"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:54:20.159826 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:20.159726    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28ebe838-968c-4986-b80d-f5654eb1e2c7-kube-api-access-kfmrj" (OuterVolumeSpecName: "kube-api-access-kfmrj") pod "28ebe838-968c-4986-b80d-f5654eb1e2c7" (UID: "28ebe838-968c-4986-b80d-f5654eb1e2c7"). InnerVolumeSpecName "kube-api-access-kfmrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:54:20.159936 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:20.159888    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28ebe838-968c-4986-b80d-f5654eb1e2c7-istio-token" (OuterVolumeSpecName: "istio-token") pod "28ebe838-968c-4986-b80d-f5654eb1e2c7" (UID: "28ebe838-968c-4986-b80d-f5654eb1e2c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:54:20.159936 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:20.159888    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28ebe838-968c-4986-b80d-f5654eb1e2c7-cacerts" (OuterVolumeSpecName: "cacerts") pod "28ebe838-968c-4986-b80d-f5654eb1e2c7" (UID: "28ebe838-968c-4986-b80d-f5654eb1e2c7"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:54:20.160049 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:20.159964    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28ebe838-968c-4986-b80d-f5654eb1e2c7-local-certs" (OuterVolumeSpecName: "local-certs") pod "28ebe838-968c-4986-b80d-f5654eb1e2c7" (UID: "28ebe838-968c-4986-b80d-f5654eb1e2c7"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:54:20.160190 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:20.160172    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28ebe838-968c-4986-b80d-f5654eb1e2c7-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "28ebe838-968c-4986-b80d-f5654eb1e2c7" (UID: "28ebe838-968c-4986-b80d-f5654eb1e2c7"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:54:20.207234 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:20.207203    2566 generic.go:356] "Generic (PLEG): container finished" podID="28ebe838-968c-4986-b80d-f5654eb1e2c7" containerID="572f63259d108bc7b99f8bbc9960101f2de580cb0d3889d56ddb33bf42a4244a" exitCode=0
Sep 17 16:54:20.207372 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:20.207247    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w4b7c"
Sep 17 16:54:20.207372 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:20.207298    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w4b7c" event={"ID":"28ebe838-968c-4986-b80d-f5654eb1e2c7","Type":"ContainerDied","Data":"572f63259d108bc7b99f8bbc9960101f2de580cb0d3889d56ddb33bf42a4244a"}
Sep 17 16:54:20.207372 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:20.207352    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w4b7c" event={"ID":"28ebe838-968c-4986-b80d-f5654eb1e2c7","Type":"ContainerDied","Data":"095cdbb2b1f63db170346a3153a72a6e269197d5dda74eaf112d436b3ad314f4"}
Sep 17 16:54:20.207570 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:20.207378    2566 scope.go:117] "RemoveContainer" containerID="572f63259d108bc7b99f8bbc9960101f2de580cb0d3889d56ddb33bf42a4244a"
Sep 17 16:54:20.216756 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:20.216741    2566 scope.go:117] "RemoveContainer" containerID="572f63259d108bc7b99f8bbc9960101f2de580cb0d3889d56ddb33bf42a4244a"
Sep 17 16:54:20.217031 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:54:20.217011    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"572f63259d108bc7b99f8bbc9960101f2de580cb0d3889d56ddb33bf42a4244a\": container with ID starting with 572f63259d108bc7b99f8bbc9960101f2de580cb0d3889d56ddb33bf42a4244a not found: ID does not exist" containerID="572f63259d108bc7b99f8bbc9960101f2de580cb0d3889d56ddb33bf42a4244a"
Sep 17 16:54:20.217068 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:20.217039    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"572f63259d108bc7b99f8bbc9960101f2de580cb0d3889d56ddb33bf42a4244a"} err="failed to get container status \"572f63259d108bc7b99f8bbc9960101f2de580cb0d3889d56ddb33bf42a4244a\": rpc error: code = NotFound desc = could not find container \"572f63259d108bc7b99f8bbc9960101f2de580cb0d3889d56ddb33bf42a4244a\": container with ID starting with 572f63259d108bc7b99f8bbc9960101f2de580cb0d3889d56ddb33bf42a4244a not found: ID does not exist"
Sep 17 16:54:20.231156 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:20.231135    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w4b7c"]
Sep 17 16:54:20.238386 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:20.238366    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w4b7c"]
Sep 17 16:54:20.257873 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:20.257860    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28ebe838-968c-4986-b80d-f5654eb1e2c7-istio-token\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:54:20.257938 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:20.257876    2566 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/28ebe838-968c-4986-b80d-f5654eb1e2c7-istio-csr-dns-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:54:20.257938 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:20.257885    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfmrj\" (UniqueName: \"kubernetes.io/projected/28ebe838-968c-4986-b80d-f5654eb1e2c7-kube-api-access-kfmrj\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:54:20.257938 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:20.257894    2566 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/28ebe838-968c-4986-b80d-f5654eb1e2c7-istio-kubeconfig\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:54:20.257938 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:20.257902    2566 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/28ebe838-968c-4986-b80d-f5654eb1e2c7-istio-csr-ca-configmap\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:54:20.257938 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:20.257911    2566 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/28ebe838-968c-4986-b80d-f5654eb1e2c7-cacerts\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:54:20.257938 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:20.257919    2566 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/28ebe838-968c-4986-b80d-f5654eb1e2c7-local-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:54:20.350319 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:20.350279    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28ebe838-968c-4986-b80d-f5654eb1e2c7" path="/var/lib/kubelet/pods/28ebe838-968c-4986-b80d-f5654eb1e2c7/volumes"
Sep 17 16:54:29.232852 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:29.232820    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-846cf5d849-2pfpl"]
Sep 17 16:54:29.233313 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:29.233109    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28ebe838-968c-4986-b80d-f5654eb1e2c7" containerName="discovery"
Sep 17 16:54:29.233313 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:29.233119    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="28ebe838-968c-4986-b80d-f5654eb1e2c7" containerName="discovery"
Sep 17 16:54:29.233313 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:29.233182    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="28ebe838-968c-4986-b80d-f5654eb1e2c7" containerName="discovery"
Sep 17 16:54:29.237385 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:29.237368    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-846cf5d849-2pfpl"
Sep 17 16:54:29.239679 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:29.239654    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 16:54:29.240635 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:29.240614    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 17 16:54:29.240635 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:29.240615    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-s5svm\""
Sep 17 16:54:29.240792 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:29.240641    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 16:54:29.249869 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:29.249844    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-846cf5d849-2pfpl"]
Sep 17 16:54:29.257276 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:29.257254    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-758fb496fc-sj9bf"]
Sep 17 16:54:29.260471 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:29.260450    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-758fb496fc-sj9bf"
Sep 17 16:54:29.262591 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:29.262569    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 17 16:54:29.262813 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:29.262795    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-9kcd2\""
Sep 17 16:54:29.272028 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:29.272003    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-758fb496fc-sj9bf"]
Sep 17 16:54:29.281323 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:29.281292    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-w8wpl"]
Sep 17 16:54:29.284415 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:29.284395    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-w8wpl"
Sep 17 16:54:29.286794 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:29.286774    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-pb7cj\""
Sep 17 16:54:29.286917 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:29.286838    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 16:54:29.299727 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:29.299702    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-w8wpl"]
Sep 17 16:54:29.322598 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:29.322573    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4822f572-ea4a-4858-a02d-d0a6012cc0db-cert\") pod \"kserve-controller-manager-846cf5d849-2pfpl\" (UID: \"4822f572-ea4a-4858-a02d-d0a6012cc0db\") " pod="kserve/kserve-controller-manager-846cf5d849-2pfpl"
Sep 17 16:54:29.322747 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:29.322617    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9d95\" (UniqueName: \"kubernetes.io/projected/4822f572-ea4a-4858-a02d-d0a6012cc0db-kube-api-access-s9d95\") pod \"kserve-controller-manager-846cf5d849-2pfpl\" (UID: \"4822f572-ea4a-4858-a02d-d0a6012cc0db\") " pod="kserve/kserve-controller-manager-846cf5d849-2pfpl"
Sep 17 16:54:29.322747 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:29.322640    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmbhr\" (UniqueName: \"kubernetes.io/projected/6c28f291-1afd-4f74-b4dc-b87c905f7f79-kube-api-access-rmbhr\") pod \"llmisvc-controller-manager-758fb496fc-sj9bf\" (UID: \"6c28f291-1afd-4f74-b4dc-b87c905f7f79\") " pod="kserve/llmisvc-controller-manager-758fb496fc-sj9bf"
Sep 17 16:54:29.322747 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:29.322703    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c28f291-1afd-4f74-b4dc-b87c905f7f79-cert\") pod \"llmisvc-controller-manager-758fb496fc-sj9bf\" (UID: \"6c28f291-1afd-4f74-b4dc-b87c905f7f79\") " pod="kserve/llmisvc-controller-manager-758fb496fc-sj9bf"
Sep 17 16:54:29.423846 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:29.423818    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1c4b7e49-7705-4fca-8c49-33ca5789eadd-data\") pod \"seaweedfs-64568bcd49-w8wpl\" (UID: \"1c4b7e49-7705-4fca-8c49-33ca5789eadd\") " pod="kserve/seaweedfs-64568bcd49-w8wpl"
Sep 17 16:54:29.424019 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:29.423852    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7wvl\" (UniqueName: \"kubernetes.io/projected/1c4b7e49-7705-4fca-8c49-33ca5789eadd-kube-api-access-f7wvl\") pod \"seaweedfs-64568bcd49-w8wpl\" (UID: \"1c4b7e49-7705-4fca-8c49-33ca5789eadd\") " pod="kserve/seaweedfs-64568bcd49-w8wpl"
Sep 17 16:54:29.424019 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:29.423884    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9d95\" (UniqueName: \"kubernetes.io/projected/4822f572-ea4a-4858-a02d-d0a6012cc0db-kube-api-access-s9d95\") pod \"kserve-controller-manager-846cf5d849-2pfpl\" (UID: \"4822f572-ea4a-4858-a02d-d0a6012cc0db\") " pod="kserve/kserve-controller-manager-846cf5d849-2pfpl"
Sep 17 16:54:29.424019 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:29.423938    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmbhr\" (UniqueName: \"kubernetes.io/projected/6c28f291-1afd-4f74-b4dc-b87c905f7f79-kube-api-access-rmbhr\") pod \"llmisvc-controller-manager-758fb496fc-sj9bf\" (UID: \"6c28f291-1afd-4f74-b4dc-b87c905f7f79\") " pod="kserve/llmisvc-controller-manager-758fb496fc-sj9bf"
Sep 17 16:54:29.424019 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:29.423983    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c28f291-1afd-4f74-b4dc-b87c905f7f79-cert\") pod \"llmisvc-controller-manager-758fb496fc-sj9bf\" (UID: \"6c28f291-1afd-4f74-b4dc-b87c905f7f79\") " pod="kserve/llmisvc-controller-manager-758fb496fc-sj9bf"
Sep 17 16:54:29.424241 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:29.424025    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4822f572-ea4a-4858-a02d-d0a6012cc0db-cert\") pod \"kserve-controller-manager-846cf5d849-2pfpl\" (UID: \"4822f572-ea4a-4858-a02d-d0a6012cc0db\") " pod="kserve/kserve-controller-manager-846cf5d849-2pfpl"
Sep 17 16:54:29.426615 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:29.426592    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4822f572-ea4a-4858-a02d-d0a6012cc0db-cert\") pod \"kserve-controller-manager-846cf5d849-2pfpl\" (UID: \"4822f572-ea4a-4858-a02d-d0a6012cc0db\") " pod="kserve/kserve-controller-manager-846cf5d849-2pfpl"
Sep 17 16:54:29.426615 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:29.426609    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c28f291-1afd-4f74-b4dc-b87c905f7f79-cert\") pod \"llmisvc-controller-manager-758fb496fc-sj9bf\" (UID: \"6c28f291-1afd-4f74-b4dc-b87c905f7f79\") " pod="kserve/llmisvc-controller-manager-758fb496fc-sj9bf"
Sep 17 16:54:29.434903 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:29.434884    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9d95\" (UniqueName: \"kubernetes.io/projected/4822f572-ea4a-4858-a02d-d0a6012cc0db-kube-api-access-s9d95\") pod \"kserve-controller-manager-846cf5d849-2pfpl\" (UID: \"4822f572-ea4a-4858-a02d-d0a6012cc0db\") " pod="kserve/kserve-controller-manager-846cf5d849-2pfpl"
Sep 17 16:54:29.435001 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:29.434935    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmbhr\" (UniqueName: \"kubernetes.io/projected/6c28f291-1afd-4f74-b4dc-b87c905f7f79-kube-api-access-rmbhr\") pod \"llmisvc-controller-manager-758fb496fc-sj9bf\" (UID: \"6c28f291-1afd-4f74-b4dc-b87c905f7f79\") " pod="kserve/llmisvc-controller-manager-758fb496fc-sj9bf"
Sep 17 16:54:29.525214 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:29.525139    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7wvl\" (UniqueName: \"kubernetes.io/projected/1c4b7e49-7705-4fca-8c49-33ca5789eadd-kube-api-access-f7wvl\") pod \"seaweedfs-64568bcd49-w8wpl\" (UID: \"1c4b7e49-7705-4fca-8c49-33ca5789eadd\") " pod="kserve/seaweedfs-64568bcd49-w8wpl"
Sep 17 16:54:29.525337 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:29.525216    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1c4b7e49-7705-4fca-8c49-33ca5789eadd-data\") pod \"seaweedfs-64568bcd49-w8wpl\" (UID: \"1c4b7e49-7705-4fca-8c49-33ca5789eadd\") " pod="kserve/seaweedfs-64568bcd49-w8wpl"
Sep 17 16:54:29.525619 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:29.525602    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1c4b7e49-7705-4fca-8c49-33ca5789eadd-data\") pod \"seaweedfs-64568bcd49-w8wpl\" (UID: \"1c4b7e49-7705-4fca-8c49-33ca5789eadd\") " pod="kserve/seaweedfs-64568bcd49-w8wpl"
Sep 17 16:54:29.535825 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:29.535805    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7wvl\" (UniqueName: \"kubernetes.io/projected/1c4b7e49-7705-4fca-8c49-33ca5789eadd-kube-api-access-f7wvl\") pod \"seaweedfs-64568bcd49-w8wpl\" (UID: \"1c4b7e49-7705-4fca-8c49-33ca5789eadd\") " pod="kserve/seaweedfs-64568bcd49-w8wpl"
Sep 17 16:54:29.547691 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:29.547649    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-846cf5d849-2pfpl"
Sep 17 16:54:29.570721 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:29.570700    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-758fb496fc-sj9bf"
Sep 17 16:54:29.593785 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:29.593757    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-w8wpl"
Sep 17 16:54:29.826093 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:29.826061    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-846cf5d849-2pfpl"]
Sep 17 16:54:29.829037 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:54:29.829005    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4822f572_ea4a_4858_a02d_d0a6012cc0db.slice/crio-1858cc3b3989cf16dde813927f207294448dfd357aafba8af324077d5d8caf49 WatchSource:0}: Error finding container 1858cc3b3989cf16dde813927f207294448dfd357aafba8af324077d5d8caf49: Status 404 returned error can't find the container with id 1858cc3b3989cf16dde813927f207294448dfd357aafba8af324077d5d8caf49
Sep 17 16:54:29.863011 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:29.862988    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-758fb496fc-sj9bf"]
Sep 17 16:54:29.865072 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:54:29.865038    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6c28f291_1afd_4f74_b4dc_b87c905f7f79.slice/crio-e5b3b3ee83d5be15d10168cd73db2b642beff334645a4770c3ce3b8b52423b97 WatchSource:0}: Error finding container e5b3b3ee83d5be15d10168cd73db2b642beff334645a4770c3ce3b8b52423b97: Status 404 returned error can't find the container with id e5b3b3ee83d5be15d10168cd73db2b642beff334645a4770c3ce3b8b52423b97
Sep 17 16:54:29.884709 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:29.884683    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-w8wpl"]
Sep 17 16:54:29.886360 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:54:29.886336    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c4b7e49_7705_4fca_8c49_33ca5789eadd.slice/crio-a7dfadaff8098e00042a65847ab74dc3aa995c73c92df68b07d4b838357adf1b WatchSource:0}: Error finding container a7dfadaff8098e00042a65847ab74dc3aa995c73c92df68b07d4b838357adf1b: Status 404 returned error can't find the container with id a7dfadaff8098e00042a65847ab74dc3aa995c73c92df68b07d4b838357adf1b
Sep 17 16:54:30.255476 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:30.255371    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-758fb496fc-sj9bf" event={"ID":"6c28f291-1afd-4f74-b4dc-b87c905f7f79","Type":"ContainerStarted","Data":"e5b3b3ee83d5be15d10168cd73db2b642beff334645a4770c3ce3b8b52423b97"}
Sep 17 16:54:30.256726 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:30.256698    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-w8wpl" event={"ID":"1c4b7e49-7705-4fca-8c49-33ca5789eadd","Type":"ContainerStarted","Data":"a7dfadaff8098e00042a65847ab74dc3aa995c73c92df68b07d4b838357adf1b"}
Sep 17 16:54:30.257770 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:30.257746    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-846cf5d849-2pfpl" event={"ID":"4822f572-ea4a-4858-a02d-d0a6012cc0db","Type":"ContainerStarted","Data":"1858cc3b3989cf16dde813927f207294448dfd357aafba8af324077d5d8caf49"}
Sep 17 16:54:35.292984 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:35.292942    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-w8wpl" event={"ID":"1c4b7e49-7705-4fca-8c49-33ca5789eadd","Type":"ContainerStarted","Data":"3d8895387331fd5ff21d24f4718106492b600c26d3dc402a94e90dea9ad64db5"}
Sep 17 16:54:35.293473 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:35.293011    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-w8wpl"
Sep 17 16:54:35.294497 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:35.294470    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-846cf5d849-2pfpl" event={"ID":"4822f572-ea4a-4858-a02d-d0a6012cc0db","Type":"ContainerStarted","Data":"9fdc5853bba8cfc1839d9a7491b40c2fe028093d6a95b1a66d0d62abb103f9d3"}
Sep 17 16:54:35.294618 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:35.294566    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-846cf5d849-2pfpl"
Sep 17 16:54:35.295956 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:35.295931    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-758fb496fc-sj9bf" event={"ID":"6c28f291-1afd-4f74-b4dc-b87c905f7f79","Type":"ContainerStarted","Data":"24eda31890f2c786529809f09cd3ac4a07a716eb4f4840d28b5fa674fc26c75c"}
Sep 17 16:54:35.296045 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:35.295971    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-758fb496fc-sj9bf"
Sep 17 16:54:35.310964 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:35.310906    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-w8wpl" podStartSLOduration=1.6966841910000001 podStartE2EDuration="6.310889033s" totalImagesPullingTime="4.614204842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:54:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:54:35.309956527 +0000 UTC m=+761.204435661" watchObservedRunningTime="2026-09-17 16:54:35.310889033 +0000 UTC m=+761.205368166"
Sep 17 16:54:35.330639 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:35.330577    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-846cf5d849-2pfpl" podStartSLOduration=1.7878398020000001 podStartE2EDuration="6.330557994s" totalImagesPullingTime="4.542718192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:54:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:54:35.329966515 +0000 UTC m=+761.224445655" watchObservedRunningTime="2026-09-17 16:54:35.330557994 +0000 UTC m=+761.225037135"
Sep 17 16:54:35.347099 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:35.347047    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-758fb496fc-sj9bf" podStartSLOduration=1.763396014 podStartE2EDuration="6.347031147s" totalImagesPullingTime="4.583635133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:54:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:54:35.346163342 +0000 UTC m=+761.240642484" watchObservedRunningTime="2026-09-17 16:54:35.347031147 +0000 UTC m=+761.241510287"
Sep 17 16:54:41.304531 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:54:41.304505    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-w8wpl"
Sep 17 16:55:06.303837 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:06.303754    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-758fb496fc-sj9bf"
Sep 17 16:55:06.306662 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:06.306634    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-846cf5d849-2pfpl"
Sep 17 16:55:15.342534 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:15.342498    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-846cf5d849-2pfpl"]
Sep 17 16:55:15.342965 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:15.342734    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-846cf5d849-2pfpl" podUID="4822f572-ea4a-4858-a02d-d0a6012cc0db" containerName="manager" containerID="cri-o://9fdc5853bba8cfc1839d9a7491b40c2fe028093d6a95b1a66d0d62abb103f9d3" gracePeriod=10
Sep 17 16:55:15.369467 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:15.369413    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-846cf5d849-ndw56"]
Sep 17 16:55:15.372416 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:15.372390    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-846cf5d849-ndw56"
Sep 17 16:55:15.392065 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:15.392042    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-846cf5d849-ndw56"]
Sep 17 16:55:15.473906 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:15.473852    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm48h\" (UniqueName: \"kubernetes.io/projected/65d593a7-e622-40a5-81b9-ff549faf9546-kube-api-access-mm48h\") pod \"kserve-controller-manager-846cf5d849-ndw56\" (UID: \"65d593a7-e622-40a5-81b9-ff549faf9546\") " pod="kserve/kserve-controller-manager-846cf5d849-ndw56"
Sep 17 16:55:15.474053 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:15.473934    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65d593a7-e622-40a5-81b9-ff549faf9546-cert\") pod \"kserve-controller-manager-846cf5d849-ndw56\" (UID: \"65d593a7-e622-40a5-81b9-ff549faf9546\") " pod="kserve/kserve-controller-manager-846cf5d849-ndw56"
Sep 17 16:55:15.489649 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:15.489623    2566 generic.go:356] "Generic (PLEG): container finished" podID="4822f572-ea4a-4858-a02d-d0a6012cc0db" containerID="9fdc5853bba8cfc1839d9a7491b40c2fe028093d6a95b1a66d0d62abb103f9d3" exitCode=0
Sep 17 16:55:15.489781 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:15.489701    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-846cf5d849-2pfpl" event={"ID":"4822f572-ea4a-4858-a02d-d0a6012cc0db","Type":"ContainerDied","Data":"9fdc5853bba8cfc1839d9a7491b40c2fe028093d6a95b1a66d0d62abb103f9d3"}
Sep 17 16:55:15.574382 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:15.574357    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm48h\" (UniqueName: \"kubernetes.io/projected/65d593a7-e622-40a5-81b9-ff549faf9546-kube-api-access-mm48h\") pod \"kserve-controller-manager-846cf5d849-ndw56\" (UID: \"65d593a7-e622-40a5-81b9-ff549faf9546\") " pod="kserve/kserve-controller-manager-846cf5d849-ndw56"
Sep 17 16:55:15.574518 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:15.574393    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65d593a7-e622-40a5-81b9-ff549faf9546-cert\") pod \"kserve-controller-manager-846cf5d849-ndw56\" (UID: \"65d593a7-e622-40a5-81b9-ff549faf9546\") " pod="kserve/kserve-controller-manager-846cf5d849-ndw56"
Sep 17 16:55:15.576927 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:15.576900    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65d593a7-e622-40a5-81b9-ff549faf9546-cert\") pod \"kserve-controller-manager-846cf5d849-ndw56\" (UID: \"65d593a7-e622-40a5-81b9-ff549faf9546\") " pod="kserve/kserve-controller-manager-846cf5d849-ndw56"
Sep 17 16:55:15.579552 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:15.579534    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-846cf5d849-2pfpl"
Sep 17 16:55:15.584505 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:15.584477    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm48h\" (UniqueName: \"kubernetes.io/projected/65d593a7-e622-40a5-81b9-ff549faf9546-kube-api-access-mm48h\") pod \"kserve-controller-manager-846cf5d849-ndw56\" (UID: \"65d593a7-e622-40a5-81b9-ff549faf9546\") " pod="kserve/kserve-controller-manager-846cf5d849-ndw56"
Sep 17 16:55:15.675715 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:15.675634    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9d95\" (UniqueName: \"kubernetes.io/projected/4822f572-ea4a-4858-a02d-d0a6012cc0db-kube-api-access-s9d95\") pod \"4822f572-ea4a-4858-a02d-d0a6012cc0db\" (UID: \"4822f572-ea4a-4858-a02d-d0a6012cc0db\") "
Sep 17 16:55:15.675715 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:15.675686    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4822f572-ea4a-4858-a02d-d0a6012cc0db-cert\") pod \"4822f572-ea4a-4858-a02d-d0a6012cc0db\" (UID: \"4822f572-ea4a-4858-a02d-d0a6012cc0db\") "
Sep 17 16:55:15.677866 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:15.677839    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4822f572-ea4a-4858-a02d-d0a6012cc0db-cert" (OuterVolumeSpecName: "cert") pod "4822f572-ea4a-4858-a02d-d0a6012cc0db" (UID: "4822f572-ea4a-4858-a02d-d0a6012cc0db"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:55:15.677980 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:15.677885    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4822f572-ea4a-4858-a02d-d0a6012cc0db-kube-api-access-s9d95" (OuterVolumeSpecName: "kube-api-access-s9d95") pod "4822f572-ea4a-4858-a02d-d0a6012cc0db" (UID: "4822f572-ea4a-4858-a02d-d0a6012cc0db"). InnerVolumeSpecName "kube-api-access-s9d95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:55:15.722217 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:15.722190    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-846cf5d849-ndw56"
Sep 17 16:55:15.777078 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:15.776899    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9d95\" (UniqueName: \"kubernetes.io/projected/4822f572-ea4a-4858-a02d-d0a6012cc0db-kube-api-access-s9d95\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:55:15.777078 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:15.776923    2566 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4822f572-ea4a-4858-a02d-d0a6012cc0db-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:55:15.994188 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:15.994162    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-846cf5d849-ndw56"]
Sep 17 16:55:15.996343 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:55:15.996319    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65d593a7_e622_40a5_81b9_ff549faf9546.slice/crio-ef1d9a0aa02acde221dff72170f3c8411b3ed0a84c961b3c3a4e90f969a1dcf3 WatchSource:0}: Error finding container ef1d9a0aa02acde221dff72170f3c8411b3ed0a84c961b3c3a4e90f969a1dcf3: Status 404 returned error can't find the container with id ef1d9a0aa02acde221dff72170f3c8411b3ed0a84c961b3c3a4e90f969a1dcf3
Sep 17 16:55:15.997660 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:15.997640    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:55:16.495513 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:16.495477    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-846cf5d849-ndw56" event={"ID":"65d593a7-e622-40a5-81b9-ff549faf9546","Type":"ContainerStarted","Data":"ef1d9a0aa02acde221dff72170f3c8411b3ed0a84c961b3c3a4e90f969a1dcf3"}
Sep 17 16:55:16.496724 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:16.496707    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-846cf5d849-2pfpl"
Sep 17 16:55:16.496724 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:16.496719    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-846cf5d849-2pfpl" event={"ID":"4822f572-ea4a-4858-a02d-d0a6012cc0db","Type":"ContainerDied","Data":"1858cc3b3989cf16dde813927f207294448dfd357aafba8af324077d5d8caf49"}
Sep 17 16:55:16.496865 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:16.496766    2566 scope.go:117] "RemoveContainer" containerID="9fdc5853bba8cfc1839d9a7491b40c2fe028093d6a95b1a66d0d62abb103f9d3"
Sep 17 16:55:16.515825 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:16.515804    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-846cf5d849-2pfpl"]
Sep 17 16:55:16.523126 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:16.523104    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-846cf5d849-2pfpl"]
Sep 17 16:55:17.503342 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:17.503304    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-846cf5d849-ndw56" event={"ID":"65d593a7-e622-40a5-81b9-ff549faf9546","Type":"ContainerStarted","Data":"3d8b5243457c746a6ff9b21074c32a17e6919609686d1804f48f6d69a1919a6a"}
Sep 17 16:55:17.503922 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:17.503402    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-846cf5d849-ndw56"
Sep 17 16:55:17.522705 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:17.522653    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-846cf5d849-ndw56" podStartSLOduration=2.012427844 podStartE2EDuration="2.522638036s" totalImagesPullingTime="510.210192ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:55:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:55:17.520749724 +0000 UTC m=+803.415228864" watchObservedRunningTime="2026-09-17 16:55:17.522638036 +0000 UTC m=+803.417117175"
Sep 17 16:55:18.346354 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:18.346316    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4822f572-ea4a-4858-a02d-d0a6012cc0db" path="/var/lib/kubelet/pods/4822f572-ea4a-4858-a02d-d0a6012cc0db/volumes"
Sep 17 16:55:48.514740 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:48.514712    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-846cf5d849-ndw56"
Sep 17 16:55:50.046032 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:50.045999    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-lssj5"]
Sep 17 16:55:50.046379 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:50.046305    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4822f572-ea4a-4858-a02d-d0a6012cc0db" containerName="manager"
Sep 17 16:55:50.046379 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:50.046315    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4822f572-ea4a-4858-a02d-d0a6012cc0db" containerName="manager"
Sep 17 16:55:50.046379 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:50.046377    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4822f572-ea4a-4858-a02d-d0a6012cc0db" containerName="manager"
Sep 17 16:55:50.049480 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:50.049458    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-lssj5"
Sep 17 16:55:50.051774 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:50.051755    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 17 16:55:50.051908 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:50.051889    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-rxvhf\""
Sep 17 16:55:50.063118 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:50.063092    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-lssj5"]
Sep 17 16:55:50.065351 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:50.065333    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-5d6kl"]
Sep 17 16:55:50.068384 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:50.068369    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-5d6kl"
Sep 17 16:55:50.070832 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:50.070811    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 17 16:55:50.070951 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:50.070927    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 17 16:55:50.071010 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:50.070991    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-llfrx\""
Sep 17 16:55:50.078512 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:50.078493    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-5d6kl"]
Sep 17 16:55:50.104987 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:50.104956    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a3b6e7c-75d9-4ab6-9319-35a0bb32bebb-tls-certs\") pod \"model-serving-api-7b954ffc95-lssj5\" (UID: \"9a3b6e7c-75d9-4ab6-9319-35a0bb32bebb\") " pod="kserve/model-serving-api-7b954ffc95-lssj5"
Sep 17 16:55:50.105099 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:50.105025    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5npbq\" (UniqueName: \"kubernetes.io/projected/9a3b6e7c-75d9-4ab6-9319-35a0bb32bebb-kube-api-access-5npbq\") pod \"model-serving-api-7b954ffc95-lssj5\" (UID: \"9a3b6e7c-75d9-4ab6-9319-35a0bb32bebb\") " pod="kserve/model-serving-api-7b954ffc95-lssj5"
Sep 17 16:55:50.206220 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:50.206193    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb6aa6ec-c87f-4948-9617-1c512c866fd3-cert\") pod \"odh-model-controller-7b966d7984-5d6kl\" (UID: \"bb6aa6ec-c87f-4948-9617-1c512c866fd3\") " pod="kserve/odh-model-controller-7b966d7984-5d6kl"
Sep 17 16:55:50.206346 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:50.206231    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5npbq\" (UniqueName: \"kubernetes.io/projected/9a3b6e7c-75d9-4ab6-9319-35a0bb32bebb-kube-api-access-5npbq\") pod \"model-serving-api-7b954ffc95-lssj5\" (UID: \"9a3b6e7c-75d9-4ab6-9319-35a0bb32bebb\") " pod="kserve/model-serving-api-7b954ffc95-lssj5"
Sep 17 16:55:50.206346 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:50.206257    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a3b6e7c-75d9-4ab6-9319-35a0bb32bebb-tls-certs\") pod \"model-serving-api-7b954ffc95-lssj5\" (UID: \"9a3b6e7c-75d9-4ab6-9319-35a0bb32bebb\") " pod="kserve/model-serving-api-7b954ffc95-lssj5"
Sep 17 16:55:50.206346 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:50.206288    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb6aa6ec-c87f-4948-9617-1c512c866fd3-metrics-certs\") pod \"odh-model-controller-7b966d7984-5d6kl\" (UID: \"bb6aa6ec-c87f-4948-9617-1c512c866fd3\") " pod="kserve/odh-model-controller-7b966d7984-5d6kl"
Sep 17 16:55:50.206526 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:55:50.206372    2566 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 17 16:55:50.206526 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:55:50.206417    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a3b6e7c-75d9-4ab6-9319-35a0bb32bebb-tls-certs podName:9a3b6e7c-75d9-4ab6-9319-35a0bb32bebb nodeName:}" failed. No retries permitted until 2026-09-17 16:55:50.706401791 +0000 UTC m=+836.600880908 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/9a3b6e7c-75d9-4ab6-9319-35a0bb32bebb-tls-certs") pod "model-serving-api-7b954ffc95-lssj5" (UID: "9a3b6e7c-75d9-4ab6-9319-35a0bb32bebb") : secret "model-serving-api-tls" not found
Sep 17 16:55:50.206526 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:50.206487    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgxch\" (UniqueName: \"kubernetes.io/projected/bb6aa6ec-c87f-4948-9617-1c512c866fd3-kube-api-access-vgxch\") pod \"odh-model-controller-7b966d7984-5d6kl\" (UID: \"bb6aa6ec-c87f-4948-9617-1c512c866fd3\") " pod="kserve/odh-model-controller-7b966d7984-5d6kl"
Sep 17 16:55:50.216214 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:50.216194    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5npbq\" (UniqueName: \"kubernetes.io/projected/9a3b6e7c-75d9-4ab6-9319-35a0bb32bebb-kube-api-access-5npbq\") pod \"model-serving-api-7b954ffc95-lssj5\" (UID: \"9a3b6e7c-75d9-4ab6-9319-35a0bb32bebb\") " pod="kserve/model-serving-api-7b954ffc95-lssj5"
Sep 17 16:55:50.307661 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:50.307592    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb6aa6ec-c87f-4948-9617-1c512c866fd3-metrics-certs\") pod \"odh-model-controller-7b966d7984-5d6kl\" (UID: \"bb6aa6ec-c87f-4948-9617-1c512c866fd3\") " pod="kserve/odh-model-controller-7b966d7984-5d6kl"
Sep 17 16:55:50.307661 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:50.307641    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgxch\" (UniqueName: \"kubernetes.io/projected/bb6aa6ec-c87f-4948-9617-1c512c866fd3-kube-api-access-vgxch\") pod \"odh-model-controller-7b966d7984-5d6kl\" (UID: \"bb6aa6ec-c87f-4948-9617-1c512c866fd3\") " pod="kserve/odh-model-controller-7b966d7984-5d6kl"
Sep 17 16:55:50.307661 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:50.307659    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb6aa6ec-c87f-4948-9617-1c512c866fd3-cert\") pod \"odh-model-controller-7b966d7984-5d6kl\" (UID: \"bb6aa6ec-c87f-4948-9617-1c512c866fd3\") " pod="kserve/odh-model-controller-7b966d7984-5d6kl"
Sep 17 16:55:50.307849 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:55:50.307749    2566 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Sep 17 16:55:50.307849 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:55:50.307829    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb6aa6ec-c87f-4948-9617-1c512c866fd3-metrics-certs podName:bb6aa6ec-c87f-4948-9617-1c512c866fd3 nodeName:}" failed. No retries permitted until 2026-09-17 16:55:50.807807997 +0000 UTC m=+836.702287127 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb6aa6ec-c87f-4948-9617-1c512c866fd3-metrics-certs") pod "odh-model-controller-7b966d7984-5d6kl" (UID: "bb6aa6ec-c87f-4948-9617-1c512c866fd3") : secret "odh-model-controller-metrics-tls" not found
Sep 17 16:55:50.310198 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:50.310177    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb6aa6ec-c87f-4948-9617-1c512c866fd3-cert\") pod \"odh-model-controller-7b966d7984-5d6kl\" (UID: \"bb6aa6ec-c87f-4948-9617-1c512c866fd3\") " pod="kserve/odh-model-controller-7b966d7984-5d6kl"
Sep 17 16:55:50.320264 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:50.320240    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgxch\" (UniqueName: \"kubernetes.io/projected/bb6aa6ec-c87f-4948-9617-1c512c866fd3-kube-api-access-vgxch\") pod \"odh-model-controller-7b966d7984-5d6kl\" (UID: \"bb6aa6ec-c87f-4948-9617-1c512c866fd3\") " pod="kserve/odh-model-controller-7b966d7984-5d6kl"
Sep 17 16:55:50.710382 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:50.710297    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a3b6e7c-75d9-4ab6-9319-35a0bb32bebb-tls-certs\") pod \"model-serving-api-7b954ffc95-lssj5\" (UID: \"9a3b6e7c-75d9-4ab6-9319-35a0bb32bebb\") " pod="kserve/model-serving-api-7b954ffc95-lssj5"
Sep 17 16:55:50.712798 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:50.712772    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a3b6e7c-75d9-4ab6-9319-35a0bb32bebb-tls-certs\") pod \"model-serving-api-7b954ffc95-lssj5\" (UID: \"9a3b6e7c-75d9-4ab6-9319-35a0bb32bebb\") " pod="kserve/model-serving-api-7b954ffc95-lssj5"
Sep 17 16:55:50.811526 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:50.811497    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb6aa6ec-c87f-4948-9617-1c512c866fd3-metrics-certs\") pod \"odh-model-controller-7b966d7984-5d6kl\" (UID: \"bb6aa6ec-c87f-4948-9617-1c512c866fd3\") " pod="kserve/odh-model-controller-7b966d7984-5d6kl"
Sep 17 16:55:50.813933 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:50.813902    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb6aa6ec-c87f-4948-9617-1c512c866fd3-metrics-certs\") pod \"odh-model-controller-7b966d7984-5d6kl\" (UID: \"bb6aa6ec-c87f-4948-9617-1c512c866fd3\") " pod="kserve/odh-model-controller-7b966d7984-5d6kl"
Sep 17 16:55:50.959138 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:50.959105    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-lssj5"
Sep 17 16:55:50.979206 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:50.979184    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-5d6kl"
Sep 17 16:55:51.232641 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:51.232617    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-lssj5"]
Sep 17 16:55:51.234788 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:55:51.234763    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a3b6e7c_75d9_4ab6_9319_35a0bb32bebb.slice/crio-fbac23d397ffda565931cbe28edad9ec045fff4c95368697a1e102a08affc7ce WatchSource:0}: Error finding container fbac23d397ffda565931cbe28edad9ec045fff4c95368697a1e102a08affc7ce: Status 404 returned error can't find the container with id fbac23d397ffda565931cbe28edad9ec045fff4c95368697a1e102a08affc7ce
Sep 17 16:55:51.261603 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:51.261580    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-5d6kl"]
Sep 17 16:55:51.262892 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:55:51.262867    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb6aa6ec_c87f_4948_9617_1c512c866fd3.slice/crio-bcdb96285aa8ce6912e43b8e17af39f7bb728f8a664fa12e086b1ca6ac5c3067 WatchSource:0}: Error finding container bcdb96285aa8ce6912e43b8e17af39f7bb728f8a664fa12e086b1ca6ac5c3067: Status 404 returned error can't find the container with id bcdb96285aa8ce6912e43b8e17af39f7bb728f8a664fa12e086b1ca6ac5c3067
Sep 17 16:55:51.668690 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:51.668650    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-5d6kl" event={"ID":"bb6aa6ec-c87f-4948-9617-1c512c866fd3","Type":"ContainerStarted","Data":"bcdb96285aa8ce6912e43b8e17af39f7bb728f8a664fa12e086b1ca6ac5c3067"}
Sep 17 16:55:51.669842 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:51.669812    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-lssj5" event={"ID":"9a3b6e7c-75d9-4ab6-9319-35a0bb32bebb","Type":"ContainerStarted","Data":"fbac23d397ffda565931cbe28edad9ec045fff4c95368697a1e102a08affc7ce"}
Sep 17 16:55:55.695043 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:55.695002    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-5d6kl" event={"ID":"bb6aa6ec-c87f-4948-9617-1c512c866fd3","Type":"ContainerStarted","Data":"bdecd9f8641ead8ad8ec4d208b53f642cf6fd09d55d61d82bc4af766247fc708"}
Sep 17 16:55:55.695506 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:55.695071    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-5d6kl"
Sep 17 16:55:55.696660 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:55.696639    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-lssj5" event={"ID":"9a3b6e7c-75d9-4ab6-9319-35a0bb32bebb","Type":"ContainerStarted","Data":"22dde38215feb2eab75cd9cd93e3d0f99a925a392f346f0ee57a158c68c3b9e2"}
Sep 17 16:55:55.696774 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:55.696761    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-lssj5"
Sep 17 16:55:55.718137 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:55.718086    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-5d6kl" podStartSLOduration=1.9061320290000001 podStartE2EDuration="5.718071705s" totalImagesPullingTime="3.811939676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:55:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:55:55.715520057 +0000 UTC m=+841.609999196" watchObservedRunningTime="2026-09-17 16:55:55.718071705 +0000 UTC m=+841.612550843"
Sep 17 16:55:55.754723 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:55:55.754420    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-lssj5" podStartSLOduration=1.875989895 podStartE2EDuration="5.754398008s" totalImagesPullingTime="3.878408113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:55:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:55:55.746960158 +0000 UTC m=+841.641439298" watchObservedRunningTime="2026-09-17 16:55:55.754398008 +0000 UTC m=+841.648877150"
Sep 17 16:56:06.703291 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:56:06.703261    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-5d6kl"
Sep 17 16:56:06.705330 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:56:06.705306    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-lssj5"
Sep 17 16:56:07.667450 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:56:07.667410    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-9vs5m"]
Sep 17 16:56:07.675209 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:56:07.675184    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9vs5m"
Sep 17 16:56:07.677402 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:56:07.677377    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 16:56:07.686823 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:56:07.686802    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-9vs5m"]
Sep 17 16:56:07.746737 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:56:07.746707    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2453727a-9cf4-444a-8e22-d7d2daae63e4-model-cache\") pod \"s3-init-9vs5m\" (UID: \"2453727a-9cf4-444a-8e22-d7d2daae63e4\") " pod="kserve/s3-init-9vs5m"
Sep 17 16:56:07.747058 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:56:07.746770    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltjld\" (UniqueName: \"kubernetes.io/projected/2453727a-9cf4-444a-8e22-d7d2daae63e4-kube-api-access-ltjld\") pod \"s3-init-9vs5m\" (UID: \"2453727a-9cf4-444a-8e22-d7d2daae63e4\") " pod="kserve/s3-init-9vs5m"
Sep 17 16:56:07.847533 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:56:07.847501    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2453727a-9cf4-444a-8e22-d7d2daae63e4-model-cache\") pod \"s3-init-9vs5m\" (UID: \"2453727a-9cf4-444a-8e22-d7d2daae63e4\") " pod="kserve/s3-init-9vs5m"
Sep 17 16:56:07.847669 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:56:07.847559    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltjld\" (UniqueName: \"kubernetes.io/projected/2453727a-9cf4-444a-8e22-d7d2daae63e4-kube-api-access-ltjld\") pod \"s3-init-9vs5m\" (UID: \"2453727a-9cf4-444a-8e22-d7d2daae63e4\") " pod="kserve/s3-init-9vs5m"
Sep 17 16:56:07.847963 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:56:07.847944    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2453727a-9cf4-444a-8e22-d7d2daae63e4-model-cache\") pod \"s3-init-9vs5m\" (UID: \"2453727a-9cf4-444a-8e22-d7d2daae63e4\") " pod="kserve/s3-init-9vs5m"
Sep 17 16:56:07.858202 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:56:07.858181    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltjld\" (UniqueName: \"kubernetes.io/projected/2453727a-9cf4-444a-8e22-d7d2daae63e4-kube-api-access-ltjld\") pod \"s3-init-9vs5m\" (UID: \"2453727a-9cf4-444a-8e22-d7d2daae63e4\") " pod="kserve/s3-init-9vs5m"
Sep 17 16:56:07.984395 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:56:07.984310    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9vs5m"
Sep 17 16:56:08.255489 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:56:08.255464    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-9vs5m"]
Sep 17 16:56:08.257873 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:56:08.257844    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2453727a_9cf4_444a_8e22_d7d2daae63e4.slice/crio-542b04564f9cfde3f15dfa10d6413cbe09d5278f0d496dfec439623cf369f3a7 WatchSource:0}: Error finding container 542b04564f9cfde3f15dfa10d6413cbe09d5278f0d496dfec439623cf369f3a7: Status 404 returned error can't find the container with id 542b04564f9cfde3f15dfa10d6413cbe09d5278f0d496dfec439623cf369f3a7
Sep 17 16:56:08.761528 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:56:08.761492    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9vs5m" event={"ID":"2453727a-9cf4-444a-8e22-d7d2daae63e4","Type":"ContainerStarted","Data":"542b04564f9cfde3f15dfa10d6413cbe09d5278f0d496dfec439623cf369f3a7"}
Sep 17 16:56:12.784103 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:56:12.784032    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9vs5m" event={"ID":"2453727a-9cf4-444a-8e22-d7d2daae63e4","Type":"ContainerStarted","Data":"567e97854feb342b5f18d233cb58eea4ea897ef1151831b6924920de80768fde"}
Sep 17 16:56:14.796169 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:56:14.796131    2566 generic.go:356] "Generic (PLEG): container finished" podID="2453727a-9cf4-444a-8e22-d7d2daae63e4" containerID="567e97854feb342b5f18d233cb58eea4ea897ef1151831b6924920de80768fde" exitCode=0
Sep 17 16:56:14.796584 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:56:14.796213    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9vs5m" event={"ID":"2453727a-9cf4-444a-8e22-d7d2daae63e4","Type":"ContainerDied","Data":"567e97854feb342b5f18d233cb58eea4ea897ef1151831b6924920de80768fde"}
Sep 17 16:56:19.827317 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:56:19.827283    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9vs5m" event={"ID":"2453727a-9cf4-444a-8e22-d7d2daae63e4","Type":"ContainerStarted","Data":"b5a7fe238704349bd3dfda8ccb39fbc991fcf7a2b43266013cdabf1045beac9b"}
Sep 17 16:56:19.850161 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:56:19.850109    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-9vs5m" podStartSLOduration=2.944519957 podStartE2EDuration="12.850091241s" totalImagesPullingTime="7.86588149s" totalInitContainerRuntime="2.039689794s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:56:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:56:19.847710148 +0000 UTC m=+865.742189287" watchObservedRunningTime="2026-09-17 16:56:19.850091241 +0000 UTC m=+865.744570393"
Sep 17 16:56:56.306205 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:56:56.306179    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gblhr_91477834-6648-4e75-8530-e01bdee5e57e/ovn-acl-logging/0.log"
Sep 17 16:56:56.306682 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:56:56.306570    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gblhr_91477834-6648-4e75-8530-e01bdee5e57e/ovn-acl-logging/0.log"
Sep 17 16:57:25.148751 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:25.148717    2566 generic.go:356] "Generic (PLEG): container finished" podID="2453727a-9cf4-444a-8e22-d7d2daae63e4" containerID="b5a7fe238704349bd3dfda8ccb39fbc991fcf7a2b43266013cdabf1045beac9b" exitCode=0
Sep 17 16:57:25.149142 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:25.148788    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9vs5m" event={"ID":"2453727a-9cf4-444a-8e22-d7d2daae63e4","Type":"ContainerDied","Data":"b5a7fe238704349bd3dfda8ccb39fbc991fcf7a2b43266013cdabf1045beac9b"}
Sep 17 16:57:26.282633 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:26.282605    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9vs5m"
Sep 17 16:57:26.337775 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:26.337750    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2453727a-9cf4-444a-8e22-d7d2daae63e4-model-cache\") pod \"2453727a-9cf4-444a-8e22-d7d2daae63e4\" (UID: \"2453727a-9cf4-444a-8e22-d7d2daae63e4\") "
Sep 17 16:57:26.337896 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:26.337780    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltjld\" (UniqueName: \"kubernetes.io/projected/2453727a-9cf4-444a-8e22-d7d2daae63e4-kube-api-access-ltjld\") pod \"2453727a-9cf4-444a-8e22-d7d2daae63e4\" (UID: \"2453727a-9cf4-444a-8e22-d7d2daae63e4\") "
Sep 17 16:57:26.339998 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:26.339971    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2453727a-9cf4-444a-8e22-d7d2daae63e4-kube-api-access-ltjld" (OuterVolumeSpecName: "kube-api-access-ltjld") pod "2453727a-9cf4-444a-8e22-d7d2daae63e4" (UID: "2453727a-9cf4-444a-8e22-d7d2daae63e4"). InnerVolumeSpecName "kube-api-access-ltjld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:57:26.357040 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:26.357014    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2453727a-9cf4-444a-8e22-d7d2daae63e4-model-cache" (OuterVolumeSpecName: "model-cache") pod "2453727a-9cf4-444a-8e22-d7d2daae63e4" (UID: "2453727a-9cf4-444a-8e22-d7d2daae63e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:57:26.439280 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:26.439223    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2453727a-9cf4-444a-8e22-d7d2daae63e4-model-cache\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:57:26.439280 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:26.439246    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltjld\" (UniqueName: \"kubernetes.io/projected/2453727a-9cf4-444a-8e22-d7d2daae63e4-kube-api-access-ltjld\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:57:27.160181 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:27.160146    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9vs5m" event={"ID":"2453727a-9cf4-444a-8e22-d7d2daae63e4","Type":"ContainerDied","Data":"542b04564f9cfde3f15dfa10d6413cbe09d5278f0d496dfec439623cf369f3a7"}
Sep 17 16:57:27.160181 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:27.160188    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="542b04564f9cfde3f15dfa10d6413cbe09d5278f0d496dfec439623cf369f3a7"
Sep 17 16:57:27.160409 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:27.160151    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9vs5m"
Sep 17 16:57:51.884543 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:51.884510    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzqdh"]
Sep 17 16:57:51.886896 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:51.884816    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2453727a-9cf4-444a-8e22-d7d2daae63e4" containerName="download-hf-model"
Sep 17 16:57:51.886896 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:51.884827    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2453727a-9cf4-444a-8e22-d7d2daae63e4" containerName="download-hf-model"
Sep 17 16:57:51.886896 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:51.884846    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2453727a-9cf4-444a-8e22-d7d2daae63e4" containerName="s3-init"
Sep 17 16:57:51.886896 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:51.884851    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2453727a-9cf4-444a-8e22-d7d2daae63e4" containerName="s3-init"
Sep 17 16:57:51.886896 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:51.884904    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2453727a-9cf4-444a-8e22-d7d2daae63e4" containerName="s3-init"
Sep 17 16:57:51.887776 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:51.887759    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzqdh"
Sep 17 16:57:51.895819 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:51.895797    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 17 16:57:51.896747 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:51.896729    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 17 16:57:51.896841 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:51.896826    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 17 16:57:51.896902 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:51.896840    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-vmnb8\""
Sep 17 16:57:51.906945 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:51.906922    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzqdh"]
Sep 17 16:57:52.052387 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:52.052352    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bee8068-67de-4d26-9509-c9777f7cdec3-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-rzqdh\" (UID: \"5bee8068-67de-4d26-9509-c9777f7cdec3\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzqdh"
Sep 17 16:57:52.052387 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:52.052386    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkk5m\" (UniqueName: \"kubernetes.io/projected/5bee8068-67de-4d26-9509-c9777f7cdec3-kube-api-access-zkk5m\") pod \"fc-smoke-test-kserve-7f4687897-rzqdh\" (UID: \"5bee8068-67de-4d26-9509-c9777f7cdec3\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzqdh"
Sep 17 16:57:52.052670 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:52.052530    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bee8068-67de-4d26-9509-c9777f7cdec3-home\") pod \"fc-smoke-test-kserve-7f4687897-rzqdh\" (UID: \"5bee8068-67de-4d26-9509-c9777f7cdec3\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzqdh"
Sep 17 16:57:52.052670 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:52.052569    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bee8068-67de-4d26-9509-c9777f7cdec3-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-rzqdh\" (UID: \"5bee8068-67de-4d26-9509-c9777f7cdec3\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzqdh"
Sep 17 16:57:52.052670 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:52.052618    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bee8068-67de-4d26-9509-c9777f7cdec3-dshm\") pod \"fc-smoke-test-kserve-7f4687897-rzqdh\" (UID: \"5bee8068-67de-4d26-9509-c9777f7cdec3\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzqdh"
Sep 17 16:57:52.052670 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:52.052642    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bee8068-67de-4d26-9509-c9777f7cdec3-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-rzqdh\" (UID: \"5bee8068-67de-4d26-9509-c9777f7cdec3\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzqdh"
Sep 17 16:57:52.153973 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:52.153888    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bee8068-67de-4d26-9509-c9777f7cdec3-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-rzqdh\" (UID: \"5bee8068-67de-4d26-9509-c9777f7cdec3\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzqdh"
Sep 17 16:57:52.153973 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:52.153922    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkk5m\" (UniqueName: \"kubernetes.io/projected/5bee8068-67de-4d26-9509-c9777f7cdec3-kube-api-access-zkk5m\") pod \"fc-smoke-test-kserve-7f4687897-rzqdh\" (UID: \"5bee8068-67de-4d26-9509-c9777f7cdec3\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzqdh"
Sep 17 16:57:52.153973 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:52.153961    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bee8068-67de-4d26-9509-c9777f7cdec3-home\") pod \"fc-smoke-test-kserve-7f4687897-rzqdh\" (UID: \"5bee8068-67de-4d26-9509-c9777f7cdec3\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzqdh"
Sep 17 16:57:52.154201 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:52.153983    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bee8068-67de-4d26-9509-c9777f7cdec3-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-rzqdh\" (UID: \"5bee8068-67de-4d26-9509-c9777f7cdec3\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzqdh"
Sep 17 16:57:52.154201 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:52.154023    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bee8068-67de-4d26-9509-c9777f7cdec3-dshm\") pod \"fc-smoke-test-kserve-7f4687897-rzqdh\" (UID: \"5bee8068-67de-4d26-9509-c9777f7cdec3\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzqdh"
Sep 17 16:57:52.154201 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:52.154047    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bee8068-67de-4d26-9509-c9777f7cdec3-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-rzqdh\" (UID: \"5bee8068-67de-4d26-9509-c9777f7cdec3\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzqdh"
Sep 17 16:57:52.154354 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:52.154328    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bee8068-67de-4d26-9509-c9777f7cdec3-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-rzqdh\" (UID: \"5bee8068-67de-4d26-9509-c9777f7cdec3\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzqdh"
Sep 17 16:57:52.154411 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:52.154386    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bee8068-67de-4d26-9509-c9777f7cdec3-home\") pod \"fc-smoke-test-kserve-7f4687897-rzqdh\" (UID: \"5bee8068-67de-4d26-9509-c9777f7cdec3\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzqdh"
Sep 17 16:57:52.154492 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:52.154408    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bee8068-67de-4d26-9509-c9777f7cdec3-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-rzqdh\" (UID: \"5bee8068-67de-4d26-9509-c9777f7cdec3\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzqdh"
Sep 17 16:57:52.156445 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:52.156408    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bee8068-67de-4d26-9509-c9777f7cdec3-dshm\") pod \"fc-smoke-test-kserve-7f4687897-rzqdh\" (UID: \"5bee8068-67de-4d26-9509-c9777f7cdec3\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzqdh"
Sep 17 16:57:52.156682 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:52.156666    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bee8068-67de-4d26-9509-c9777f7cdec3-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-rzqdh\" (UID: \"5bee8068-67de-4d26-9509-c9777f7cdec3\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzqdh"
Sep 17 16:57:52.167999 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:52.167973    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkk5m\" (UniqueName: \"kubernetes.io/projected/5bee8068-67de-4d26-9509-c9777f7cdec3-kube-api-access-zkk5m\") pod \"fc-smoke-test-kserve-7f4687897-rzqdh\" (UID: \"5bee8068-67de-4d26-9509-c9777f7cdec3\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzqdh"
Sep 17 16:57:52.197593 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:52.197568    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzqdh"
Sep 17 16:57:52.487780 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:52.487751    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzqdh"]
Sep 17 16:57:52.490662 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:57:52.490635    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bee8068_67de_4d26_9509_c9777f7cdec3.slice/crio-6238148c75af0f4054339640754843a207e24efba2d2a48733cbf6ce4b098077 WatchSource:0}: Error finding container 6238148c75af0f4054339640754843a207e24efba2d2a48733cbf6ce4b098077: Status 404 returned error can't find the container with id 6238148c75af0f4054339640754843a207e24efba2d2a48733cbf6ce4b098077
Sep 17 16:57:53.297007 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:53.296966    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzqdh" event={"ID":"5bee8068-67de-4d26-9509-c9777f7cdec3","Type":"ContainerStarted","Data":"6238148c75af0f4054339640754843a207e24efba2d2a48733cbf6ce4b098077"}
Sep 17 16:57:54.304530 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:54.304411    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzqdh" event={"ID":"5bee8068-67de-4d26-9509-c9777f7cdec3","Type":"ContainerStarted","Data":"4d136467e4a69983aebe560932b5eeebcc322e01ca54599a92d9db9908b70b21"}
Sep 17 16:57:54.325547 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:54.325492    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzqdh" podStartSLOduration=1.765720093 podStartE2EDuration="3.325471876s" totalImagesPullingTime="1.559751783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:57:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:57:54.323972236 +0000 UTC m=+960.218451375" watchObservedRunningTime="2026-09-17 16:57:54.325471876 +0000 UTC m=+960.219951014"
Sep 17 16:57:55.198285 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:55.198242    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzqdh"
Sep 17 16:57:56.973351 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:56.973318    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mftxq"]
Sep 17 16:57:56.976746 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:56.976721    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mftxq"
Sep 17 16:57:56.981677 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:56.981654    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 17 16:57:56.981781 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:56.981657    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 17 16:57:56.981781 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:56.981732    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 17 16:57:56.981895 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:56.981787    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-f8dng\""
Sep 17 16:57:56.981895 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:56.981851    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-bf47f\""
Sep 17 16:57:56.989033 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:56.989010    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mftxq"]
Sep 17 16:57:57.091620 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:57.091585    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc66fe0c-da88-4ff6-9f2e-99fb4be33e5b-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mftxq\" (UID: \"fc66fe0c-da88-4ff6-9f2e-99fb4be33e5b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mftxq"
Sep 17 16:57:57.091773 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:57.091650    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45xpx\" (UniqueName: \"kubernetes.io/projected/fc66fe0c-da88-4ff6-9f2e-99fb4be33e5b-kube-api-access-45xpx\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mftxq\" (UID: \"fc66fe0c-da88-4ff6-9f2e-99fb4be33e5b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mftxq"
Sep 17 16:57:57.192702 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:57.192662    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc66fe0c-da88-4ff6-9f2e-99fb4be33e5b-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mftxq\" (UID: \"fc66fe0c-da88-4ff6-9f2e-99fb4be33e5b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mftxq"
Sep 17 16:57:57.192898 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:57.192734    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45xpx\" (UniqueName: \"kubernetes.io/projected/fc66fe0c-da88-4ff6-9f2e-99fb4be33e5b-kube-api-access-45xpx\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mftxq\" (UID: \"fc66fe0c-da88-4ff6-9f2e-99fb4be33e5b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mftxq"
Sep 17 16:57:57.195347 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:57.195312    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc66fe0c-da88-4ff6-9f2e-99fb4be33e5b-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mftxq\" (UID: \"fc66fe0c-da88-4ff6-9f2e-99fb4be33e5b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mftxq"
Sep 17 16:57:57.203795 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:57.203768    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45xpx\" (UniqueName: \"kubernetes.io/projected/fc66fe0c-da88-4ff6-9f2e-99fb4be33e5b-kube-api-access-45xpx\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mftxq\" (UID: \"fc66fe0c-da88-4ff6-9f2e-99fb4be33e5b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mftxq"
Sep 17 16:57:57.286359 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:57.286288    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mftxq"
Sep 17 16:57:57.565884 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:57.565859    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mftxq"]
Sep 17 16:57:57.568150 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:57:57.568070    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc66fe0c_da88_4ff6_9f2e_99fb4be33e5b.slice/crio-80911fcfb23037b97d993b9affdd9087adfe5c540b220c89521b3dc4ab67c7db WatchSource:0}: Error finding container 80911fcfb23037b97d993b9affdd9087adfe5c540b220c89521b3dc4ab67c7db: Status 404 returned error can't find the container with id 80911fcfb23037b97d993b9affdd9087adfe5c540b220c89521b3dc4ab67c7db
Sep 17 16:57:58.330210 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:57:58.330134    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mftxq" event={"ID":"fc66fe0c-da88-4ff6-9f2e-99fb4be33e5b","Type":"ContainerStarted","Data":"80911fcfb23037b97d993b9affdd9087adfe5c540b220c89521b3dc4ab67c7db"}
Sep 17 16:58:00.346393 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:00.346357    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mftxq"
Sep 17 16:58:00.346876 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:00.346407    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mftxq" event={"ID":"fc66fe0c-da88-4ff6-9f2e-99fb4be33e5b","Type":"ContainerStarted","Data":"1ec6147215e6244401587c05ce9b6c7ba8e9cb32d8bf10349c4a2d3007282cf5"}
Sep 17 16:58:00.363035 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:00.362993    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mftxq" podStartSLOduration=2.152150505 podStartE2EDuration="4.362979464s" totalImagesPullingTime="2.210828959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:57:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:58:00.361321921 +0000 UTC m=+966.255801060" watchObservedRunningTime="2026-09-17 16:58:00.362979464 +0000 UTC m=+966.257458603"
Sep 17 16:58:02.197930 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:02.197894    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzqdh"
Sep 17 16:58:02.211138 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:02.211108    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzqdh"
Sep 17 16:58:02.362425 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:02.362400    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzqdh"
Sep 17 16:58:31.348820 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:31.348738    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mftxq"
Sep 17 16:58:33.180689 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:33.180656    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mftxq"]
Sep 17 16:58:35.340331 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:35.340299    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzqdh"]
Sep 17 16:58:35.349279 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:58:35.349245    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 16:58:35.349279 ip-10-0-141-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:58:35.349279 ip-10-0-141-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 16:58:35.349279 ip-10-0-141-66 kubenswrapper[2566]:  > containerID="4d136467e4a69983aebe560932b5eeebcc322e01ca54599a92d9db9908b70b21" cmd=["/bin/sleep","15"]
Sep 17 16:58:35.349555 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:35.349295    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzqdh" message="" err=<
Sep 17 16:58:35.349555 ip-10-0-141-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:58:35.349555 ip-10-0-141-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 16:58:35.349555 ip-10-0-141-66 kubenswrapper[2566]:  >
Sep 17 16:58:35.349555 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:58:35.349309    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 16:58:35.349555 ip-10-0-141-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:58:35.349555 ip-10-0-141-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 16:58:35.349555 ip-10-0-141-66 kubenswrapper[2566]:  > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzqdh" podUID="5bee8068-67de-4d26-9509-c9777f7cdec3" containerName="main" containerID="cri-o://4d136467e4a69983aebe560932b5eeebcc322e01ca54599a92d9db9908b70b21"
Sep 17 16:58:35.349555 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:35.349370    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzqdh" podUID="5bee8068-67de-4d26-9509-c9777f7cdec3" containerName="main" containerID="cri-o://4d136467e4a69983aebe560932b5eeebcc322e01ca54599a92d9db9908b70b21" gracePeriod=60
Sep 17 16:58:35.352366 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:35.352335    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzqdh" podUID="5bee8068-67de-4d26-9509-c9777f7cdec3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Sep 17 16:58:35.697222 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:35.697198    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzqdh"
Sep 17 16:58:35.794531 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:35.794497    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bee8068-67de-4d26-9509-c9777f7cdec3-dshm\") pod \"5bee8068-67de-4d26-9509-c9777f7cdec3\" (UID: \"5bee8068-67de-4d26-9509-c9777f7cdec3\") "
Sep 17 16:58:35.794710 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:35.794552    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bee8068-67de-4d26-9509-c9777f7cdec3-home\") pod \"5bee8068-67de-4d26-9509-c9777f7cdec3\" (UID: \"5bee8068-67de-4d26-9509-c9777f7cdec3\") "
Sep 17 16:58:35.794710 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:35.794576    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bee8068-67de-4d26-9509-c9777f7cdec3-tmp-dir\") pod \"5bee8068-67de-4d26-9509-c9777f7cdec3\" (UID: \"5bee8068-67de-4d26-9509-c9777f7cdec3\") "
Sep 17 16:58:35.794710 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:35.794609    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bee8068-67de-4d26-9509-c9777f7cdec3-model-cache\") pod \"5bee8068-67de-4d26-9509-c9777f7cdec3\" (UID: \"5bee8068-67de-4d26-9509-c9777f7cdec3\") "
Sep 17 16:58:35.794710 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:35.794665    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkk5m\" (UniqueName: \"kubernetes.io/projected/5bee8068-67de-4d26-9509-c9777f7cdec3-kube-api-access-zkk5m\") pod \"5bee8068-67de-4d26-9509-c9777f7cdec3\" (UID: \"5bee8068-67de-4d26-9509-c9777f7cdec3\") "
Sep 17 16:58:35.794710 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:35.794704    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bee8068-67de-4d26-9509-c9777f7cdec3-tls-certs\") pod \"5bee8068-67de-4d26-9509-c9777f7cdec3\" (UID: \"5bee8068-67de-4d26-9509-c9777f7cdec3\") "
Sep 17 16:58:35.794966 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:35.794874    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bee8068-67de-4d26-9509-c9777f7cdec3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5bee8068-67de-4d26-9509-c9777f7cdec3" (UID: "5bee8068-67de-4d26-9509-c9777f7cdec3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:58:35.794966 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:35.794890    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bee8068-67de-4d26-9509-c9777f7cdec3-home" (OuterVolumeSpecName: "home") pod "5bee8068-67de-4d26-9509-c9777f7cdec3" (UID: "5bee8068-67de-4d26-9509-c9777f7cdec3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:58:35.794966 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:35.794913    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bee8068-67de-4d26-9509-c9777f7cdec3-model-cache" (OuterVolumeSpecName: "model-cache") pod "5bee8068-67de-4d26-9509-c9777f7cdec3" (UID: "5bee8068-67de-4d26-9509-c9777f7cdec3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:58:35.796965 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:35.796936    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bee8068-67de-4d26-9509-c9777f7cdec3-dshm" (OuterVolumeSpecName: "dshm") pod "5bee8068-67de-4d26-9509-c9777f7cdec3" (UID: "5bee8068-67de-4d26-9509-c9777f7cdec3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:58:35.797049 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:35.796971    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bee8068-67de-4d26-9509-c9777f7cdec3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5bee8068-67de-4d26-9509-c9777f7cdec3" (UID: "5bee8068-67de-4d26-9509-c9777f7cdec3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:58:35.797087 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:35.797056    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bee8068-67de-4d26-9509-c9777f7cdec3-kube-api-access-zkk5m" (OuterVolumeSpecName: "kube-api-access-zkk5m") pod "5bee8068-67de-4d26-9509-c9777f7cdec3" (UID: "5bee8068-67de-4d26-9509-c9777f7cdec3"). InnerVolumeSpecName "kube-api-access-zkk5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:58:35.896335 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:35.896243    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bee8068-67de-4d26-9509-c9777f7cdec3-home\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:58:35.896335 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:35.896272    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bee8068-67de-4d26-9509-c9777f7cdec3-tmp-dir\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:58:35.896335 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:35.896285    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bee8068-67de-4d26-9509-c9777f7cdec3-model-cache\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:58:35.896335 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:35.896297    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkk5m\" (UniqueName: \"kubernetes.io/projected/5bee8068-67de-4d26-9509-c9777f7cdec3-kube-api-access-zkk5m\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:58:35.896335 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:35.896308    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bee8068-67de-4d26-9509-c9777f7cdec3-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:58:35.896335 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:35.896319    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bee8068-67de-4d26-9509-c9777f7cdec3-dshm\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:58:36.520621 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:36.520581    2566 generic.go:356] "Generic (PLEG): container finished" podID="5bee8068-67de-4d26-9509-c9777f7cdec3" containerID="4d136467e4a69983aebe560932b5eeebcc322e01ca54599a92d9db9908b70b21" exitCode=0
Sep 17 16:58:36.521109 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:36.520644    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzqdh"
Sep 17 16:58:36.521109 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:36.520666    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzqdh" event={"ID":"5bee8068-67de-4d26-9509-c9777f7cdec3","Type":"ContainerDied","Data":"4d136467e4a69983aebe560932b5eeebcc322e01ca54599a92d9db9908b70b21"}
Sep 17 16:58:36.521109 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:36.520728    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzqdh" event={"ID":"5bee8068-67de-4d26-9509-c9777f7cdec3","Type":"ContainerDied","Data":"6238148c75af0f4054339640754843a207e24efba2d2a48733cbf6ce4b098077"}
Sep 17 16:58:36.521109 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:36.520750    2566 scope.go:117] "RemoveContainer" containerID="4d136467e4a69983aebe560932b5eeebcc322e01ca54599a92d9db9908b70b21"
Sep 17 16:58:36.532043 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:36.532028    2566 scope.go:117] "RemoveContainer" containerID="4d136467e4a69983aebe560932b5eeebcc322e01ca54599a92d9db9908b70b21"
Sep 17 16:58:36.532334 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:58:36.532316    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d136467e4a69983aebe560932b5eeebcc322e01ca54599a92d9db9908b70b21\": container with ID starting with 4d136467e4a69983aebe560932b5eeebcc322e01ca54599a92d9db9908b70b21 not found: ID does not exist" containerID="4d136467e4a69983aebe560932b5eeebcc322e01ca54599a92d9db9908b70b21"
Sep 17 16:58:36.532413 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:36.532341    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d136467e4a69983aebe560932b5eeebcc322e01ca54599a92d9db9908b70b21"} err="failed to get container status \"4d136467e4a69983aebe560932b5eeebcc322e01ca54599a92d9db9908b70b21\": rpc error: code = NotFound desc = could not find container \"4d136467e4a69983aebe560932b5eeebcc322e01ca54599a92d9db9908b70b21\": container with ID starting with 4d136467e4a69983aebe560932b5eeebcc322e01ca54599a92d9db9908b70b21 not found: ID does not exist"
Sep 17 16:58:36.546298 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:36.546277    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzqdh"]
Sep 17 16:58:36.554673 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:36.554651    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzqdh"]
Sep 17 16:58:38.346165 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:38.346123    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bee8068-67de-4d26-9509-c9777f7cdec3" path="/var/lib/kubelet/pods/5bee8068-67de-4d26-9509-c9777f7cdec3/volumes"
Sep 17 16:58:41.348457 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:41.348401    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mftxq" podUID="fc66fe0c-da88-4ff6-9f2e-99fb4be33e5b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:58:44.103061 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:44.102999    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mftxq" message="" err="command '/bin/sleep 15' exited with 137: "
Sep 17 16:58:44.103061 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:58:44.103033    2566 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mftxq" podUID="fc66fe0c-da88-4ff6-9f2e-99fb4be33e5b" containerName="main" containerID="cri-o://1ec6147215e6244401587c05ce9b6c7ba8e9cb32d8bf10349c4a2d3007282cf5"
Sep 17 16:58:44.103337 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:44.103091    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mftxq" podUID="fc66fe0c-da88-4ff6-9f2e-99fb4be33e5b" containerName="main" containerID="cri-o://1ec6147215e6244401587c05ce9b6c7ba8e9cb32d8bf10349c4a2d3007282cf5" gracePeriod=50
Sep 17 16:58:44.244378 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:44.244353    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mftxq_fc66fe0c-da88-4ff6-9f2e-99fb4be33e5b/main/0.log"
Sep 17 16:58:44.246895 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:44.246879    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mftxq"
Sep 17 16:58:44.364101 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:44.364028    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45xpx\" (UniqueName: \"kubernetes.io/projected/fc66fe0c-da88-4ff6-9f2e-99fb4be33e5b-kube-api-access-45xpx\") pod \"fc66fe0c-da88-4ff6-9f2e-99fb4be33e5b\" (UID: \"fc66fe0c-da88-4ff6-9f2e-99fb4be33e5b\") "
Sep 17 16:58:44.364101 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:44.364072    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc66fe0c-da88-4ff6-9f2e-99fb4be33e5b-tls-certs\") pod \"fc66fe0c-da88-4ff6-9f2e-99fb4be33e5b\" (UID: \"fc66fe0c-da88-4ff6-9f2e-99fb4be33e5b\") "
Sep 17 16:58:44.366344 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:44.366315    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc66fe0c-da88-4ff6-9f2e-99fb4be33e5b-kube-api-access-45xpx" (OuterVolumeSpecName: "kube-api-access-45xpx") pod "fc66fe0c-da88-4ff6-9f2e-99fb4be33e5b" (UID: "fc66fe0c-da88-4ff6-9f2e-99fb4be33e5b"). InnerVolumeSpecName "kube-api-access-45xpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:58:44.366344 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:44.366344    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc66fe0c-da88-4ff6-9f2e-99fb4be33e5b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fc66fe0c-da88-4ff6-9f2e-99fb4be33e5b" (UID: "fc66fe0c-da88-4ff6-9f2e-99fb4be33e5b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:58:44.464848 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:44.464819    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45xpx\" (UniqueName: \"kubernetes.io/projected/fc66fe0c-da88-4ff6-9f2e-99fb4be33e5b-kube-api-access-45xpx\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:58:44.464848 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:44.464842    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc66fe0c-da88-4ff6-9f2e-99fb4be33e5b-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 16:58:44.560621 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:44.560598    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mftxq_fc66fe0c-da88-4ff6-9f2e-99fb4be33e5b/main/0.log"
Sep 17 16:58:44.560775 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:44.560635    2566 generic.go:356] "Generic (PLEG): container finished" podID="fc66fe0c-da88-4ff6-9f2e-99fb4be33e5b" containerID="1ec6147215e6244401587c05ce9b6c7ba8e9cb32d8bf10349c4a2d3007282cf5" exitCode=1
Sep 17 16:58:44.560775 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:44.560668    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mftxq" event={"ID":"fc66fe0c-da88-4ff6-9f2e-99fb4be33e5b","Type":"ContainerDied","Data":"1ec6147215e6244401587c05ce9b6c7ba8e9cb32d8bf10349c4a2d3007282cf5"}
Sep 17 16:58:44.560775 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:44.560702    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mftxq"
Sep 17 16:58:44.560775 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:44.560719    2566 scope.go:117] "RemoveContainer" containerID="1ec6147215e6244401587c05ce9b6c7ba8e9cb32d8bf10349c4a2d3007282cf5"
Sep 17 16:58:44.560985 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:44.560707    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mftxq" event={"ID":"fc66fe0c-da88-4ff6-9f2e-99fb4be33e5b","Type":"ContainerDied","Data":"80911fcfb23037b97d993b9affdd9087adfe5c540b220c89521b3dc4ab67c7db"}
Sep 17 16:58:44.569169 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:44.569153    2566 scope.go:117] "RemoveContainer" containerID="1ec6147215e6244401587c05ce9b6c7ba8e9cb32d8bf10349c4a2d3007282cf5"
Sep 17 16:58:44.569429 ip-10-0-141-66 kubenswrapper[2566]: E0917 16:58:44.569415    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ec6147215e6244401587c05ce9b6c7ba8e9cb32d8bf10349c4a2d3007282cf5\": container with ID starting with 1ec6147215e6244401587c05ce9b6c7ba8e9cb32d8bf10349c4a2d3007282cf5 not found: ID does not exist" containerID="1ec6147215e6244401587c05ce9b6c7ba8e9cb32d8bf10349c4a2d3007282cf5"
Sep 17 16:58:44.569497 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:44.569459    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ec6147215e6244401587c05ce9b6c7ba8e9cb32d8bf10349c4a2d3007282cf5"} err="failed to get container status \"1ec6147215e6244401587c05ce9b6c7ba8e9cb32d8bf10349c4a2d3007282cf5\": rpc error: code = NotFound desc = could not find container \"1ec6147215e6244401587c05ce9b6c7ba8e9cb32d8bf10349c4a2d3007282cf5\": container with ID starting with 1ec6147215e6244401587c05ce9b6c7ba8e9cb32d8bf10349c4a2d3007282cf5 not found: ID does not exist"
Sep 17 16:58:44.583915 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:44.583892    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mftxq"]
Sep 17 16:58:44.588246 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:44.588226    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mftxq"]
Sep 17 16:58:46.346230 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:58:46.346198    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc66fe0c-da88-4ff6-9f2e-99fb4be33e5b" path="/var/lib/kubelet/pods/fc66fe0c-da88-4ff6-9f2e-99fb4be33e5b/volumes"
Sep 17 16:59:33.074135 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:33.074102    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xw6sk"]
Sep 17 16:59:33.074757 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:33.074616    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc66fe0c-da88-4ff6-9f2e-99fb4be33e5b" containerName="main"
Sep 17 16:59:33.074757 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:33.074636    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc66fe0c-da88-4ff6-9f2e-99fb4be33e5b" containerName="main"
Sep 17 16:59:33.074757 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:33.074655    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bee8068-67de-4d26-9509-c9777f7cdec3" containerName="main"
Sep 17 16:59:33.074757 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:33.074663    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bee8068-67de-4d26-9509-c9777f7cdec3" containerName="main"
Sep 17 16:59:33.074757 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:33.074744    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc66fe0c-da88-4ff6-9f2e-99fb4be33e5b" containerName="main"
Sep 17 16:59:33.074757 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:33.074756    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bee8068-67de-4d26-9509-c9777f7cdec3" containerName="main"
Sep 17 16:59:33.077756 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:33.077733    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xw6sk"
Sep 17 16:59:33.080732 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:33.080713    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 17 16:59:33.080843 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:33.080714    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 17 16:59:33.081697 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:33.081681    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-4xkr9\""
Sep 17 16:59:33.081802 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:33.081700    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 17 16:59:33.089068 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:33.089048    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xw6sk"]
Sep 17 16:59:33.136188 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:33.136155    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlk49\" (UniqueName: \"kubernetes.io/projected/cfd0f37c-bcb7-4bef-a626-cc142dadcff5-kube-api-access-qlk49\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-xw6sk\" (UID: \"cfd0f37c-bcb7-4bef-a626-cc142dadcff5\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xw6sk"
Sep 17 16:59:33.136321 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:33.136196    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfd0f37c-bcb7-4bef-a626-cc142dadcff5-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-xw6sk\" (UID: \"cfd0f37c-bcb7-4bef-a626-cc142dadcff5\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xw6sk"
Sep 17 16:59:33.136321 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:33.136240    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfd0f37c-bcb7-4bef-a626-cc142dadcff5-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-xw6sk\" (UID: \"cfd0f37c-bcb7-4bef-a626-cc142dadcff5\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xw6sk"
Sep 17 16:59:33.136321 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:33.136272    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfd0f37c-bcb7-4bef-a626-cc142dadcff5-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-xw6sk\" (UID: \"cfd0f37c-bcb7-4bef-a626-cc142dadcff5\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xw6sk"
Sep 17 16:59:33.136321 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:33.136290    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfd0f37c-bcb7-4bef-a626-cc142dadcff5-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-xw6sk\" (UID: \"cfd0f37c-bcb7-4bef-a626-cc142dadcff5\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xw6sk"
Sep 17 16:59:33.136502 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:33.136349    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfd0f37c-bcb7-4bef-a626-cc142dadcff5-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-xw6sk\" (UID: \"cfd0f37c-bcb7-4bef-a626-cc142dadcff5\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xw6sk"
Sep 17 16:59:33.237245 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:33.237209    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfd0f37c-bcb7-4bef-a626-cc142dadcff5-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-xw6sk\" (UID: \"cfd0f37c-bcb7-4bef-a626-cc142dadcff5\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xw6sk"
Sep 17 16:59:33.237462 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:33.237257    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfd0f37c-bcb7-4bef-a626-cc142dadcff5-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-xw6sk\" (UID: \"cfd0f37c-bcb7-4bef-a626-cc142dadcff5\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xw6sk"
Sep 17 16:59:33.237462 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:33.237306    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfd0f37c-bcb7-4bef-a626-cc142dadcff5-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-xw6sk\" (UID: \"cfd0f37c-bcb7-4bef-a626-cc142dadcff5\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xw6sk"
Sep 17 16:59:33.237462 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:33.237346    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfd0f37c-bcb7-4bef-a626-cc142dadcff5-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-xw6sk\" (UID: \"cfd0f37c-bcb7-4bef-a626-cc142dadcff5\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xw6sk"
Sep 17 16:59:33.237462 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:33.237379    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfd0f37c-bcb7-4bef-a626-cc142dadcff5-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-xw6sk\" (UID: \"cfd0f37c-bcb7-4bef-a626-cc142dadcff5\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xw6sk"
Sep 17 16:59:33.237462 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:33.237412    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlk49\" (UniqueName: \"kubernetes.io/projected/cfd0f37c-bcb7-4bef-a626-cc142dadcff5-kube-api-access-qlk49\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-xw6sk\" (UID: \"cfd0f37c-bcb7-4bef-a626-cc142dadcff5\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xw6sk"
Sep 17 16:59:33.237798 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:33.237772    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfd0f37c-bcb7-4bef-a626-cc142dadcff5-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-xw6sk\" (UID: \"cfd0f37c-bcb7-4bef-a626-cc142dadcff5\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xw6sk"
Sep 17 16:59:33.237874 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:33.237841    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfd0f37c-bcb7-4bef-a626-cc142dadcff5-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-xw6sk\" (UID: \"cfd0f37c-bcb7-4bef-a626-cc142dadcff5\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xw6sk"
Sep 17 16:59:33.237874 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:33.237856    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfd0f37c-bcb7-4bef-a626-cc142dadcff5-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-xw6sk\" (UID: \"cfd0f37c-bcb7-4bef-a626-cc142dadcff5\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xw6sk"
Sep 17 16:59:33.239816 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:33.239797    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfd0f37c-bcb7-4bef-a626-cc142dadcff5-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-xw6sk\" (UID: \"cfd0f37c-bcb7-4bef-a626-cc142dadcff5\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xw6sk"
Sep 17 16:59:33.239984 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:33.239968    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfd0f37c-bcb7-4bef-a626-cc142dadcff5-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-xw6sk\" (UID: \"cfd0f37c-bcb7-4bef-a626-cc142dadcff5\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xw6sk"
Sep 17 16:59:33.246910 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:33.246885    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlk49\" (UniqueName: \"kubernetes.io/projected/cfd0f37c-bcb7-4bef-a626-cc142dadcff5-kube-api-access-qlk49\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-xw6sk\" (UID: \"cfd0f37c-bcb7-4bef-a626-cc142dadcff5\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xw6sk"
Sep 17 16:59:33.387400 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:33.387324    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xw6sk"
Sep 17 16:59:33.666116 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:33.666084    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xw6sk"]
Sep 17 16:59:33.667160 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:59:33.667123    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfd0f37c_bcb7_4bef_a626_cc142dadcff5.slice/crio-8eb15becdee359c63c61f83322fae7be347ea7a6c8b2ac0a29d66e738c1d922b WatchSource:0}: Error finding container 8eb15becdee359c63c61f83322fae7be347ea7a6c8b2ac0a29d66e738c1d922b: Status 404 returned error can't find the container with id 8eb15becdee359c63c61f83322fae7be347ea7a6c8b2ac0a29d66e738c1d922b
Sep 17 16:59:33.811748 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:33.811709    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xw6sk" event={"ID":"cfd0f37c-bcb7-4bef-a626-cc142dadcff5","Type":"ContainerStarted","Data":"15223c220a6e36761c58ba185bd42af7e725b6b9398dce0645a5b45667076045"}
Sep 17 16:59:33.811905 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:33.811763    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xw6sk" event={"ID":"cfd0f37c-bcb7-4bef-a626-cc142dadcff5","Type":"ContainerStarted","Data":"8eb15becdee359c63c61f83322fae7be347ea7a6c8b2ac0a29d66e738c1d922b"}
Sep 17 16:59:33.832844 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:33.832795    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xw6sk" podStartSLOduration=0.832779934 podStartE2EDuration="832.779934ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:59:33.832509 +0000 UTC m=+1059.726988143" watchObservedRunningTime="2026-09-17 16:59:33.832779934 +0000 UTC m=+1059.727259076"
Sep 17 16:59:34.388314 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:34.388225    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xw6sk"
Sep 17 16:59:43.388120 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:43.388089    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xw6sk"
Sep 17 16:59:43.400477 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:43.400453    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xw6sk"
Sep 17 16:59:43.872743 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:43.872716    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xw6sk"
Sep 17 16:59:47.443774 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:47.443741    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-hrp9b"]
Sep 17 16:59:47.447011 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:47.446992    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-hrp9b"
Sep 17 16:59:47.449552 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:47.449528    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 17 16:59:47.450299 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:47.450277    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 17 16:59:47.450403 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:47.450372    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-8hmbd\""
Sep 17 16:59:47.450486 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:47.450464    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-epp-sa-dockercfg-d5428\""
Sep 17 16:59:47.450486 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:47.450474    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 17 16:59:47.459684 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:47.459662    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-hrp9b"]
Sep 17 16:59:47.537879 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:47.537843    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhl7g\" (UniqueName: \"kubernetes.io/projected/11252a51-d76e-44eb-89c1-0f69d4cc0e03-kube-api-access-mhl7g\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-hrp9b\" (UID: \"11252a51-d76e-44eb-89c1-0f69d4cc0e03\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-hrp9b"
Sep 17 16:59:47.538017 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:47.537886    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11252a51-d76e-44eb-89c1-0f69d4cc0e03-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-hrp9b\" (UID: \"11252a51-d76e-44eb-89c1-0f69d4cc0e03\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-hrp9b"
Sep 17 16:59:47.639116 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:47.639080    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhl7g\" (UniqueName: \"kubernetes.io/projected/11252a51-d76e-44eb-89c1-0f69d4cc0e03-kube-api-access-mhl7g\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-hrp9b\" (UID: \"11252a51-d76e-44eb-89c1-0f69d4cc0e03\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-hrp9b"
Sep 17 16:59:47.639270 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:47.639128    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11252a51-d76e-44eb-89c1-0f69d4cc0e03-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-hrp9b\" (UID: \"11252a51-d76e-44eb-89c1-0f69d4cc0e03\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-hrp9b"
Sep 17 16:59:47.641816 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:47.641792    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11252a51-d76e-44eb-89c1-0f69d4cc0e03-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-hrp9b\" (UID: \"11252a51-d76e-44eb-89c1-0f69d4cc0e03\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-hrp9b"
Sep 17 16:59:47.648566 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:47.648547    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhl7g\" (UniqueName: \"kubernetes.io/projected/11252a51-d76e-44eb-89c1-0f69d4cc0e03-kube-api-access-mhl7g\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-hrp9b\" (UID: \"11252a51-d76e-44eb-89c1-0f69d4cc0e03\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-hrp9b"
Sep 17 16:59:47.759315 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:47.759234    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-hrp9b"
Sep 17 16:59:48.035784 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:48.035755    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-hrp9b"]
Sep 17 16:59:48.037669 ip-10-0-141-66 kubenswrapper[2566]: W0917 16:59:48.037638    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11252a51_d76e_44eb_89c1_0f69d4cc0e03.slice/crio-97359286f58a1b4c6413d0caf078af95b263a8ba1f2634de8495d5b84e2059c7 WatchSource:0}: Error finding container 97359286f58a1b4c6413d0caf078af95b263a8ba1f2634de8495d5b84e2059c7: Status 404 returned error can't find the container with id 97359286f58a1b4c6413d0caf078af95b263a8ba1f2634de8495d5b84e2059c7
Sep 17 16:59:48.890630 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:48.890599    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-hrp9b" event={"ID":"11252a51-d76e-44eb-89c1-0f69d4cc0e03","Type":"ContainerStarted","Data":"ae6255fd17044ce0cdfa5d72a5d0429232290f90ec2fd5c8bae9ba2ad7a6caed"}
Sep 17 16:59:48.891018 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:48.890647    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-hrp9b" event={"ID":"11252a51-d76e-44eb-89c1-0f69d4cc0e03","Type":"ContainerStarted","Data":"97359286f58a1b4c6413d0caf078af95b263a8ba1f2634de8495d5b84e2059c7"}
Sep 17 16:59:48.891018 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:48.890676    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-hrp9b"
Sep 17 16:59:48.913945 ip-10-0-141-66 kubenswrapper[2566]: I0917 16:59:48.913894    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-hrp9b" podStartSLOduration=1.913879009 podStartE2EDuration="1.913879009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:59:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:59:48.912081085 +0000 UTC m=+1074.806560224" watchObservedRunningTime="2026-09-17 16:59:48.913879009 +0000 UTC m=+1074.808358147"
Sep 17 17:00:11.515253 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:00:11.515219    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xw6sk"]
Sep 17 17:00:11.524511 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:00:11.524479    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:00:11.524511 ip-10-0-141-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:00:11.524511 ip-10-0-141-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:00:11.524511 ip-10-0-141-66 kubenswrapper[2566]:  > containerID="15223c220a6e36761c58ba185bd42af7e725b6b9398dce0645a5b45667076045" cmd=["/bin/sleep","15"]
Sep 17 17:00:11.524726 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:00:11.524528    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xw6sk" message="" err=<
Sep 17 17:00:11.524726 ip-10-0-141-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:00:11.524726 ip-10-0-141-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:00:11.524726 ip-10-0-141-66 kubenswrapper[2566]:  >
Sep 17 17:00:11.524726 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:00:11.524540    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:00:11.524726 ip-10-0-141-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:00:11.524726 ip-10-0-141-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:00:11.524726 ip-10-0-141-66 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xw6sk" podUID="cfd0f37c-bcb7-4bef-a626-cc142dadcff5" containerName="main" containerID="cri-o://15223c220a6e36761c58ba185bd42af7e725b6b9398dce0645a5b45667076045"
Sep 17 17:00:11.524726 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:00:11.524584    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xw6sk" podUID="cfd0f37c-bcb7-4bef-a626-cc142dadcff5" containerName="main" containerID="cri-o://15223c220a6e36761c58ba185bd42af7e725b6b9398dce0645a5b45667076045" gracePeriod=60
Sep 17 17:00:11.878692 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:00:11.878667    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xw6sk"
Sep 17 17:00:11.936623 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:00:11.936587    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfd0f37c-bcb7-4bef-a626-cc142dadcff5-tmp-dir\") pod \"cfd0f37c-bcb7-4bef-a626-cc142dadcff5\" (UID: \"cfd0f37c-bcb7-4bef-a626-cc142dadcff5\") "
Sep 17 17:00:11.936800 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:00:11.936641    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfd0f37c-bcb7-4bef-a626-cc142dadcff5-tls-certs\") pod \"cfd0f37c-bcb7-4bef-a626-cc142dadcff5\" (UID: \"cfd0f37c-bcb7-4bef-a626-cc142dadcff5\") "
Sep 17 17:00:11.936800 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:00:11.936689    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfd0f37c-bcb7-4bef-a626-cc142dadcff5-home\") pod \"cfd0f37c-bcb7-4bef-a626-cc142dadcff5\" (UID: \"cfd0f37c-bcb7-4bef-a626-cc142dadcff5\") "
Sep 17 17:00:11.936800 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:00:11.936713    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfd0f37c-bcb7-4bef-a626-cc142dadcff5-model-cache\") pod \"cfd0f37c-bcb7-4bef-a626-cc142dadcff5\" (UID: \"cfd0f37c-bcb7-4bef-a626-cc142dadcff5\") "
Sep 17 17:00:11.936800 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:00:11.936770    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfd0f37c-bcb7-4bef-a626-cc142dadcff5-dshm\") pod \"cfd0f37c-bcb7-4bef-a626-cc142dadcff5\" (UID: \"cfd0f37c-bcb7-4bef-a626-cc142dadcff5\") "
Sep 17 17:00:11.937002 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:00:11.936809    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlk49\" (UniqueName: \"kubernetes.io/projected/cfd0f37c-bcb7-4bef-a626-cc142dadcff5-kube-api-access-qlk49\") pod \"cfd0f37c-bcb7-4bef-a626-cc142dadcff5\" (UID: \"cfd0f37c-bcb7-4bef-a626-cc142dadcff5\") "
Sep 17 17:00:11.937002 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:00:11.936862    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd0f37c-bcb7-4bef-a626-cc142dadcff5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cfd0f37c-bcb7-4bef-a626-cc142dadcff5" (UID: "cfd0f37c-bcb7-4bef-a626-cc142dadcff5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:00:11.937002 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:00:11.936962    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd0f37c-bcb7-4bef-a626-cc142dadcff5-home" (OuterVolumeSpecName: "home") pod "cfd0f37c-bcb7-4bef-a626-cc142dadcff5" (UID: "cfd0f37c-bcb7-4bef-a626-cc142dadcff5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:00:11.937140 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:00:11.937062    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd0f37c-bcb7-4bef-a626-cc142dadcff5-model-cache" (OuterVolumeSpecName: "model-cache") pod "cfd0f37c-bcb7-4bef-a626-cc142dadcff5" (UID: "cfd0f37c-bcb7-4bef-a626-cc142dadcff5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:00:11.937140 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:00:11.937113    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfd0f37c-bcb7-4bef-a626-cc142dadcff5-tmp-dir\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:00:11.937140 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:00:11.937135    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfd0f37c-bcb7-4bef-a626-cc142dadcff5-home\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:00:11.939139 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:00:11.939108    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd0f37c-bcb7-4bef-a626-cc142dadcff5-dshm" (OuterVolumeSpecName: "dshm") pod "cfd0f37c-bcb7-4bef-a626-cc142dadcff5" (UID: "cfd0f37c-bcb7-4bef-a626-cc142dadcff5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:00:11.939248 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:00:11.939155    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfd0f37c-bcb7-4bef-a626-cc142dadcff5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cfd0f37c-bcb7-4bef-a626-cc142dadcff5" (UID: "cfd0f37c-bcb7-4bef-a626-cc142dadcff5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:00:11.939248 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:00:11.939177    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfd0f37c-bcb7-4bef-a626-cc142dadcff5-kube-api-access-qlk49" (OuterVolumeSpecName: "kube-api-access-qlk49") pod "cfd0f37c-bcb7-4bef-a626-cc142dadcff5" (UID: "cfd0f37c-bcb7-4bef-a626-cc142dadcff5"). InnerVolumeSpecName "kube-api-access-qlk49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:00:12.010841 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:00:12.010807    2566 generic.go:356] "Generic (PLEG): container finished" podID="cfd0f37c-bcb7-4bef-a626-cc142dadcff5" containerID="15223c220a6e36761c58ba185bd42af7e725b6b9398dce0645a5b45667076045" exitCode=0
Sep 17 17:00:12.011008 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:00:12.010853    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xw6sk"
Sep 17 17:00:12.011008 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:00:12.010891    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xw6sk" event={"ID":"cfd0f37c-bcb7-4bef-a626-cc142dadcff5","Type":"ContainerDied","Data":"15223c220a6e36761c58ba185bd42af7e725b6b9398dce0645a5b45667076045"}
Sep 17 17:00:12.011008 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:00:12.010945    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xw6sk" event={"ID":"cfd0f37c-bcb7-4bef-a626-cc142dadcff5","Type":"ContainerDied","Data":"8eb15becdee359c63c61f83322fae7be347ea7a6c8b2ac0a29d66e738c1d922b"}
Sep 17 17:00:12.011008 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:00:12.010967    2566 scope.go:117] "RemoveContainer" containerID="15223c220a6e36761c58ba185bd42af7e725b6b9398dce0645a5b45667076045"
Sep 17 17:00:12.020224 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:00:12.020205    2566 scope.go:117] "RemoveContainer" containerID="15223c220a6e36761c58ba185bd42af7e725b6b9398dce0645a5b45667076045"
Sep 17 17:00:12.020495 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:00:12.020476    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15223c220a6e36761c58ba185bd42af7e725b6b9398dce0645a5b45667076045\": container with ID starting with 15223c220a6e36761c58ba185bd42af7e725b6b9398dce0645a5b45667076045 not found: ID does not exist" containerID="15223c220a6e36761c58ba185bd42af7e725b6b9398dce0645a5b45667076045"
Sep 17 17:00:12.020554 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:00:12.020503    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15223c220a6e36761c58ba185bd42af7e725b6b9398dce0645a5b45667076045"} err="failed to get container status \"15223c220a6e36761c58ba185bd42af7e725b6b9398dce0645a5b45667076045\": rpc error: code = NotFound desc = could not find container \"15223c220a6e36761c58ba185bd42af7e725b6b9398dce0645a5b45667076045\": container with ID starting with 15223c220a6e36761c58ba185bd42af7e725b6b9398dce0645a5b45667076045 not found: ID does not exist"
Sep 17 17:00:12.037203 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:00:12.037139    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xw6sk"]
Sep 17 17:00:12.037473 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:00:12.037460    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfd0f37c-bcb7-4bef-a626-cc142dadcff5-dshm\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:00:12.037533 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:00:12.037480    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlk49\" (UniqueName: \"kubernetes.io/projected/cfd0f37c-bcb7-4bef-a626-cc142dadcff5-kube-api-access-qlk49\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:00:12.037533 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:00:12.037494    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfd0f37c-bcb7-4bef-a626-cc142dadcff5-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:00:12.037533 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:00:12.037507    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfd0f37c-bcb7-4bef-a626-cc142dadcff5-model-cache\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:00:12.042873 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:00:12.042849    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xw6sk"]
Sep 17 17:00:12.346848 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:00:12.346812    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfd0f37c-bcb7-4bef-a626-cc142dadcff5" path="/var/lib/kubelet/pods/cfd0f37c-bcb7-4bef-a626-cc142dadcff5/volumes"
Sep 17 17:00:12.862657 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:00:12.862618    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xw6sk" podUID="cfd0f37c-bcb7-4bef-a626-cc142dadcff5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": context deadline exceeded"
Sep 17 17:00:19.897642 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:00:19.897612    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-hrp9b"
Sep 17 17:00:26.463706 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:00:26.463667    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-hrp9b"]
Sep 17 17:00:29.897156 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:00:29.897118    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-hrp9b" podUID="11252a51-d76e-44eb-89c1-0f69d4cc0e03" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:00:39.897568 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:00:39.897525    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-hrp9b" podUID="11252a51-d76e-44eb-89c1-0f69d4cc0e03" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:00:41.483608 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:00:41.483554    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-hrp9b" podUID="11252a51-d76e-44eb-89c1-0f69d4cc0e03" containerName="main" containerID="cri-o://ae6255fd17044ce0cdfa5d72a5d0429232290f90ec2fd5c8bae9ba2ad7a6caed" gracePeriod=45
Sep 17 17:00:49.897629 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:00:49.897593    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-hrp9b" podUID="11252a51-d76e-44eb-89c1-0f69d4cc0e03" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:00:49.898001 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:00:49.897715    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-hrp9b"
Sep 17 17:00:59.897554 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:00:59.897513    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-hrp9b" podUID="11252a51-d76e-44eb-89c1-0f69d4cc0e03" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:01:04.237520 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:04.237484    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs"]
Sep 17 17:01:04.238012 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:04.237995    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfd0f37c-bcb7-4bef-a626-cc142dadcff5" containerName="main"
Sep 17 17:01:04.238097 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:04.238014    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfd0f37c-bcb7-4bef-a626-cc142dadcff5" containerName="main"
Sep 17 17:01:04.238157 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:04.238101    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfd0f37c-bcb7-4bef-a626-cc142dadcff5" containerName="main"
Sep 17 17:01:04.242609 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:04.242587    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs"
Sep 17 17:01:04.245981 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:04.245954    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-nrxmc\""
Sep 17 17:01:04.245981 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:04.245979    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 17 17:01:04.246166 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:04.245999    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 17 17:01:04.246166 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:04.246006    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 17 17:01:04.256351 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:04.256324    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs"]
Sep 17 17:01:04.331653 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:04.331551    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8860e9bc-46c7-47fd-b243-04efc6d8fc4c-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs\" (UID: \"8860e9bc-46c7-47fd-b243-04efc6d8fc4c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs"
Sep 17 17:01:04.331806 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:04.331673    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8860e9bc-46c7-47fd-b243-04efc6d8fc4c-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs\" (UID: \"8860e9bc-46c7-47fd-b243-04efc6d8fc4c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs"
Sep 17 17:01:04.331806 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:04.331716    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8860e9bc-46c7-47fd-b243-04efc6d8fc4c-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs\" (UID: \"8860e9bc-46c7-47fd-b243-04efc6d8fc4c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs"
Sep 17 17:01:04.331806 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:04.331794    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8860e9bc-46c7-47fd-b243-04efc6d8fc4c-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs\" (UID: \"8860e9bc-46c7-47fd-b243-04efc6d8fc4c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs"
Sep 17 17:01:04.331938 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:04.331833    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn5q5\" (UniqueName: \"kubernetes.io/projected/8860e9bc-46c7-47fd-b243-04efc6d8fc4c-kube-api-access-zn5q5\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs\" (UID: \"8860e9bc-46c7-47fd-b243-04efc6d8fc4c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs"
Sep 17 17:01:04.331938 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:04.331867    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8860e9bc-46c7-47fd-b243-04efc6d8fc4c-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs\" (UID: \"8860e9bc-46c7-47fd-b243-04efc6d8fc4c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs"
Sep 17 17:01:04.432359 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:04.432286    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8860e9bc-46c7-47fd-b243-04efc6d8fc4c-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs\" (UID: \"8860e9bc-46c7-47fd-b243-04efc6d8fc4c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs"
Sep 17 17:01:04.432359 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:04.432327    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8860e9bc-46c7-47fd-b243-04efc6d8fc4c-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs\" (UID: \"8860e9bc-46c7-47fd-b243-04efc6d8fc4c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs"
Sep 17 17:01:04.432594 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:04.432367    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8860e9bc-46c7-47fd-b243-04efc6d8fc4c-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs\" (UID: \"8860e9bc-46c7-47fd-b243-04efc6d8fc4c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs"
Sep 17 17:01:04.432594 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:04.432408    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8860e9bc-46c7-47fd-b243-04efc6d8fc4c-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs\" (UID: \"8860e9bc-46c7-47fd-b243-04efc6d8fc4c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs"
Sep 17 17:01:04.432594 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:04.432465    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn5q5\" (UniqueName: \"kubernetes.io/projected/8860e9bc-46c7-47fd-b243-04efc6d8fc4c-kube-api-access-zn5q5\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs\" (UID: \"8860e9bc-46c7-47fd-b243-04efc6d8fc4c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs"
Sep 17 17:01:04.432594 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:04.432498    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8860e9bc-46c7-47fd-b243-04efc6d8fc4c-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs\" (UID: \"8860e9bc-46c7-47fd-b243-04efc6d8fc4c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs"
Sep 17 17:01:04.432785 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:04.432733    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8860e9bc-46c7-47fd-b243-04efc6d8fc4c-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs\" (UID: \"8860e9bc-46c7-47fd-b243-04efc6d8fc4c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs"
Sep 17 17:01:04.432832 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:04.432793    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8860e9bc-46c7-47fd-b243-04efc6d8fc4c-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs\" (UID: \"8860e9bc-46c7-47fd-b243-04efc6d8fc4c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs"
Sep 17 17:01:04.432917 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:04.432896    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8860e9bc-46c7-47fd-b243-04efc6d8fc4c-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs\" (UID: \"8860e9bc-46c7-47fd-b243-04efc6d8fc4c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs"
Sep 17 17:01:04.434871 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:04.434853    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8860e9bc-46c7-47fd-b243-04efc6d8fc4c-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs\" (UID: \"8860e9bc-46c7-47fd-b243-04efc6d8fc4c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs"
Sep 17 17:01:04.435019 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:04.435003    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8860e9bc-46c7-47fd-b243-04efc6d8fc4c-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs\" (UID: \"8860e9bc-46c7-47fd-b243-04efc6d8fc4c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs"
Sep 17 17:01:04.444240 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:04.444214    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn5q5\" (UniqueName: \"kubernetes.io/projected/8860e9bc-46c7-47fd-b243-04efc6d8fc4c-kube-api-access-zn5q5\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs\" (UID: \"8860e9bc-46c7-47fd-b243-04efc6d8fc4c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs"
Sep 17 17:01:04.553510 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:04.553483    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs"
Sep 17 17:01:04.832391 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:04.832362    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs"]
Sep 17 17:01:04.833220 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:01:04.833195    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8860e9bc_46c7_47fd_b243_04efc6d8fc4c.slice/crio-c0b2540ce0ffb58c435f66fcd744c8db4eed1bb76ebabaae699d69dc1a6fa33b WatchSource:0}: Error finding container c0b2540ce0ffb58c435f66fcd744c8db4eed1bb76ebabaae699d69dc1a6fa33b: Status 404 returned error can't find the container with id c0b2540ce0ffb58c435f66fcd744c8db4eed1bb76ebabaae699d69dc1a6fa33b
Sep 17 17:01:05.278998 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:05.278917    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs" event={"ID":"8860e9bc-46c7-47fd-b243-04efc6d8fc4c","Type":"ContainerStarted","Data":"0d78c089b8beac701f9054b9c587a4b252501b03be7a6e262a3a763ae903c699"}
Sep 17 17:01:05.278998 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:05.278974    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs" event={"ID":"8860e9bc-46c7-47fd-b243-04efc6d8fc4c","Type":"ContainerStarted","Data":"c0b2540ce0ffb58c435f66fcd744c8db4eed1bb76ebabaae699d69dc1a6fa33b"}
Sep 17 17:01:05.304129 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:05.304067    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs" podStartSLOduration=1.304045424 podStartE2EDuration="1.304045424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:01:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:01:05.301293168 +0000 UTC m=+1151.195772308" watchObservedRunningTime="2026-09-17 17:01:05.304045424 +0000 UTC m=+1151.198524564"
Sep 17 17:01:05.554462 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:05.554417    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs"
Sep 17 17:01:09.897251 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:09.897213    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-hrp9b" podUID="11252a51-d76e-44eb-89c1-0f69d4cc0e03" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:01:11.731154 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:11.731125    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-hrp9b"
Sep 17 17:01:11.785303 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:11.785222    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhl7g\" (UniqueName: \"kubernetes.io/projected/11252a51-d76e-44eb-89c1-0f69d4cc0e03-kube-api-access-mhl7g\") pod \"11252a51-d76e-44eb-89c1-0f69d4cc0e03\" (UID: \"11252a51-d76e-44eb-89c1-0f69d4cc0e03\") "
Sep 17 17:01:11.785303 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:11.785266    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11252a51-d76e-44eb-89c1-0f69d4cc0e03-tls-certs\") pod \"11252a51-d76e-44eb-89c1-0f69d4cc0e03\" (UID: \"11252a51-d76e-44eb-89c1-0f69d4cc0e03\") "
Sep 17 17:01:11.787508 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:11.787483    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11252a51-d76e-44eb-89c1-0f69d4cc0e03-kube-api-access-mhl7g" (OuterVolumeSpecName: "kube-api-access-mhl7g") pod "11252a51-d76e-44eb-89c1-0f69d4cc0e03" (UID: "11252a51-d76e-44eb-89c1-0f69d4cc0e03"). InnerVolumeSpecName "kube-api-access-mhl7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:01:11.787569 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:11.787499    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11252a51-d76e-44eb-89c1-0f69d4cc0e03-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "11252a51-d76e-44eb-89c1-0f69d4cc0e03" (UID: "11252a51-d76e-44eb-89c1-0f69d4cc0e03"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:01:11.886027 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:11.885986    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhl7g\" (UniqueName: \"kubernetes.io/projected/11252a51-d76e-44eb-89c1-0f69d4cc0e03-kube-api-access-mhl7g\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:01:11.886027 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:11.886022    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11252a51-d76e-44eb-89c1-0f69d4cc0e03-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:01:12.315921 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:12.315886    2566 generic.go:356] "Generic (PLEG): container finished" podID="11252a51-d76e-44eb-89c1-0f69d4cc0e03" containerID="ae6255fd17044ce0cdfa5d72a5d0429232290f90ec2fd5c8bae9ba2ad7a6caed" exitCode=0
Sep 17 17:01:12.316114 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:12.315932    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-hrp9b"
Sep 17 17:01:12.316114 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:12.315981    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-hrp9b" event={"ID":"11252a51-d76e-44eb-89c1-0f69d4cc0e03","Type":"ContainerDied","Data":"ae6255fd17044ce0cdfa5d72a5d0429232290f90ec2fd5c8bae9ba2ad7a6caed"}
Sep 17 17:01:12.316114 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:12.316042    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-hrp9b" event={"ID":"11252a51-d76e-44eb-89c1-0f69d4cc0e03","Type":"ContainerDied","Data":"97359286f58a1b4c6413d0caf078af95b263a8ba1f2634de8495d5b84e2059c7"}
Sep 17 17:01:12.316114 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:12.316073    2566 scope.go:117] "RemoveContainer" containerID="ae6255fd17044ce0cdfa5d72a5d0429232290f90ec2fd5c8bae9ba2ad7a6caed"
Sep 17 17:01:12.325945 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:12.325929    2566 scope.go:117] "RemoveContainer" containerID="ae6255fd17044ce0cdfa5d72a5d0429232290f90ec2fd5c8bae9ba2ad7a6caed"
Sep 17 17:01:12.326198 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:01:12.326181    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae6255fd17044ce0cdfa5d72a5d0429232290f90ec2fd5c8bae9ba2ad7a6caed\": container with ID starting with ae6255fd17044ce0cdfa5d72a5d0429232290f90ec2fd5c8bae9ba2ad7a6caed not found: ID does not exist" containerID="ae6255fd17044ce0cdfa5d72a5d0429232290f90ec2fd5c8bae9ba2ad7a6caed"
Sep 17 17:01:12.326250 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:12.326205    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae6255fd17044ce0cdfa5d72a5d0429232290f90ec2fd5c8bae9ba2ad7a6caed"} err="failed to get container status \"ae6255fd17044ce0cdfa5d72a5d0429232290f90ec2fd5c8bae9ba2ad7a6caed\": rpc error: code = NotFound desc = could not find container \"ae6255fd17044ce0cdfa5d72a5d0429232290f90ec2fd5c8bae9ba2ad7a6caed\": container with ID starting with ae6255fd17044ce0cdfa5d72a5d0429232290f90ec2fd5c8bae9ba2ad7a6caed not found: ID does not exist"
Sep 17 17:01:12.339407 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:12.339386    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-hrp9b"]
Sep 17 17:01:12.347034 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:12.347016    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-hrp9b"]
Sep 17 17:01:14.345927 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:14.345894    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11252a51-d76e-44eb-89c1-0f69d4cc0e03" path="/var/lib/kubelet/pods/11252a51-d76e-44eb-89c1-0f69d4cc0e03/volumes"
Sep 17 17:01:14.554179 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:14.554144    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs"
Sep 17 17:01:14.566526 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:14.566498    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs"
Sep 17 17:01:15.348544 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:15.348517    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs"
Sep 17 17:01:16.842760 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:16.842730    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t"]
Sep 17 17:01:16.843188 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:16.843069    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11252a51-d76e-44eb-89c1-0f69d4cc0e03" containerName="main"
Sep 17 17:01:16.843188 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:16.843080    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="11252a51-d76e-44eb-89c1-0f69d4cc0e03" containerName="main"
Sep 17 17:01:16.843188 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:16.843151    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="11252a51-d76e-44eb-89c1-0f69d4cc0e03" containerName="main"
Sep 17 17:01:16.847344 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:16.847322    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t"
Sep 17 17:01:16.853200 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:16.853177    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Sep 17 17:01:16.853200 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:16.853195    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Sep 17 17:01:16.853396 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:16.853215    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Sep 17 17:01:16.854421 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:16.854066    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-zzkjh\""
Sep 17 17:01:16.860090 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:16.860069    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t"]
Sep 17 17:01:16.928690 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:16.928663    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6147805b-bc03-4708-8c2e-1325291d065d-istio-token\") pod \"router-gateway-1-openshift-default-6b46477c5b-l6n8t\" (UID: \"6147805b-bc03-4708-8c2e-1325291d065d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t"
Sep 17 17:01:16.928842 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:16.928714    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6147805b-bc03-4708-8c2e-1325291d065d-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6b46477c5b-l6n8t\" (UID: \"6147805b-bc03-4708-8c2e-1325291d065d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t"
Sep 17 17:01:16.928842 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:16.928741    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6147805b-bc03-4708-8c2e-1325291d065d-credential-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-l6n8t\" (UID: \"6147805b-bc03-4708-8c2e-1325291d065d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t"
Sep 17 17:01:16.928842 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:16.928773    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcmv2\" (UniqueName: \"kubernetes.io/projected/6147805b-bc03-4708-8c2e-1325291d065d-kube-api-access-xcmv2\") pod \"router-gateway-1-openshift-default-6b46477c5b-l6n8t\" (UID: \"6147805b-bc03-4708-8c2e-1325291d065d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t"
Sep 17 17:01:16.928842 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:16.928804    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6147805b-bc03-4708-8c2e-1325291d065d-workload-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-l6n8t\" (UID: \"6147805b-bc03-4708-8c2e-1325291d065d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t"
Sep 17 17:01:16.929031 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:16.928869    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6147805b-bc03-4708-8c2e-1325291d065d-istio-podinfo\") pod \"router-gateway-1-openshift-default-6b46477c5b-l6n8t\" (UID: \"6147805b-bc03-4708-8c2e-1325291d065d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t"
Sep 17 17:01:16.929031 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:16.928901    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6147805b-bc03-4708-8c2e-1325291d065d-istio-envoy\") pod \"router-gateway-1-openshift-default-6b46477c5b-l6n8t\" (UID: \"6147805b-bc03-4708-8c2e-1325291d065d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t"
Sep 17 17:01:16.929031 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:16.928931    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6147805b-bc03-4708-8c2e-1325291d065d-workload-certs\") pod \"router-gateway-1-openshift-default-6b46477c5b-l6n8t\" (UID: \"6147805b-bc03-4708-8c2e-1325291d065d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t"
Sep 17 17:01:16.929031 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:16.928959    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6147805b-bc03-4708-8c2e-1325291d065d-istio-data\") pod \"router-gateway-1-openshift-default-6b46477c5b-l6n8t\" (UID: \"6147805b-bc03-4708-8c2e-1325291d065d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t"
Sep 17 17:01:17.030081 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:17.030049    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6147805b-bc03-4708-8c2e-1325291d065d-workload-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-l6n8t\" (UID: \"6147805b-bc03-4708-8c2e-1325291d065d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t"
Sep 17 17:01:17.030226 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:17.030092    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6147805b-bc03-4708-8c2e-1325291d065d-istio-podinfo\") pod \"router-gateway-1-openshift-default-6b46477c5b-l6n8t\" (UID: \"6147805b-bc03-4708-8c2e-1325291d065d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t"
Sep 17 17:01:17.030226 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:17.030147    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6147805b-bc03-4708-8c2e-1325291d065d-istio-envoy\") pod \"router-gateway-1-openshift-default-6b46477c5b-l6n8t\" (UID: \"6147805b-bc03-4708-8c2e-1325291d065d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t"
Sep 17 17:01:17.030226 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:17.030187    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6147805b-bc03-4708-8c2e-1325291d065d-workload-certs\") pod \"router-gateway-1-openshift-default-6b46477c5b-l6n8t\" (UID: \"6147805b-bc03-4708-8c2e-1325291d065d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t"
Sep 17 17:01:17.030226 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:17.030207    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6147805b-bc03-4708-8c2e-1325291d065d-istio-data\") pod \"router-gateway-1-openshift-default-6b46477c5b-l6n8t\" (UID: \"6147805b-bc03-4708-8c2e-1325291d065d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t"
Sep 17 17:01:17.030477 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:17.030257    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6147805b-bc03-4708-8c2e-1325291d065d-istio-token\") pod \"router-gateway-1-openshift-default-6b46477c5b-l6n8t\" (UID: \"6147805b-bc03-4708-8c2e-1325291d065d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t"
Sep 17 17:01:17.030477 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:17.030429    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6147805b-bc03-4708-8c2e-1325291d065d-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6b46477c5b-l6n8t\" (UID: \"6147805b-bc03-4708-8c2e-1325291d065d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t"
Sep 17 17:01:17.030604 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:17.030481    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6147805b-bc03-4708-8c2e-1325291d065d-credential-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-l6n8t\" (UID: \"6147805b-bc03-4708-8c2e-1325291d065d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t"
Sep 17 17:01:17.030604 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:17.030512    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcmv2\" (UniqueName: \"kubernetes.io/projected/6147805b-bc03-4708-8c2e-1325291d065d-kube-api-access-xcmv2\") pod \"router-gateway-1-openshift-default-6b46477c5b-l6n8t\" (UID: \"6147805b-bc03-4708-8c2e-1325291d065d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t"
Sep 17 17:01:17.030719 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:17.030674    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6147805b-bc03-4708-8c2e-1325291d065d-workload-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-l6n8t\" (UID: \"6147805b-bc03-4708-8c2e-1325291d065d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t"
Sep 17 17:01:17.030719 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:17.030700    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6147805b-bc03-4708-8c2e-1325291d065d-istio-data\") pod \"router-gateway-1-openshift-default-6b46477c5b-l6n8t\" (UID: \"6147805b-bc03-4708-8c2e-1325291d065d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t"
Sep 17 17:01:17.030852 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:17.030837    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6147805b-bc03-4708-8c2e-1325291d065d-credential-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-l6n8t\" (UID: \"6147805b-bc03-4708-8c2e-1325291d065d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t"
Sep 17 17:01:17.030917 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:17.030847    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6147805b-bc03-4708-8c2e-1325291d065d-workload-certs\") pod \"router-gateway-1-openshift-default-6b46477c5b-l6n8t\" (UID: \"6147805b-bc03-4708-8c2e-1325291d065d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t"
Sep 17 17:01:17.031109 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:17.031051    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6147805b-bc03-4708-8c2e-1325291d065d-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6b46477c5b-l6n8t\" (UID: \"6147805b-bc03-4708-8c2e-1325291d065d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t"
Sep 17 17:01:17.032538 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:17.032518    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6147805b-bc03-4708-8c2e-1325291d065d-istio-envoy\") pod \"router-gateway-1-openshift-default-6b46477c5b-l6n8t\" (UID: \"6147805b-bc03-4708-8c2e-1325291d065d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t"
Sep 17 17:01:17.032908 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:17.032891    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6147805b-bc03-4708-8c2e-1325291d065d-istio-podinfo\") pod \"router-gateway-1-openshift-default-6b46477c5b-l6n8t\" (UID: \"6147805b-bc03-4708-8c2e-1325291d065d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t"
Sep 17 17:01:17.047390 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:17.047363    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6147805b-bc03-4708-8c2e-1325291d065d-istio-token\") pod \"router-gateway-1-openshift-default-6b46477c5b-l6n8t\" (UID: \"6147805b-bc03-4708-8c2e-1325291d065d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t"
Sep 17 17:01:17.047525 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:17.047504    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcmv2\" (UniqueName: \"kubernetes.io/projected/6147805b-bc03-4708-8c2e-1325291d065d-kube-api-access-xcmv2\") pod \"router-gateway-1-openshift-default-6b46477c5b-l6n8t\" (UID: \"6147805b-bc03-4708-8c2e-1325291d065d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t"
Sep 17 17:01:17.158355 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:17.158272    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t"
Sep 17 17:01:17.454406 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:17.454382    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t"]
Sep 17 17:01:17.456096 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:01:17.456069    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6147805b_bc03_4708_8c2e_1325291d065d.slice/crio-ad0a12a021da6cd0276dc16d1a0519a7905234474b49eabba78469792158c9fc WatchSource:0}: Error finding container ad0a12a021da6cd0276dc16d1a0519a7905234474b49eabba78469792158c9fc: Status 404 returned error can't find the container with id ad0a12a021da6cd0276dc16d1a0519a7905234474b49eabba78469792158c9fc
Sep 17 17:01:17.458048 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:17.458027    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:01:18.353210 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:18.353173    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t" event={"ID":"6147805b-bc03-4708-8c2e-1325291d065d","Type":"ContainerStarted","Data":"ad0a12a021da6cd0276dc16d1a0519a7905234474b49eabba78469792158c9fc"}
Sep 17 17:01:22.391896 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:22.391860    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 17:01:22.392202 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:22.391934    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 17:01:22.392202 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:22.391965    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 17:01:23.380333 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:23.380292    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t" event={"ID":"6147805b-bc03-4708-8c2e-1325291d065d","Type":"ContainerStarted","Data":"ed83e196934cbecfd7c3d46c5798979220b5a521d807d5e521cba56fe00a38d8"}
Sep 17 17:01:23.406506 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:23.406461    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t" podStartSLOduration=2.472936653 podStartE2EDuration="7.40642266s" totalImagesPullingTime="4.933486007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:01:23.404751278 +0000 UTC m=+1169.299230417" watchObservedRunningTime="2026-09-17 17:01:23.40642266 +0000 UTC m=+1169.300901799"
Sep 17 17:01:24.159012 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:24.158978    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t"
Sep 17 17:01:24.163699 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:24.163674    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t"
Sep 17 17:01:24.385053 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:24.385022    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t"
Sep 17 17:01:24.385884 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:24.385867    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t"
Sep 17 17:01:48.673416 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:48.673384    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k"]
Sep 17 17:01:48.676736 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:48.676717    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k"
Sep 17 17:01:48.680237 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:48.680209    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Sep 17 17:01:48.680361 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:48.680213    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-chd6l\""
Sep 17 17:01:48.691271 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:48.691242    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k"]
Sep 17 17:01:48.783183 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:48.783151    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95e236a8-bb1e-4560-98bb-015f57af75fb-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k\" (UID: \"95e236a8-bb1e-4560-98bb-015f57af75fb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k"
Sep 17 17:01:48.783336 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:48.783198    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95e236a8-bb1e-4560-98bb-015f57af75fb-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k\" (UID: \"95e236a8-bb1e-4560-98bb-015f57af75fb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k"
Sep 17 17:01:48.783336 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:48.783227    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjv2w\" (UniqueName: \"kubernetes.io/projected/95e236a8-bb1e-4560-98bb-015f57af75fb-kube-api-access-cjv2w\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k\" (UID: \"95e236a8-bb1e-4560-98bb-015f57af75fb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k"
Sep 17 17:01:48.783336 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:48.783249    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95e236a8-bb1e-4560-98bb-015f57af75fb-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k\" (UID: \"95e236a8-bb1e-4560-98bb-015f57af75fb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k"
Sep 17 17:01:48.783336 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:48.783272    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95e236a8-bb1e-4560-98bb-015f57af75fb-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k\" (UID: \"95e236a8-bb1e-4560-98bb-015f57af75fb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k"
Sep 17 17:01:48.783336 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:48.783293    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95e236a8-bb1e-4560-98bb-015f57af75fb-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k\" (UID: \"95e236a8-bb1e-4560-98bb-015f57af75fb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k"
Sep 17 17:01:48.883952 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:48.883921    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95e236a8-bb1e-4560-98bb-015f57af75fb-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k\" (UID: \"95e236a8-bb1e-4560-98bb-015f57af75fb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k"
Sep 17 17:01:48.883952 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:48.883959    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95e236a8-bb1e-4560-98bb-015f57af75fb-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k\" (UID: \"95e236a8-bb1e-4560-98bb-015f57af75fb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k"
Sep 17 17:01:48.884160 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:48.883992    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95e236a8-bb1e-4560-98bb-015f57af75fb-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k\" (UID: \"95e236a8-bb1e-4560-98bb-015f57af75fb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k"
Sep 17 17:01:48.884160 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:48.884027    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjv2w\" (UniqueName: \"kubernetes.io/projected/95e236a8-bb1e-4560-98bb-015f57af75fb-kube-api-access-cjv2w\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k\" (UID: \"95e236a8-bb1e-4560-98bb-015f57af75fb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k"
Sep 17 17:01:48.884160 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:48.884053    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95e236a8-bb1e-4560-98bb-015f57af75fb-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k\" (UID: \"95e236a8-bb1e-4560-98bb-015f57af75fb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k"
Sep 17 17:01:48.884160 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:48.884088    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95e236a8-bb1e-4560-98bb-015f57af75fb-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k\" (UID: \"95e236a8-bb1e-4560-98bb-015f57af75fb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k"
Sep 17 17:01:48.884386 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:48.884364    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95e236a8-bb1e-4560-98bb-015f57af75fb-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k\" (UID: \"95e236a8-bb1e-4560-98bb-015f57af75fb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k"
Sep 17 17:01:48.884519 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:48.884489    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95e236a8-bb1e-4560-98bb-015f57af75fb-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k\" (UID: \"95e236a8-bb1e-4560-98bb-015f57af75fb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k"
Sep 17 17:01:48.884604 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:48.884520    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95e236a8-bb1e-4560-98bb-015f57af75fb-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k\" (UID: \"95e236a8-bb1e-4560-98bb-015f57af75fb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k"
Sep 17 17:01:48.886349 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:48.886331    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95e236a8-bb1e-4560-98bb-015f57af75fb-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k\" (UID: \"95e236a8-bb1e-4560-98bb-015f57af75fb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k"
Sep 17 17:01:48.886707 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:48.886689    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95e236a8-bb1e-4560-98bb-015f57af75fb-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k\" (UID: \"95e236a8-bb1e-4560-98bb-015f57af75fb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k"
Sep 17 17:01:48.894332 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:48.894309    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjv2w\" (UniqueName: \"kubernetes.io/projected/95e236a8-bb1e-4560-98bb-015f57af75fb-kube-api-access-cjv2w\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k\" (UID: \"95e236a8-bb1e-4560-98bb-015f57af75fb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k"
Sep 17 17:01:48.986610 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:48.986531    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k"
Sep 17 17:01:49.288191 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:01:49.288158    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95e236a8_bb1e_4560_98bb_015f57af75fb.slice/crio-a86e2319fc83c21c02edb27b0de6a23502c119dac21000edfc81987deef01c3a WatchSource:0}: Error finding container a86e2319fc83c21c02edb27b0de6a23502c119dac21000edfc81987deef01c3a: Status 404 returned error can't find the container with id a86e2319fc83c21c02edb27b0de6a23502c119dac21000edfc81987deef01c3a
Sep 17 17:01:49.288667 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:49.288649    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k"]
Sep 17 17:01:49.522366 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:49.522327    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k" event={"ID":"95e236a8-bb1e-4560-98bb-015f57af75fb","Type":"ContainerStarted","Data":"1fc1462676b38fe56d8cfdb7e5037512c347f2d11c7f2bd40894e221b838b732"}
Sep 17 17:01:49.522574 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:49.522393    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k" event={"ID":"95e236a8-bb1e-4560-98bb-015f57af75fb","Type":"ContainerStarted","Data":"a86e2319fc83c21c02edb27b0de6a23502c119dac21000edfc81987deef01c3a"}
Sep 17 17:01:49.552006 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:49.551962    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k" podStartSLOduration=1.5519469190000001 podStartE2EDuration="1.551946919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:01:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:01:49.550347949 +0000 UTC m=+1195.444827098" watchObservedRunningTime="2026-09-17 17:01:49.551946919 +0000 UTC m=+1195.446426057"
Sep 17 17:01:49.986897 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:49.986811    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k"
Sep 17 17:01:51.239407 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:51.239372    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r"]
Sep 17 17:01:51.243364 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:51.243344    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r"
Sep 17 17:01:51.247235 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:51.247213    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 17 17:01:51.247352 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:51.247217    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Sep 17 17:01:51.248019 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:51.247993    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-7m6hq\""
Sep 17 17:01:51.249728 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:51.249709    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 17 17:01:51.256754 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:51.256734    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r"]
Sep 17 17:01:51.315460 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:51.314566    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k"]
Sep 17 17:01:51.414602 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:51.414560    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cafcd578-1d33-4039-8d3f-703580427a78-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7dcbd489bd-58n9r\" (UID: \"cafcd578-1d33-4039-8d3f-703580427a78\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r"
Sep 17 17:01:51.414788 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:51.414639    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cafcd578-1d33-4039-8d3f-703580427a78-istio-data\") pod \"router-gateway-1-openshift-default-7dcbd489bd-58n9r\" (UID: \"cafcd578-1d33-4039-8d3f-703580427a78\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r"
Sep 17 17:01:51.414788 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:51.414694    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb2kj\" (UniqueName: \"kubernetes.io/projected/cafcd578-1d33-4039-8d3f-703580427a78-kube-api-access-xb2kj\") pod \"router-gateway-1-openshift-default-7dcbd489bd-58n9r\" (UID: \"cafcd578-1d33-4039-8d3f-703580427a78\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r"
Sep 17 17:01:51.414788 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:51.414769    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cafcd578-1d33-4039-8d3f-703580427a78-istio-token\") pod \"router-gateway-1-openshift-default-7dcbd489bd-58n9r\" (UID: \"cafcd578-1d33-4039-8d3f-703580427a78\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r"
Sep 17 17:01:51.414979 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:51.414798    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cafcd578-1d33-4039-8d3f-703580427a78-istio-podinfo\") pod \"router-gateway-1-openshift-default-7dcbd489bd-58n9r\" (UID: \"cafcd578-1d33-4039-8d3f-703580427a78\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r"
Sep 17 17:01:51.414979 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:51.414840    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cafcd578-1d33-4039-8d3f-703580427a78-istio-envoy\") pod \"router-gateway-1-openshift-default-7dcbd489bd-58n9r\" (UID: \"cafcd578-1d33-4039-8d3f-703580427a78\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r"
Sep 17 17:01:51.414979 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:51.414880    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cafcd578-1d33-4039-8d3f-703580427a78-workload-certs\") pod \"router-gateway-1-openshift-default-7dcbd489bd-58n9r\" (UID: \"cafcd578-1d33-4039-8d3f-703580427a78\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r"
Sep 17 17:01:51.414979 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:51.414926    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cafcd578-1d33-4039-8d3f-703580427a78-credential-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-58n9r\" (UID: \"cafcd578-1d33-4039-8d3f-703580427a78\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r"
Sep 17 17:01:51.414979 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:51.414963    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cafcd578-1d33-4039-8d3f-703580427a78-workload-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-58n9r\" (UID: \"cafcd578-1d33-4039-8d3f-703580427a78\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r"
Sep 17 17:01:51.515929 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:51.515825    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cafcd578-1d33-4039-8d3f-703580427a78-istio-token\") pod \"router-gateway-1-openshift-default-7dcbd489bd-58n9r\" (UID: \"cafcd578-1d33-4039-8d3f-703580427a78\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r"
Sep 17 17:01:51.515929 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:51.515884    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cafcd578-1d33-4039-8d3f-703580427a78-istio-podinfo\") pod \"router-gateway-1-openshift-default-7dcbd489bd-58n9r\" (UID: \"cafcd578-1d33-4039-8d3f-703580427a78\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r"
Sep 17 17:01:51.515929 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:51.515917    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cafcd578-1d33-4039-8d3f-703580427a78-istio-envoy\") pod \"router-gateway-1-openshift-default-7dcbd489bd-58n9r\" (UID: \"cafcd578-1d33-4039-8d3f-703580427a78\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r"
Sep 17 17:01:51.516234 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:51.515954    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cafcd578-1d33-4039-8d3f-703580427a78-workload-certs\") pod \"router-gateway-1-openshift-default-7dcbd489bd-58n9r\" (UID: \"cafcd578-1d33-4039-8d3f-703580427a78\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r"
Sep 17 17:01:51.516234 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:51.515981    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cafcd578-1d33-4039-8d3f-703580427a78-credential-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-58n9r\" (UID: \"cafcd578-1d33-4039-8d3f-703580427a78\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r"
Sep 17 17:01:51.516234 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:51.516016    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cafcd578-1d33-4039-8d3f-703580427a78-workload-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-58n9r\" (UID: \"cafcd578-1d33-4039-8d3f-703580427a78\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r"
Sep 17 17:01:51.516234 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:51.516063    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cafcd578-1d33-4039-8d3f-703580427a78-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7dcbd489bd-58n9r\" (UID: \"cafcd578-1d33-4039-8d3f-703580427a78\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r"
Sep 17 17:01:51.516234 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:51.516106    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cafcd578-1d33-4039-8d3f-703580427a78-istio-data\") pod \"router-gateway-1-openshift-default-7dcbd489bd-58n9r\" (UID: \"cafcd578-1d33-4039-8d3f-703580427a78\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r"
Sep 17 17:01:51.516234 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:51.516143    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb2kj\" (UniqueName: \"kubernetes.io/projected/cafcd578-1d33-4039-8d3f-703580427a78-kube-api-access-xb2kj\") pod \"router-gateway-1-openshift-default-7dcbd489bd-58n9r\" (UID: \"cafcd578-1d33-4039-8d3f-703580427a78\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r"
Sep 17 17:01:51.517632 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:51.517602    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cafcd578-1d33-4039-8d3f-703580427a78-credential-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-58n9r\" (UID: \"cafcd578-1d33-4039-8d3f-703580427a78\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r"
Sep 17 17:01:51.518747 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:51.518165    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cafcd578-1d33-4039-8d3f-703580427a78-workload-certs\") pod \"router-gateway-1-openshift-default-7dcbd489bd-58n9r\" (UID: \"cafcd578-1d33-4039-8d3f-703580427a78\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r"
Sep 17 17:01:51.518747 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:51.518261    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cafcd578-1d33-4039-8d3f-703580427a78-istio-data\") pod \"router-gateway-1-openshift-default-7dcbd489bd-58n9r\" (UID: \"cafcd578-1d33-4039-8d3f-703580427a78\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r"
Sep 17 17:01:51.518747 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:51.518601    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cafcd578-1d33-4039-8d3f-703580427a78-workload-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-58n9r\" (UID: \"cafcd578-1d33-4039-8d3f-703580427a78\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r"
Sep 17 17:01:51.518747 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:51.518710    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cafcd578-1d33-4039-8d3f-703580427a78-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7dcbd489bd-58n9r\" (UID: \"cafcd578-1d33-4039-8d3f-703580427a78\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r"
Sep 17 17:01:51.522679 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:51.522653    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cafcd578-1d33-4039-8d3f-703580427a78-istio-envoy\") pod \"router-gateway-1-openshift-default-7dcbd489bd-58n9r\" (UID: \"cafcd578-1d33-4039-8d3f-703580427a78\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r"
Sep 17 17:01:51.525086 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:51.525058    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cafcd578-1d33-4039-8d3f-703580427a78-istio-podinfo\") pod \"router-gateway-1-openshift-default-7dcbd489bd-58n9r\" (UID: \"cafcd578-1d33-4039-8d3f-703580427a78\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r"
Sep 17 17:01:51.526413 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:51.526269    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb2kj\" (UniqueName: \"kubernetes.io/projected/cafcd578-1d33-4039-8d3f-703580427a78-kube-api-access-xb2kj\") pod \"router-gateway-1-openshift-default-7dcbd489bd-58n9r\" (UID: \"cafcd578-1d33-4039-8d3f-703580427a78\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r"
Sep 17 17:01:51.527367 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:51.527340    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cafcd578-1d33-4039-8d3f-703580427a78-istio-token\") pod \"router-gateway-1-openshift-default-7dcbd489bd-58n9r\" (UID: \"cafcd578-1d33-4039-8d3f-703580427a78\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r"
Sep 17 17:01:51.556023 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:51.555997    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r"
Sep 17 17:01:51.620461 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:01:51.617048    2566 secret.go:189] Couldn't get secret e2e-test-gateway-section-name-propagation-49b5edd9/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs: secret "gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs" not found
Sep 17 17:01:51.620461 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:01:51.617128    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95e236a8-bb1e-4560-98bb-015f57af75fb-tls-certs podName:95e236a8-bb1e-4560-98bb-015f57af75fb nodeName:}" failed. No retries permitted until 2026-09-17 17:01:52.117106974 +0000 UTC m=+1198.011586095 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/95e236a8-bb1e-4560-98bb-015f57af75fb-tls-certs") pod "gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k" (UID: "95e236a8-bb1e-4560-98bb-015f57af75fb") : secret "gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs" not found
Sep 17 17:01:51.863663 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:51.863624    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r"]
Sep 17 17:01:51.864943 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:01:51.864917    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcafcd578_1d33_4039_8d3f_703580427a78.slice/crio-09a9a1752c80b9d1b5b6bc74fb0a7577b9783b5d7f3e350581ce0f05666e1a7a WatchSource:0}: Error finding container 09a9a1752c80b9d1b5b6bc74fb0a7577b9783b5d7f3e350581ce0f05666e1a7a: Status 404 returned error can't find the container with id 09a9a1752c80b9d1b5b6bc74fb0a7577b9783b5d7f3e350581ce0f05666e1a7a
Sep 17 17:01:51.867134 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:51.867101    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 17:01:51.867230 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:51.867166    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 17:01:51.867230 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:51.867201    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 17:01:52.123673 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:01:52.123575    2566 secret.go:189] Couldn't get secret e2e-test-gateway-section-name-propagation-49b5edd9/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs: secret "gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs" not found
Sep 17 17:01:52.123673 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:01:52.123662    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95e236a8-bb1e-4560-98bb-015f57af75fb-tls-certs podName:95e236a8-bb1e-4560-98bb-015f57af75fb nodeName:}" failed. No retries permitted until 2026-09-17 17:01:53.123641332 +0000 UTC m=+1199.018120454 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/95e236a8-bb1e-4560-98bb-015f57af75fb-tls-certs") pod "gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k" (UID: "95e236a8-bb1e-4560-98bb-015f57af75fb") : secret "gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs" not found
Sep 17 17:01:52.542022 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:52.541995    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r" event={"ID":"cafcd578-1d33-4039-8d3f-703580427a78","Type":"ContainerStarted","Data":"ecec9ec8be8d580ad43d4653729f466e988ba4886f5b0c4f5a8b94de8582b930"}
Sep 17 17:01:52.542505 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:52.542048    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r" event={"ID":"cafcd578-1d33-4039-8d3f-703580427a78","Type":"ContainerStarted","Data":"09a9a1752c80b9d1b5b6bc74fb0a7577b9783b5d7f3e350581ce0f05666e1a7a"}
Sep 17 17:01:52.551086 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:01:52.551058    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:01:52.551086 ip-10-0-141-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:01:52.551086 ip-10-0-141-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:01:52.551086 ip-10-0-141-66 kubenswrapper[2566]:  > containerID="1fc1462676b38fe56d8cfdb7e5037512c347f2d11c7f2bd40894e221b838b732" cmd=["/bin/sleep","15"]
Sep 17 17:01:52.551287 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:52.551099    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k" message="" err=<
Sep 17 17:01:52.551287 ip-10-0-141-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:01:52.551287 ip-10-0-141-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:01:52.551287 ip-10-0-141-66 kubenswrapper[2566]:  >
Sep 17 17:01:52.551287 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:01:52.551111    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:01:52.551287 ip-10-0-141-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:01:52.551287 ip-10-0-141-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:01:52.551287 ip-10-0-141-66 kubenswrapper[2566]:  > pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k" podUID="95e236a8-bb1e-4560-98bb-015f57af75fb" containerName="main" containerID="cri-o://1fc1462676b38fe56d8cfdb7e5037512c347f2d11c7f2bd40894e221b838b732"
Sep 17 17:01:52.551287 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:52.551145    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k" podUID="95e236a8-bb1e-4560-98bb-015f57af75fb" containerName="main" containerID="cri-o://1fc1462676b38fe56d8cfdb7e5037512c347f2d11c7f2bd40894e221b838b732" gracePeriod=60
Sep 17 17:01:52.556875 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:52.556851    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r"
Sep 17 17:01:52.558529 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:52.558504    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r" podUID="cafcd578-1d33-4039-8d3f-703580427a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 17 17:01:52.594826 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:52.594777    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r" podStartSLOduration=1.594760778 podStartE2EDuration="1.594760778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:01:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:01:52.592605901 +0000 UTC m=+1198.487085042" watchObservedRunningTime="2026-09-17 17:01:52.594760778 +0000 UTC m=+1198.489239916"
Sep 17 17:01:52.836179 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:52.836143    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k"
Sep 17 17:01:52.931317 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:52.931284    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95e236a8-bb1e-4560-98bb-015f57af75fb-model-cache\") pod \"95e236a8-bb1e-4560-98bb-015f57af75fb\" (UID: \"95e236a8-bb1e-4560-98bb-015f57af75fb\") "
Sep 17 17:01:52.931499 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:52.931328    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95e236a8-bb1e-4560-98bb-015f57af75fb-home\") pod \"95e236a8-bb1e-4560-98bb-015f57af75fb\" (UID: \"95e236a8-bb1e-4560-98bb-015f57af75fb\") "
Sep 17 17:01:52.931499 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:52.931389    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjv2w\" (UniqueName: \"kubernetes.io/projected/95e236a8-bb1e-4560-98bb-015f57af75fb-kube-api-access-cjv2w\") pod \"95e236a8-bb1e-4560-98bb-015f57af75fb\" (UID: \"95e236a8-bb1e-4560-98bb-015f57af75fb\") "
Sep 17 17:01:52.931499 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:52.931427    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95e236a8-bb1e-4560-98bb-015f57af75fb-tmp-dir\") pod \"95e236a8-bb1e-4560-98bb-015f57af75fb\" (UID: \"95e236a8-bb1e-4560-98bb-015f57af75fb\") "
Sep 17 17:01:52.931499 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:52.931487    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95e236a8-bb1e-4560-98bb-015f57af75fb-tls-certs\") pod \"95e236a8-bb1e-4560-98bb-015f57af75fb\" (UID: \"95e236a8-bb1e-4560-98bb-015f57af75fb\") "
Sep 17 17:01:52.931726 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:52.931541    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95e236a8-bb1e-4560-98bb-015f57af75fb-dshm\") pod \"95e236a8-bb1e-4560-98bb-015f57af75fb\" (UID: \"95e236a8-bb1e-4560-98bb-015f57af75fb\") "
Sep 17 17:01:52.931726 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:52.931600    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95e236a8-bb1e-4560-98bb-015f57af75fb-model-cache" (OuterVolumeSpecName: "model-cache") pod "95e236a8-bb1e-4560-98bb-015f57af75fb" (UID: "95e236a8-bb1e-4560-98bb-015f57af75fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:01:52.931726 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:52.931640    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95e236a8-bb1e-4560-98bb-015f57af75fb-home" (OuterVolumeSpecName: "home") pod "95e236a8-bb1e-4560-98bb-015f57af75fb" (UID: "95e236a8-bb1e-4560-98bb-015f57af75fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:01:52.931726 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:52.931665    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95e236a8-bb1e-4560-98bb-015f57af75fb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "95e236a8-bb1e-4560-98bb-015f57af75fb" (UID: "95e236a8-bb1e-4560-98bb-015f57af75fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:01:52.931916 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:52.931829    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95e236a8-bb1e-4560-98bb-015f57af75fb-tmp-dir\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:01:52.931916 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:52.931846    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95e236a8-bb1e-4560-98bb-015f57af75fb-model-cache\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:01:52.931916 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:52.931861    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95e236a8-bb1e-4560-98bb-015f57af75fb-home\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:01:52.933869 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:52.933845    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95e236a8-bb1e-4560-98bb-015f57af75fb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "95e236a8-bb1e-4560-98bb-015f57af75fb" (UID: "95e236a8-bb1e-4560-98bb-015f57af75fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:01:52.934138 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:52.934120    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95e236a8-bb1e-4560-98bb-015f57af75fb-dshm" (OuterVolumeSpecName: "dshm") pod "95e236a8-bb1e-4560-98bb-015f57af75fb" (UID: "95e236a8-bb1e-4560-98bb-015f57af75fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:01:52.934187 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:52.934144    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95e236a8-bb1e-4560-98bb-015f57af75fb-kube-api-access-cjv2w" (OuterVolumeSpecName: "kube-api-access-cjv2w") pod "95e236a8-bb1e-4560-98bb-015f57af75fb" (UID: "95e236a8-bb1e-4560-98bb-015f57af75fb"). InnerVolumeSpecName "kube-api-access-cjv2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:01:53.032626 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:53.032590    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjv2w\" (UniqueName: \"kubernetes.io/projected/95e236a8-bb1e-4560-98bb-015f57af75fb-kube-api-access-cjv2w\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:01:53.032626 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:53.032619    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95e236a8-bb1e-4560-98bb-015f57af75fb-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:01:53.032626 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:53.032630    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95e236a8-bb1e-4560-98bb-015f57af75fb-dshm\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:01:53.549541 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:53.549507    2566 generic.go:356] "Generic (PLEG): container finished" podID="95e236a8-bb1e-4560-98bb-015f57af75fb" containerID="1fc1462676b38fe56d8cfdb7e5037512c347f2d11c7f2bd40894e221b838b732" exitCode=0
Sep 17 17:01:53.549927 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:53.549556    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k"
Sep 17 17:01:53.549927 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:53.549571    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k" event={"ID":"95e236a8-bb1e-4560-98bb-015f57af75fb","Type":"ContainerDied","Data":"1fc1462676b38fe56d8cfdb7e5037512c347f2d11c7f2bd40894e221b838b732"}
Sep 17 17:01:53.549927 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:53.549642    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k" event={"ID":"95e236a8-bb1e-4560-98bb-015f57af75fb","Type":"ContainerDied","Data":"a86e2319fc83c21c02edb27b0de6a23502c119dac21000edfc81987deef01c3a"}
Sep 17 17:01:53.549927 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:53.549682    2566 scope.go:117] "RemoveContainer" containerID="1fc1462676b38fe56d8cfdb7e5037512c347f2d11c7f2bd40894e221b838b732"
Sep 17 17:01:53.559960 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:53.559941    2566 scope.go:117] "RemoveContainer" containerID="1fc1462676b38fe56d8cfdb7e5037512c347f2d11c7f2bd40894e221b838b732"
Sep 17 17:01:53.560243 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:01:53.560224    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fc1462676b38fe56d8cfdb7e5037512c347f2d11c7f2bd40894e221b838b732\": container with ID starting with 1fc1462676b38fe56d8cfdb7e5037512c347f2d11c7f2bd40894e221b838b732 not found: ID does not exist" containerID="1fc1462676b38fe56d8cfdb7e5037512c347f2d11c7f2bd40894e221b838b732"
Sep 17 17:01:53.560288 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:53.560251    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fc1462676b38fe56d8cfdb7e5037512c347f2d11c7f2bd40894e221b838b732"} err="failed to get container status \"1fc1462676b38fe56d8cfdb7e5037512c347f2d11c7f2bd40894e221b838b732\": rpc error: code = NotFound desc = could not find container \"1fc1462676b38fe56d8cfdb7e5037512c347f2d11c7f2bd40894e221b838b732\": container with ID starting with 1fc1462676b38fe56d8cfdb7e5037512c347f2d11c7f2bd40894e221b838b732 not found: ID does not exist"
Sep 17 17:01:53.568306 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:53.568283    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r"
Sep 17 17:01:53.574599 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:53.574579    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k"]
Sep 17 17:01:53.580164 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:53.580132    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-c7v5k"]
Sep 17 17:01:54.348189 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:54.348144    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95e236a8-bb1e-4560-98bb-015f57af75fb" path="/var/lib/kubelet/pods/95e236a8-bb1e-4560-98bb-015f57af75fb/volumes"
Sep 17 17:01:54.557698 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:54.557282    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r"
Sep 17 17:01:54.558480 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:54.558461    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r"
Sep 17 17:01:55.885220 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:55.885181    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t"]
Sep 17 17:01:55.885735 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:55.885429    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t" podUID="6147805b-bc03-4708-8c2e-1325291d065d" containerName="istio-proxy" containerID="cri-o://ed83e196934cbecfd7c3d46c5798979220b5a521d807d5e521cba56fe00a38d8" gracePeriod=30
Sep 17 17:01:56.342229 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:56.342200    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gblhr_91477834-6648-4e75-8530-e01bdee5e57e/ovn-acl-logging/0.log"
Sep 17 17:01:56.344571 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:56.344540    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gblhr_91477834-6648-4e75-8530-e01bdee5e57e/ovn-acl-logging/0.log"
Sep 17 17:01:58.824751 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:58.824722    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t"
Sep 17 17:01:58.987007 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:58.986929    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6147805b-bc03-4708-8c2e-1325291d065d-credential-socket\") pod \"6147805b-bc03-4708-8c2e-1325291d065d\" (UID: \"6147805b-bc03-4708-8c2e-1325291d065d\") "
Sep 17 17:01:58.987007 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:58.986998    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6147805b-bc03-4708-8c2e-1325291d065d-istiod-ca-cert\") pod \"6147805b-bc03-4708-8c2e-1325291d065d\" (UID: \"6147805b-bc03-4708-8c2e-1325291d065d\") "
Sep 17 17:01:58.987190 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:58.987031    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6147805b-bc03-4708-8c2e-1325291d065d-istio-data\") pod \"6147805b-bc03-4708-8c2e-1325291d065d\" (UID: \"6147805b-bc03-4708-8c2e-1325291d065d\") "
Sep 17 17:01:58.987190 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:58.987101    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6147805b-bc03-4708-8c2e-1325291d065d-workload-socket\") pod \"6147805b-bc03-4708-8c2e-1325291d065d\" (UID: \"6147805b-bc03-4708-8c2e-1325291d065d\") "
Sep 17 17:01:58.987190 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:58.987137    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xcmv2\" (UniqueName: \"kubernetes.io/projected/6147805b-bc03-4708-8c2e-1325291d065d-kube-api-access-xcmv2\") pod \"6147805b-bc03-4708-8c2e-1325291d065d\" (UID: \"6147805b-bc03-4708-8c2e-1325291d065d\") "
Sep 17 17:01:58.987190 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:58.987163    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6147805b-bc03-4708-8c2e-1325291d065d-istio-token\") pod \"6147805b-bc03-4708-8c2e-1325291d065d\" (UID: \"6147805b-bc03-4708-8c2e-1325291d065d\") "
Sep 17 17:01:58.987190 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:58.987186    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6147805b-bc03-4708-8c2e-1325291d065d-workload-certs\") pod \"6147805b-bc03-4708-8c2e-1325291d065d\" (UID: \"6147805b-bc03-4708-8c2e-1325291d065d\") "
Sep 17 17:01:58.987422 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:58.987206    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6147805b-bc03-4708-8c2e-1325291d065d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6147805b-bc03-4708-8c2e-1325291d065d" (UID: "6147805b-bc03-4708-8c2e-1325291d065d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:01:58.987422 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:58.987214    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6147805b-bc03-4708-8c2e-1325291d065d-istio-envoy\") pod \"6147805b-bc03-4708-8c2e-1325291d065d\" (UID: \"6147805b-bc03-4708-8c2e-1325291d065d\") "
Sep 17 17:01:58.987422 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:58.987263    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6147805b-bc03-4708-8c2e-1325291d065d-istio-podinfo\") pod \"6147805b-bc03-4708-8c2e-1325291d065d\" (UID: \"6147805b-bc03-4708-8c2e-1325291d065d\") "
Sep 17 17:01:58.987607 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:58.987511    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6147805b-bc03-4708-8c2e-1325291d065d-istio-data" (OuterVolumeSpecName: "istio-data") pod "6147805b-bc03-4708-8c2e-1325291d065d" (UID: "6147805b-bc03-4708-8c2e-1325291d065d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:01:58.987607 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:58.987542    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6147805b-bc03-4708-8c2e-1325291d065d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6147805b-bc03-4708-8c2e-1325291d065d" (UID: "6147805b-bc03-4708-8c2e-1325291d065d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:01:58.987607 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:58.987552    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6147805b-bc03-4708-8c2e-1325291d065d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6147805b-bc03-4708-8c2e-1325291d065d" (UID: "6147805b-bc03-4708-8c2e-1325291d065d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:01:58.987607 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:58.987562    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6147805b-bc03-4708-8c2e-1325291d065d-credential-socket\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:01:58.987748 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:58.987625    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6147805b-bc03-4708-8c2e-1325291d065d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6147805b-bc03-4708-8c2e-1325291d065d" (UID: "6147805b-bc03-4708-8c2e-1325291d065d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:01:58.989515 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:58.989489    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6147805b-bc03-4708-8c2e-1325291d065d-istio-token" (OuterVolumeSpecName: "istio-token") pod "6147805b-bc03-4708-8c2e-1325291d065d" (UID: "6147805b-bc03-4708-8c2e-1325291d065d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:01:58.989624 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:58.989542    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6147805b-bc03-4708-8c2e-1325291d065d-kube-api-access-xcmv2" (OuterVolumeSpecName: "kube-api-access-xcmv2") pod "6147805b-bc03-4708-8c2e-1325291d065d" (UID: "6147805b-bc03-4708-8c2e-1325291d065d"). InnerVolumeSpecName "kube-api-access-xcmv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:01:58.989624 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:58.989578    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6147805b-bc03-4708-8c2e-1325291d065d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6147805b-bc03-4708-8c2e-1325291d065d" (UID: "6147805b-bc03-4708-8c2e-1325291d065d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:01:58.989699 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:58.989621    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6147805b-bc03-4708-8c2e-1325291d065d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6147805b-bc03-4708-8c2e-1325291d065d" (UID: "6147805b-bc03-4708-8c2e-1325291d065d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 17:01:59.088071 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:59.088045    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6147805b-bc03-4708-8c2e-1325291d065d-workload-socket\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:01:59.088071 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:59.088067    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcmv2\" (UniqueName: \"kubernetes.io/projected/6147805b-bc03-4708-8c2e-1325291d065d-kube-api-access-xcmv2\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:01:59.088232 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:59.088078    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6147805b-bc03-4708-8c2e-1325291d065d-istio-token\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:01:59.088232 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:59.088088    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6147805b-bc03-4708-8c2e-1325291d065d-workload-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:01:59.088232 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:59.088096    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6147805b-bc03-4708-8c2e-1325291d065d-istio-envoy\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:01:59.088232 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:59.088104    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6147805b-bc03-4708-8c2e-1325291d065d-istio-podinfo\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:01:59.088232 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:59.088111    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6147805b-bc03-4708-8c2e-1325291d065d-istiod-ca-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:01:59.088232 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:59.088119    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6147805b-bc03-4708-8c2e-1325291d065d-istio-data\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:01:59.583490 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:59.583419    2566 generic.go:356] "Generic (PLEG): container finished" podID="6147805b-bc03-4708-8c2e-1325291d065d" containerID="ed83e196934cbecfd7c3d46c5798979220b5a521d807d5e521cba56fe00a38d8" exitCode=0
Sep 17 17:01:59.583674 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:59.583502    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t"
Sep 17 17:01:59.583674 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:59.583514    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t" event={"ID":"6147805b-bc03-4708-8c2e-1325291d065d","Type":"ContainerDied","Data":"ed83e196934cbecfd7c3d46c5798979220b5a521d807d5e521cba56fe00a38d8"}
Sep 17 17:01:59.583674 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:59.583569    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t" event={"ID":"6147805b-bc03-4708-8c2e-1325291d065d","Type":"ContainerDied","Data":"ad0a12a021da6cd0276dc16d1a0519a7905234474b49eabba78469792158c9fc"}
Sep 17 17:01:59.583674 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:59.583592    2566 scope.go:117] "RemoveContainer" containerID="ed83e196934cbecfd7c3d46c5798979220b5a521d807d5e521cba56fe00a38d8"
Sep 17 17:01:59.593590 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:59.593566    2566 scope.go:117] "RemoveContainer" containerID="ed83e196934cbecfd7c3d46c5798979220b5a521d807d5e521cba56fe00a38d8"
Sep 17 17:01:59.593865 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:01:59.593844    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed83e196934cbecfd7c3d46c5798979220b5a521d807d5e521cba56fe00a38d8\": container with ID starting with ed83e196934cbecfd7c3d46c5798979220b5a521d807d5e521cba56fe00a38d8 not found: ID does not exist" containerID="ed83e196934cbecfd7c3d46c5798979220b5a521d807d5e521cba56fe00a38d8"
Sep 17 17:01:59.593935 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:59.593873    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed83e196934cbecfd7c3d46c5798979220b5a521d807d5e521cba56fe00a38d8"} err="failed to get container status \"ed83e196934cbecfd7c3d46c5798979220b5a521d807d5e521cba56fe00a38d8\": rpc error: code = NotFound desc = could not find container \"ed83e196934cbecfd7c3d46c5798979220b5a521d807d5e521cba56fe00a38d8\": container with ID starting with ed83e196934cbecfd7c3d46c5798979220b5a521d807d5e521cba56fe00a38d8 not found: ID does not exist"
Sep 17 17:01:59.607861 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:59.607839    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t"]
Sep 17 17:01:59.612730 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:01:59.612709    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-l6n8t"]
Sep 17 17:02:00.352204 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:00.352172    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6147805b-bc03-4708-8c2e-1325291d065d" path="/var/lib/kubelet/pods/6147805b-bc03-4708-8c2e-1325291d065d/volumes"
Sep 17 17:02:05.476018 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:05.475977    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828"]
Sep 17 17:02:05.476507 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:05.476377    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6147805b-bc03-4708-8c2e-1325291d065d" containerName="istio-proxy"
Sep 17 17:02:05.476507 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:05.476393    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6147805b-bc03-4708-8c2e-1325291d065d" containerName="istio-proxy"
Sep 17 17:02:05.476507 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:05.476404    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95e236a8-bb1e-4560-98bb-015f57af75fb" containerName="main"
Sep 17 17:02:05.476507 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:05.476409    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="95e236a8-bb1e-4560-98bb-015f57af75fb" containerName="main"
Sep 17 17:02:05.476507 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:05.476504    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="95e236a8-bb1e-4560-98bb-015f57af75fb" containerName="main"
Sep 17 17:02:05.476507 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:05.476512    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6147805b-bc03-4708-8c2e-1325291d065d" containerName="istio-proxy"
Sep 17 17:02:05.479376 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:05.479352    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828"
Sep 17 17:02:05.482301 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:05.482283    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Sep 17 17:02:05.484300 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:05.484281    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-wfjcm\""
Sep 17 17:02:05.500803 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:05.498425    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828"]
Sep 17 17:02:05.539748 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:05.539718    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce71b32a-bf09-44eb-94be-da08acf50fa6-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828\" (UID: \"ce71b32a-bf09-44eb-94be-da08acf50fa6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828"
Sep 17 17:02:05.539916 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:05.539764    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce71b32a-bf09-44eb-94be-da08acf50fa6-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828\" (UID: \"ce71b32a-bf09-44eb-94be-da08acf50fa6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828"
Sep 17 17:02:05.539916 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:05.539796    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce71b32a-bf09-44eb-94be-da08acf50fa6-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828\" (UID: \"ce71b32a-bf09-44eb-94be-da08acf50fa6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828"
Sep 17 17:02:05.539916 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:05.539818    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce71b32a-bf09-44eb-94be-da08acf50fa6-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828\" (UID: \"ce71b32a-bf09-44eb-94be-da08acf50fa6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828"
Sep 17 17:02:05.540077 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:05.539943    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce71b32a-bf09-44eb-94be-da08acf50fa6-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828\" (UID: \"ce71b32a-bf09-44eb-94be-da08acf50fa6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828"
Sep 17 17:02:05.540077 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:05.539979    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqc7r\" (UniqueName: \"kubernetes.io/projected/ce71b32a-bf09-44eb-94be-da08acf50fa6-kube-api-access-zqc7r\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828\" (UID: \"ce71b32a-bf09-44eb-94be-da08acf50fa6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828"
Sep 17 17:02:05.641031 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:05.640997    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce71b32a-bf09-44eb-94be-da08acf50fa6-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828\" (UID: \"ce71b32a-bf09-44eb-94be-da08acf50fa6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828"
Sep 17 17:02:05.641237 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:05.641053    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce71b32a-bf09-44eb-94be-da08acf50fa6-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828\" (UID: \"ce71b32a-bf09-44eb-94be-da08acf50fa6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828"
Sep 17 17:02:05.641237 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:05.641079    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce71b32a-bf09-44eb-94be-da08acf50fa6-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828\" (UID: \"ce71b32a-bf09-44eb-94be-da08acf50fa6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828"
Sep 17 17:02:05.641237 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:05.641094    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce71b32a-bf09-44eb-94be-da08acf50fa6-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828\" (UID: \"ce71b32a-bf09-44eb-94be-da08acf50fa6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828"
Sep 17 17:02:05.641237 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:05.641185    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce71b32a-bf09-44eb-94be-da08acf50fa6-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828\" (UID: \"ce71b32a-bf09-44eb-94be-da08acf50fa6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828"
Sep 17 17:02:05.641237 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:05.641215    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqc7r\" (UniqueName: \"kubernetes.io/projected/ce71b32a-bf09-44eb-94be-da08acf50fa6-kube-api-access-zqc7r\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828\" (UID: \"ce71b32a-bf09-44eb-94be-da08acf50fa6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828"
Sep 17 17:02:05.641562 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:05.641449    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce71b32a-bf09-44eb-94be-da08acf50fa6-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828\" (UID: \"ce71b32a-bf09-44eb-94be-da08acf50fa6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828"
Sep 17 17:02:05.641562 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:05.641496    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce71b32a-bf09-44eb-94be-da08acf50fa6-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828\" (UID: \"ce71b32a-bf09-44eb-94be-da08acf50fa6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828"
Sep 17 17:02:05.641709 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:05.641689    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce71b32a-bf09-44eb-94be-da08acf50fa6-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828\" (UID: \"ce71b32a-bf09-44eb-94be-da08acf50fa6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828"
Sep 17 17:02:05.643594 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:05.643566    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce71b32a-bf09-44eb-94be-da08acf50fa6-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828\" (UID: \"ce71b32a-bf09-44eb-94be-da08acf50fa6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828"
Sep 17 17:02:05.643917 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:05.643900    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce71b32a-bf09-44eb-94be-da08acf50fa6-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828\" (UID: \"ce71b32a-bf09-44eb-94be-da08acf50fa6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828"
Sep 17 17:02:05.652526 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:05.652504    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqc7r\" (UniqueName: \"kubernetes.io/projected/ce71b32a-bf09-44eb-94be-da08acf50fa6-kube-api-access-zqc7r\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828\" (UID: \"ce71b32a-bf09-44eb-94be-da08acf50fa6\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828"
Sep 17 17:02:05.789776 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:05.789744    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828"
Sep 17 17:02:06.116394 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:06.116367    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828"]
Sep 17 17:02:06.119102 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:02:06.119073    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce71b32a_bf09_44eb_94be_da08acf50fa6.slice/crio-cb863dd0f76b61cbcbdf851c1fb4edce8fcd6662135e45c0b2b7a17820cc914b WatchSource:0}: Error finding container cb863dd0f76b61cbcbdf851c1fb4edce8fcd6662135e45c0b2b7a17820cc914b: Status 404 returned error can't find the container with id cb863dd0f76b61cbcbdf851c1fb4edce8fcd6662135e45c0b2b7a17820cc914b
Sep 17 17:02:06.625644 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:06.625604    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828" event={"ID":"ce71b32a-bf09-44eb-94be-da08acf50fa6","Type":"ContainerStarted","Data":"be4de00eb02136b39778104f72e97e4de7d9d74e57965c21b17fc02cb3b97b96"}
Sep 17 17:02:06.626185 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:06.625671    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828" event={"ID":"ce71b32a-bf09-44eb-94be-da08acf50fa6","Type":"ContainerStarted","Data":"cb863dd0f76b61cbcbdf851c1fb4edce8fcd6662135e45c0b2b7a17820cc914b"}
Sep 17 17:02:06.658474 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:06.658400    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828" podStartSLOduration=1.658384576 podStartE2EDuration="1.658384576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:02:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:02:06.65674376 +0000 UTC m=+1212.551222898" watchObservedRunningTime="2026-09-17 17:02:06.658384576 +0000 UTC m=+1212.552863715"
Sep 17 17:02:06.790148 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:06.790118    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828"
Sep 17 17:02:08.376779 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:08.376740    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs"]
Sep 17 17:02:08.386270 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:02:08.386238    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:02:08.386270 ip-10-0-141-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:02:08.386270 ip-10-0-141-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:02:08.386270 ip-10-0-141-66 kubenswrapper[2566]:  > containerID="0d78c089b8beac701f9054b9c587a4b252501b03be7a6e262a3a763ae903c699" cmd=["/bin/sleep","15"]
Sep 17 17:02:08.386598 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:08.386292    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs" message="" err=<
Sep 17 17:02:08.386598 ip-10-0-141-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:02:08.386598 ip-10-0-141-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:02:08.386598 ip-10-0-141-66 kubenswrapper[2566]:  >
Sep 17 17:02:08.386598 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:02:08.386310    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:02:08.386598 ip-10-0-141-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:02:08.386598 ip-10-0-141-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:02:08.386598 ip-10-0-141-66 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs" podUID="8860e9bc-46c7-47fd-b243-04efc6d8fc4c" containerName="main" containerID="cri-o://0d78c089b8beac701f9054b9c587a4b252501b03be7a6e262a3a763ae903c699"
Sep 17 17:02:08.386598 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:08.386351    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs" podUID="8860e9bc-46c7-47fd-b243-04efc6d8fc4c" containerName="main" containerID="cri-o://0d78c089b8beac701f9054b9c587a4b252501b03be7a6e262a3a763ae903c699" gracePeriod=60
Sep 17 17:02:08.637601 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:08.637520    2566 generic.go:356] "Generic (PLEG): container finished" podID="8860e9bc-46c7-47fd-b243-04efc6d8fc4c" containerID="0d78c089b8beac701f9054b9c587a4b252501b03be7a6e262a3a763ae903c699" exitCode=0
Sep 17 17:02:08.637813 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:08.637754    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs" event={"ID":"8860e9bc-46c7-47fd-b243-04efc6d8fc4c","Type":"ContainerDied","Data":"0d78c089b8beac701f9054b9c587a4b252501b03be7a6e262a3a763ae903c699"}
Sep 17 17:02:08.773423 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:08.773400    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs"
Sep 17 17:02:08.867746 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:08.867716    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8860e9bc-46c7-47fd-b243-04efc6d8fc4c-tmp-dir\") pod \"8860e9bc-46c7-47fd-b243-04efc6d8fc4c\" (UID: \"8860e9bc-46c7-47fd-b243-04efc6d8fc4c\") "
Sep 17 17:02:08.867988 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:08.867753    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8860e9bc-46c7-47fd-b243-04efc6d8fc4c-model-cache\") pod \"8860e9bc-46c7-47fd-b243-04efc6d8fc4c\" (UID: \"8860e9bc-46c7-47fd-b243-04efc6d8fc4c\") "
Sep 17 17:02:08.867988 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:08.867779    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zn5q5\" (UniqueName: \"kubernetes.io/projected/8860e9bc-46c7-47fd-b243-04efc6d8fc4c-kube-api-access-zn5q5\") pod \"8860e9bc-46c7-47fd-b243-04efc6d8fc4c\" (UID: \"8860e9bc-46c7-47fd-b243-04efc6d8fc4c\") "
Sep 17 17:02:08.867988 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:08.867820    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8860e9bc-46c7-47fd-b243-04efc6d8fc4c-home\") pod \"8860e9bc-46c7-47fd-b243-04efc6d8fc4c\" (UID: \"8860e9bc-46c7-47fd-b243-04efc6d8fc4c\") "
Sep 17 17:02:08.867988 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:08.867853    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8860e9bc-46c7-47fd-b243-04efc6d8fc4c-dshm\") pod \"8860e9bc-46c7-47fd-b243-04efc6d8fc4c\" (UID: \"8860e9bc-46c7-47fd-b243-04efc6d8fc4c\") "
Sep 17 17:02:08.867988 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:08.867906    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8860e9bc-46c7-47fd-b243-04efc6d8fc4c-tls-certs\") pod \"8860e9bc-46c7-47fd-b243-04efc6d8fc4c\" (UID: \"8860e9bc-46c7-47fd-b243-04efc6d8fc4c\") "
Sep 17 17:02:08.868297 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:08.867992    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8860e9bc-46c7-47fd-b243-04efc6d8fc4c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8860e9bc-46c7-47fd-b243-04efc6d8fc4c" (UID: "8860e9bc-46c7-47fd-b243-04efc6d8fc4c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:02:08.868297 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:08.868001    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8860e9bc-46c7-47fd-b243-04efc6d8fc4c-model-cache" (OuterVolumeSpecName: "model-cache") pod "8860e9bc-46c7-47fd-b243-04efc6d8fc4c" (UID: "8860e9bc-46c7-47fd-b243-04efc6d8fc4c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:02:08.868297 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:08.868077    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8860e9bc-46c7-47fd-b243-04efc6d8fc4c-home" (OuterVolumeSpecName: "home") pod "8860e9bc-46c7-47fd-b243-04efc6d8fc4c" (UID: "8860e9bc-46c7-47fd-b243-04efc6d8fc4c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:02:08.868297 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:08.868132    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8860e9bc-46c7-47fd-b243-04efc6d8fc4c-home\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:02:08.868297 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:08.868155    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8860e9bc-46c7-47fd-b243-04efc6d8fc4c-tmp-dir\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:02:08.868297 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:08.868170    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8860e9bc-46c7-47fd-b243-04efc6d8fc4c-model-cache\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:02:08.870211 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:08.870181    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8860e9bc-46c7-47fd-b243-04efc6d8fc4c-dshm" (OuterVolumeSpecName: "dshm") pod "8860e9bc-46c7-47fd-b243-04efc6d8fc4c" (UID: "8860e9bc-46c7-47fd-b243-04efc6d8fc4c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:02:08.870302 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:08.870181    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8860e9bc-46c7-47fd-b243-04efc6d8fc4c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8860e9bc-46c7-47fd-b243-04efc6d8fc4c" (UID: "8860e9bc-46c7-47fd-b243-04efc6d8fc4c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:02:08.870302 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:08.870222    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8860e9bc-46c7-47fd-b243-04efc6d8fc4c-kube-api-access-zn5q5" (OuterVolumeSpecName: "kube-api-access-zn5q5") pod "8860e9bc-46c7-47fd-b243-04efc6d8fc4c" (UID: "8860e9bc-46c7-47fd-b243-04efc6d8fc4c"). InnerVolumeSpecName "kube-api-access-zn5q5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:02:08.969424 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:08.969347    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn5q5\" (UniqueName: \"kubernetes.io/projected/8860e9bc-46c7-47fd-b243-04efc6d8fc4c-kube-api-access-zn5q5\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:02:08.969424 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:08.969371    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8860e9bc-46c7-47fd-b243-04efc6d8fc4c-dshm\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:02:08.969424 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:08.969382    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8860e9bc-46c7-47fd-b243-04efc6d8fc4c-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:02:09.468140 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:09.468102    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828"]
Sep 17 17:02:09.645502 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:09.645473    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs"
Sep 17 17:02:09.645664 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:09.645505    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs" event={"ID":"8860e9bc-46c7-47fd-b243-04efc6d8fc4c","Type":"ContainerDied","Data":"c0b2540ce0ffb58c435f66fcd744c8db4eed1bb76ebabaae699d69dc1a6fa33b"}
Sep 17 17:02:09.645664 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:09.645575    2566 scope.go:117] "RemoveContainer" containerID="0d78c089b8beac701f9054b9c587a4b252501b03be7a6e262a3a763ae903c699"
Sep 17 17:02:09.656353 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:02:09.656223    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:02:09.656353 ip-10-0-141-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:02:09.656353 ip-10-0-141-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:02:09.656353 ip-10-0-141-66 kubenswrapper[2566]:  > containerID="be4de00eb02136b39778104f72e97e4de7d9d74e57965c21b17fc02cb3b97b96" cmd=["/bin/sleep","15"]
Sep 17 17:02:09.656353 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:09.656264    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828" message="" err=<
Sep 17 17:02:09.656353 ip-10-0-141-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:02:09.656353 ip-10-0-141-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:02:09.656353 ip-10-0-141-66 kubenswrapper[2566]:  >
Sep 17 17:02:09.656353 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:02:09.656276    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:02:09.656353 ip-10-0-141-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:02:09.656353 ip-10-0-141-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:02:09.656353 ip-10-0-141-66 kubenswrapper[2566]:  > pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828" podUID="ce71b32a-bf09-44eb-94be-da08acf50fa6" containerName="main" containerID="cri-o://be4de00eb02136b39778104f72e97e4de7d9d74e57965c21b17fc02cb3b97b96"
Sep 17 17:02:09.656353 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:09.656315    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828" podUID="ce71b32a-bf09-44eb-94be-da08acf50fa6" containerName="main" containerID="cri-o://be4de00eb02136b39778104f72e97e4de7d9d74e57965c21b17fc02cb3b97b96" gracePeriod=60
Sep 17 17:02:09.714540 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:09.714502    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs"]
Sep 17 17:02:09.736795 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:09.736728    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4k8fs"]
Sep 17 17:02:09.949288 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:09.949260    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828"
Sep 17 17:02:09.978018 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:09.977994    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce71b32a-bf09-44eb-94be-da08acf50fa6-model-cache\") pod \"ce71b32a-bf09-44eb-94be-da08acf50fa6\" (UID: \"ce71b32a-bf09-44eb-94be-da08acf50fa6\") "
Sep 17 17:02:09.978155 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:09.978041    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce71b32a-bf09-44eb-94be-da08acf50fa6-home\") pod \"ce71b32a-bf09-44eb-94be-da08acf50fa6\" (UID: \"ce71b32a-bf09-44eb-94be-da08acf50fa6\") "
Sep 17 17:02:09.978155 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:09.978069    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqc7r\" (UniqueName: \"kubernetes.io/projected/ce71b32a-bf09-44eb-94be-da08acf50fa6-kube-api-access-zqc7r\") pod \"ce71b32a-bf09-44eb-94be-da08acf50fa6\" (UID: \"ce71b32a-bf09-44eb-94be-da08acf50fa6\") "
Sep 17 17:02:09.978155 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:09.978104    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce71b32a-bf09-44eb-94be-da08acf50fa6-dshm\") pod \"ce71b32a-bf09-44eb-94be-da08acf50fa6\" (UID: \"ce71b32a-bf09-44eb-94be-da08acf50fa6\") "
Sep 17 17:02:09.978155 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:09.978123    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce71b32a-bf09-44eb-94be-da08acf50fa6-tmp-dir\") pod \"ce71b32a-bf09-44eb-94be-da08acf50fa6\" (UID: \"ce71b32a-bf09-44eb-94be-da08acf50fa6\") "
Sep 17 17:02:09.978393 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:09.978161    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce71b32a-bf09-44eb-94be-da08acf50fa6-tls-certs\") pod \"ce71b32a-bf09-44eb-94be-da08acf50fa6\" (UID: \"ce71b32a-bf09-44eb-94be-da08acf50fa6\") "
Sep 17 17:02:09.978393 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:09.978270    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce71b32a-bf09-44eb-94be-da08acf50fa6-model-cache" (OuterVolumeSpecName: "model-cache") pod "ce71b32a-bf09-44eb-94be-da08acf50fa6" (UID: "ce71b32a-bf09-44eb-94be-da08acf50fa6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:02:09.978393 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:09.978286    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce71b32a-bf09-44eb-94be-da08acf50fa6-home" (OuterVolumeSpecName: "home") pod "ce71b32a-bf09-44eb-94be-da08acf50fa6" (UID: "ce71b32a-bf09-44eb-94be-da08acf50fa6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:02:09.978608 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:09.978413    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce71b32a-bf09-44eb-94be-da08acf50fa6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ce71b32a-bf09-44eb-94be-da08acf50fa6" (UID: "ce71b32a-bf09-44eb-94be-da08acf50fa6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:02:09.978608 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:09.978420    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce71b32a-bf09-44eb-94be-da08acf50fa6-model-cache\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:02:09.978608 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:09.978480    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce71b32a-bf09-44eb-94be-da08acf50fa6-home\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:02:09.980926 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:09.980890    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce71b32a-bf09-44eb-94be-da08acf50fa6-kube-api-access-zqc7r" (OuterVolumeSpecName: "kube-api-access-zqc7r") pod "ce71b32a-bf09-44eb-94be-da08acf50fa6" (UID: "ce71b32a-bf09-44eb-94be-da08acf50fa6"). InnerVolumeSpecName "kube-api-access-zqc7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:02:09.981188 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:09.981140    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce71b32a-bf09-44eb-94be-da08acf50fa6-dshm" (OuterVolumeSpecName: "dshm") pod "ce71b32a-bf09-44eb-94be-da08acf50fa6" (UID: "ce71b32a-bf09-44eb-94be-da08acf50fa6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:02:09.981318 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:09.981275    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce71b32a-bf09-44eb-94be-da08acf50fa6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ce71b32a-bf09-44eb-94be-da08acf50fa6" (UID: "ce71b32a-bf09-44eb-94be-da08acf50fa6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:02:10.079104 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:10.079075    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce71b32a-bf09-44eb-94be-da08acf50fa6-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:02:10.079104 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:10.079103    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqc7r\" (UniqueName: \"kubernetes.io/projected/ce71b32a-bf09-44eb-94be-da08acf50fa6-kube-api-access-zqc7r\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:02:10.079281 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:10.079113    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce71b32a-bf09-44eb-94be-da08acf50fa6-dshm\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:02:10.079281 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:10.079139    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce71b32a-bf09-44eb-94be-da08acf50fa6-tmp-dir\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:02:10.348409 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:10.348326    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8860e9bc-46c7-47fd-b243-04efc6d8fc4c" path="/var/lib/kubelet/pods/8860e9bc-46c7-47fd-b243-04efc6d8fc4c/volumes"
Sep 17 17:02:10.653856 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:10.653755    2566 generic.go:356] "Generic (PLEG): container finished" podID="ce71b32a-bf09-44eb-94be-da08acf50fa6" containerID="be4de00eb02136b39778104f72e97e4de7d9d74e57965c21b17fc02cb3b97b96" exitCode=0
Sep 17 17:02:10.653856 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:10.653807    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828"
Sep 17 17:02:10.653856 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:10.653845    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828" event={"ID":"ce71b32a-bf09-44eb-94be-da08acf50fa6","Type":"ContainerDied","Data":"be4de00eb02136b39778104f72e97e4de7d9d74e57965c21b17fc02cb3b97b96"}
Sep 17 17:02:10.654373 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:10.653903    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828" event={"ID":"ce71b32a-bf09-44eb-94be-da08acf50fa6","Type":"ContainerDied","Data":"cb863dd0f76b61cbcbdf851c1fb4edce8fcd6662135e45c0b2b7a17820cc914b"}
Sep 17 17:02:10.654373 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:10.653934    2566 scope.go:117] "RemoveContainer" containerID="be4de00eb02136b39778104f72e97e4de7d9d74e57965c21b17fc02cb3b97b96"
Sep 17 17:02:10.663007 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:10.662991    2566 scope.go:117] "RemoveContainer" containerID="be4de00eb02136b39778104f72e97e4de7d9d74e57965c21b17fc02cb3b97b96"
Sep 17 17:02:10.663252 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:02:10.663234    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be4de00eb02136b39778104f72e97e4de7d9d74e57965c21b17fc02cb3b97b96\": container with ID starting with be4de00eb02136b39778104f72e97e4de7d9d74e57965c21b17fc02cb3b97b96 not found: ID does not exist" containerID="be4de00eb02136b39778104f72e97e4de7d9d74e57965c21b17fc02cb3b97b96"
Sep 17 17:02:10.663322 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:10.663257    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be4de00eb02136b39778104f72e97e4de7d9d74e57965c21b17fc02cb3b97b96"} err="failed to get container status \"be4de00eb02136b39778104f72e97e4de7d9d74e57965c21b17fc02cb3b97b96\": rpc error: code = NotFound desc = could not find container \"be4de00eb02136b39778104f72e97e4de7d9d74e57965c21b17fc02cb3b97b96\": container with ID starting with be4de00eb02136b39778104f72e97e4de7d9d74e57965c21b17fc02cb3b97b96 not found: ID does not exist"
Sep 17 17:02:10.681138 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:10.681112    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828"]
Sep 17 17:02:10.684488 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:10.684466    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj6828"]
Sep 17 17:02:12.347078 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:12.347042    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce71b32a-bf09-44eb-94be-da08acf50fa6" path="/var/lib/kubelet/pods/ce71b32a-bf09-44eb-94be-da08acf50fa6/volumes"
Sep 17 17:02:12.995088 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:12.995048    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r"]
Sep 17 17:02:12.995294 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:12.995266    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r" podUID="cafcd578-1d33-4039-8d3f-703580427a78" containerName="istio-proxy" containerID="cri-o://ecec9ec8be8d580ad43d4653729f466e988ba4886f5b0c4f5a8b94de8582b930" gracePeriod=30
Sep 17 17:02:15.241044 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:15.241018    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r"
Sep 17 17:02:15.319125 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:15.319095    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cafcd578-1d33-4039-8d3f-703580427a78-workload-socket\") pod \"cafcd578-1d33-4039-8d3f-703580427a78\" (UID: \"cafcd578-1d33-4039-8d3f-703580427a78\") "
Sep 17 17:02:15.319279 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:15.319137    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cafcd578-1d33-4039-8d3f-703580427a78-credential-socket\") pod \"cafcd578-1d33-4039-8d3f-703580427a78\" (UID: \"cafcd578-1d33-4039-8d3f-703580427a78\") "
Sep 17 17:02:15.319279 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:15.319161    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cafcd578-1d33-4039-8d3f-703580427a78-istio-data\") pod \"cafcd578-1d33-4039-8d3f-703580427a78\" (UID: \"cafcd578-1d33-4039-8d3f-703580427a78\") "
Sep 17 17:02:15.319279 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:15.319192    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xb2kj\" (UniqueName: \"kubernetes.io/projected/cafcd578-1d33-4039-8d3f-703580427a78-kube-api-access-xb2kj\") pod \"cafcd578-1d33-4039-8d3f-703580427a78\" (UID: \"cafcd578-1d33-4039-8d3f-703580427a78\") "
Sep 17 17:02:15.319279 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:15.319238    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cafcd578-1d33-4039-8d3f-703580427a78-istio-envoy\") pod \"cafcd578-1d33-4039-8d3f-703580427a78\" (UID: \"cafcd578-1d33-4039-8d3f-703580427a78\") "
Sep 17 17:02:15.319279 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:15.319260    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cafcd578-1d33-4039-8d3f-703580427a78-workload-certs\") pod \"cafcd578-1d33-4039-8d3f-703580427a78\" (UID: \"cafcd578-1d33-4039-8d3f-703580427a78\") "
Sep 17 17:02:15.319591 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:15.319293    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cafcd578-1d33-4039-8d3f-703580427a78-istiod-ca-cert\") pod \"cafcd578-1d33-4039-8d3f-703580427a78\" (UID: \"cafcd578-1d33-4039-8d3f-703580427a78\") "
Sep 17 17:02:15.319591 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:15.319323    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cafcd578-1d33-4039-8d3f-703580427a78-istio-token\") pod \"cafcd578-1d33-4039-8d3f-703580427a78\" (UID: \"cafcd578-1d33-4039-8d3f-703580427a78\") "
Sep 17 17:02:15.319591 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:15.319345    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cafcd578-1d33-4039-8d3f-703580427a78-istio-podinfo\") pod \"cafcd578-1d33-4039-8d3f-703580427a78\" (UID: \"cafcd578-1d33-4039-8d3f-703580427a78\") "
Sep 17 17:02:15.319591 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:15.319398    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cafcd578-1d33-4039-8d3f-703580427a78-istio-data" (OuterVolumeSpecName: "istio-data") pod "cafcd578-1d33-4039-8d3f-703580427a78" (UID: "cafcd578-1d33-4039-8d3f-703580427a78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:02:15.319591 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:15.319423    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cafcd578-1d33-4039-8d3f-703580427a78-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cafcd578-1d33-4039-8d3f-703580427a78" (UID: "cafcd578-1d33-4039-8d3f-703580427a78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:02:15.319853 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:15.319608    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cafcd578-1d33-4039-8d3f-703580427a78-workload-socket\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:02:15.319853 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:15.319626    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cafcd578-1d33-4039-8d3f-703580427a78-istio-data\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:02:15.319954 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:15.319893    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cafcd578-1d33-4039-8d3f-703580427a78-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cafcd578-1d33-4039-8d3f-703580427a78" (UID: "cafcd578-1d33-4039-8d3f-703580427a78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:02:15.319954 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:15.319914    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cafcd578-1d33-4039-8d3f-703580427a78-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cafcd578-1d33-4039-8d3f-703580427a78" (UID: "cafcd578-1d33-4039-8d3f-703580427a78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:02:15.320103 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:15.320073    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cafcd578-1d33-4039-8d3f-703580427a78-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cafcd578-1d33-4039-8d3f-703580427a78" (UID: "cafcd578-1d33-4039-8d3f-703580427a78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:02:15.322096 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:15.322060    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cafcd578-1d33-4039-8d3f-703580427a78-istio-token" (OuterVolumeSpecName: "istio-token") pod "cafcd578-1d33-4039-8d3f-703580427a78" (UID: "cafcd578-1d33-4039-8d3f-703580427a78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:02:15.322198 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:15.322132    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cafcd578-1d33-4039-8d3f-703580427a78-kube-api-access-xb2kj" (OuterVolumeSpecName: "kube-api-access-xb2kj") pod "cafcd578-1d33-4039-8d3f-703580427a78" (UID: "cafcd578-1d33-4039-8d3f-703580427a78"). InnerVolumeSpecName "kube-api-access-xb2kj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:02:15.322271 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:15.322190    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cafcd578-1d33-4039-8d3f-703580427a78-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cafcd578-1d33-4039-8d3f-703580427a78" (UID: "cafcd578-1d33-4039-8d3f-703580427a78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:02:15.322271 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:15.322248    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cafcd578-1d33-4039-8d3f-703580427a78-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cafcd578-1d33-4039-8d3f-703580427a78" (UID: "cafcd578-1d33-4039-8d3f-703580427a78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 17:02:15.420549 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:15.420517    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cafcd578-1d33-4039-8d3f-703580427a78-credential-socket\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:02:15.420549 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:15.420545    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb2kj\" (UniqueName: \"kubernetes.io/projected/cafcd578-1d33-4039-8d3f-703580427a78-kube-api-access-xb2kj\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:02:15.420549 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:15.420555    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cafcd578-1d33-4039-8d3f-703580427a78-istio-envoy\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:02:15.420770 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:15.420563    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cafcd578-1d33-4039-8d3f-703580427a78-workload-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:02:15.420770 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:15.420572    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cafcd578-1d33-4039-8d3f-703580427a78-istiod-ca-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:02:15.420770 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:15.420579    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cafcd578-1d33-4039-8d3f-703580427a78-istio-token\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:02:15.420770 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:15.420586    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cafcd578-1d33-4039-8d3f-703580427a78-istio-podinfo\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:02:15.683838 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:15.683756    2566 generic.go:356] "Generic (PLEG): container finished" podID="cafcd578-1d33-4039-8d3f-703580427a78" containerID="ecec9ec8be8d580ad43d4653729f466e988ba4886f5b0c4f5a8b94de8582b930" exitCode=0
Sep 17 17:02:15.683838 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:15.683805    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r"
Sep 17 17:02:15.684010 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:15.683801    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r" event={"ID":"cafcd578-1d33-4039-8d3f-703580427a78","Type":"ContainerDied","Data":"ecec9ec8be8d580ad43d4653729f466e988ba4886f5b0c4f5a8b94de8582b930"}
Sep 17 17:02:15.684010 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:15.683934    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r" event={"ID":"cafcd578-1d33-4039-8d3f-703580427a78","Type":"ContainerDied","Data":"09a9a1752c80b9d1b5b6bc74fb0a7577b9783b5d7f3e350581ce0f05666e1a7a"}
Sep 17 17:02:15.684010 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:15.683963    2566 scope.go:117] "RemoveContainer" containerID="ecec9ec8be8d580ad43d4653729f466e988ba4886f5b0c4f5a8b94de8582b930"
Sep 17 17:02:15.692802 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:15.692783    2566 scope.go:117] "RemoveContainer" containerID="ecec9ec8be8d580ad43d4653729f466e988ba4886f5b0c4f5a8b94de8582b930"
Sep 17 17:02:15.693084 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:02:15.693062    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecec9ec8be8d580ad43d4653729f466e988ba4886f5b0c4f5a8b94de8582b930\": container with ID starting with ecec9ec8be8d580ad43d4653729f466e988ba4886f5b0c4f5a8b94de8582b930 not found: ID does not exist" containerID="ecec9ec8be8d580ad43d4653729f466e988ba4886f5b0c4f5a8b94de8582b930"
Sep 17 17:02:15.693159 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:15.693093    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecec9ec8be8d580ad43d4653729f466e988ba4886f5b0c4f5a8b94de8582b930"} err="failed to get container status \"ecec9ec8be8d580ad43d4653729f466e988ba4886f5b0c4f5a8b94de8582b930\": rpc error: code = NotFound desc = could not find container \"ecec9ec8be8d580ad43d4653729f466e988ba4886f5b0c4f5a8b94de8582b930\": container with ID starting with ecec9ec8be8d580ad43d4653729f466e988ba4886f5b0c4f5a8b94de8582b930 not found: ID does not exist"
Sep 17 17:02:15.710851 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:15.710828    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r"]
Sep 17 17:02:15.720469 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:15.720423    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-58n9r"]
Sep 17 17:02:16.346102 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:16.346059    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cafcd578-1d33-4039-8d3f-703580427a78" path="/var/lib/kubelet/pods/cafcd578-1d33-4039-8d3f-703580427a78/volumes"
Sep 17 17:02:22.326075 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.326039    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt"]
Sep 17 17:02:22.326481 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.326353    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cafcd578-1d33-4039-8d3f-703580427a78" containerName="istio-proxy"
Sep 17 17:02:22.326481 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.326364    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cafcd578-1d33-4039-8d3f-703580427a78" containerName="istio-proxy"
Sep 17 17:02:22.326481 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.326375    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce71b32a-bf09-44eb-94be-da08acf50fa6" containerName="main"
Sep 17 17:02:22.326481 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.326380    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce71b32a-bf09-44eb-94be-da08acf50fa6" containerName="main"
Sep 17 17:02:22.326481 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.326398    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8860e9bc-46c7-47fd-b243-04efc6d8fc4c" containerName="main"
Sep 17 17:02:22.326481 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.326403    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8860e9bc-46c7-47fd-b243-04efc6d8fc4c" containerName="main"
Sep 17 17:02:22.326481 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.326485    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce71b32a-bf09-44eb-94be-da08acf50fa6" containerName="main"
Sep 17 17:02:22.326745 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.326494    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="cafcd578-1d33-4039-8d3f-703580427a78" containerName="istio-proxy"
Sep 17 17:02:22.326745 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.326502    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8860e9bc-46c7-47fd-b243-04efc6d8fc4c" containerName="main"
Sep 17 17:02:22.329413 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.329397    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt"
Sep 17 17:02:22.334810 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:02:22.334770    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"auth-enabled-test-kserve-self-signed-certs\" is forbidden: User \"system:node:ip-10-0-141-66.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\": no relationship found between node 'ip-10-0-141-66.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\"" type="*v1.Secret"
Sep 17 17:02:22.334968 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.334853    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:02:22.334968 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.334893    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 17 17:02:22.335425 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.335410    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Sep 17 17:02:22.336160 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.335874    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 17 17:02:22.336160 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.335956    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-s92kj\""
Sep 17 17:02:22.365482 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.365428    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt"]
Sep 17 17:02:22.372130 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.372105    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fa3f78d-cd66-468c-b73c-6c83e138f099-dshm\") pod \"auth-enabled-test-kserve-8cb6d4fb5-z2rrt\" (UID: \"7fa3f78d-cd66-468c-b73c-6c83e138f099\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt"
Sep 17 17:02:22.372284 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.372136    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fa3f78d-cd66-468c-b73c-6c83e138f099-tls-certs\") pod \"auth-enabled-test-kserve-8cb6d4fb5-z2rrt\" (UID: \"7fa3f78d-cd66-468c-b73c-6c83e138f099\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt"
Sep 17 17:02:22.372284 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.372168    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fa3f78d-cd66-468c-b73c-6c83e138f099-tmp-dir\") pod \"auth-enabled-test-kserve-8cb6d4fb5-z2rrt\" (UID: \"7fa3f78d-cd66-468c-b73c-6c83e138f099\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt"
Sep 17 17:02:22.372284 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.372218    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fa3f78d-cd66-468c-b73c-6c83e138f099-cabundle-cert\") pod \"auth-enabled-test-kserve-8cb6d4fb5-z2rrt\" (UID: \"7fa3f78d-cd66-468c-b73c-6c83e138f099\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt"
Sep 17 17:02:22.372483 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.372301    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fa3f78d-cd66-468c-b73c-6c83e138f099-home\") pod \"auth-enabled-test-kserve-8cb6d4fb5-z2rrt\" (UID: \"7fa3f78d-cd66-468c-b73c-6c83e138f099\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt"
Sep 17 17:02:22.372483 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.372360    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rns4l\" (UniqueName: \"kubernetes.io/projected/7fa3f78d-cd66-468c-b73c-6c83e138f099-kube-api-access-rns4l\") pod \"auth-enabled-test-kserve-8cb6d4fb5-z2rrt\" (UID: \"7fa3f78d-cd66-468c-b73c-6c83e138f099\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt"
Sep 17 17:02:22.372483 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.372462    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fa3f78d-cd66-468c-b73c-6c83e138f099-model-cache\") pod \"auth-enabled-test-kserve-8cb6d4fb5-z2rrt\" (UID: \"7fa3f78d-cd66-468c-b73c-6c83e138f099\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt"
Sep 17 17:02:22.372611 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.372487    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fa3f78d-cd66-468c-b73c-6c83e138f099-kserve-provision-location\") pod \"auth-enabled-test-kserve-8cb6d4fb5-z2rrt\" (UID: \"7fa3f78d-cd66-468c-b73c-6c83e138f099\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt"
Sep 17 17:02:22.473061 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.473029    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fa3f78d-cd66-468c-b73c-6c83e138f099-tmp-dir\") pod \"auth-enabled-test-kserve-8cb6d4fb5-z2rrt\" (UID: \"7fa3f78d-cd66-468c-b73c-6c83e138f099\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt"
Sep 17 17:02:22.473061 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.473061    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fa3f78d-cd66-468c-b73c-6c83e138f099-cabundle-cert\") pod \"auth-enabled-test-kserve-8cb6d4fb5-z2rrt\" (UID: \"7fa3f78d-cd66-468c-b73c-6c83e138f099\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt"
Sep 17 17:02:22.473264 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.473211    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fa3f78d-cd66-468c-b73c-6c83e138f099-home\") pod \"auth-enabled-test-kserve-8cb6d4fb5-z2rrt\" (UID: \"7fa3f78d-cd66-468c-b73c-6c83e138f099\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt"
Sep 17 17:02:22.473301 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.473261    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rns4l\" (UniqueName: \"kubernetes.io/projected/7fa3f78d-cd66-468c-b73c-6c83e138f099-kube-api-access-rns4l\") pod \"auth-enabled-test-kserve-8cb6d4fb5-z2rrt\" (UID: \"7fa3f78d-cd66-468c-b73c-6c83e138f099\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt"
Sep 17 17:02:22.473345 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.473329    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fa3f78d-cd66-468c-b73c-6c83e138f099-model-cache\") pod \"auth-enabled-test-kserve-8cb6d4fb5-z2rrt\" (UID: \"7fa3f78d-cd66-468c-b73c-6c83e138f099\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt"
Sep 17 17:02:22.473392 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.473354    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fa3f78d-cd66-468c-b73c-6c83e138f099-kserve-provision-location\") pod \"auth-enabled-test-kserve-8cb6d4fb5-z2rrt\" (UID: \"7fa3f78d-cd66-468c-b73c-6c83e138f099\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt"
Sep 17 17:02:22.473754 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.473523    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fa3f78d-cd66-468c-b73c-6c83e138f099-tmp-dir\") pod \"auth-enabled-test-kserve-8cb6d4fb5-z2rrt\" (UID: \"7fa3f78d-cd66-468c-b73c-6c83e138f099\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt"
Sep 17 17:02:22.473754 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.473504    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fa3f78d-cd66-468c-b73c-6c83e138f099-dshm\") pod \"auth-enabled-test-kserve-8cb6d4fb5-z2rrt\" (UID: \"7fa3f78d-cd66-468c-b73c-6c83e138f099\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt"
Sep 17 17:02:22.473754 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.473579    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fa3f78d-cd66-468c-b73c-6c83e138f099-home\") pod \"auth-enabled-test-kserve-8cb6d4fb5-z2rrt\" (UID: \"7fa3f78d-cd66-468c-b73c-6c83e138f099\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt"
Sep 17 17:02:22.473754 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.473594    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fa3f78d-cd66-468c-b73c-6c83e138f099-tls-certs\") pod \"auth-enabled-test-kserve-8cb6d4fb5-z2rrt\" (UID: \"7fa3f78d-cd66-468c-b73c-6c83e138f099\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt"
Sep 17 17:02:22.473754 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.473673    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fa3f78d-cd66-468c-b73c-6c83e138f099-kserve-provision-location\") pod \"auth-enabled-test-kserve-8cb6d4fb5-z2rrt\" (UID: \"7fa3f78d-cd66-468c-b73c-6c83e138f099\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt"
Sep 17 17:02:22.473754 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.473754    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fa3f78d-cd66-468c-b73c-6c83e138f099-model-cache\") pod \"auth-enabled-test-kserve-8cb6d4fb5-z2rrt\" (UID: \"7fa3f78d-cd66-468c-b73c-6c83e138f099\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt"
Sep 17 17:02:22.474021 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.473842    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fa3f78d-cd66-468c-b73c-6c83e138f099-cabundle-cert\") pod \"auth-enabled-test-kserve-8cb6d4fb5-z2rrt\" (UID: \"7fa3f78d-cd66-468c-b73c-6c83e138f099\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt"
Sep 17 17:02:22.475781 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.475763    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fa3f78d-cd66-468c-b73c-6c83e138f099-dshm\") pod \"auth-enabled-test-kserve-8cb6d4fb5-z2rrt\" (UID: \"7fa3f78d-cd66-468c-b73c-6c83e138f099\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt"
Sep 17 17:02:22.494457 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.494416    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rns4l\" (UniqueName: \"kubernetes.io/projected/7fa3f78d-cd66-468c-b73c-6c83e138f099-kube-api-access-rns4l\") pod \"auth-enabled-test-kserve-8cb6d4fb5-z2rrt\" (UID: \"7fa3f78d-cd66-468c-b73c-6c83e138f099\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt"
Sep 17 17:02:22.672975 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.672894    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-n5xx5"]
Sep 17 17:02:22.675261 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.675243    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-n5xx5"
Sep 17 17:02:22.678278 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.678254    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-epp-sa-dockercfg-ctj4q\""
Sep 17 17:02:22.709340 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.709312    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-n5xx5"]
Sep 17 17:02:22.775580 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.775547    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab90fd45-9d64-4e50-9994-265804f78370-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-n5xx5\" (UID: \"ab90fd45-9d64-4e50-9994-265804f78370\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-n5xx5"
Sep 17 17:02:22.775726 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.775656    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lmv2\" (UniqueName: \"kubernetes.io/projected/ab90fd45-9d64-4e50-9994-265804f78370-kube-api-access-6lmv2\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-n5xx5\" (UID: \"ab90fd45-9d64-4e50-9994-265804f78370\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-n5xx5"
Sep 17 17:02:22.876555 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.876515    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lmv2\" (UniqueName: \"kubernetes.io/projected/ab90fd45-9d64-4e50-9994-265804f78370-kube-api-access-6lmv2\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-n5xx5\" (UID: \"ab90fd45-9d64-4e50-9994-265804f78370\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-n5xx5"
Sep 17 17:02:22.876729 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.876585    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab90fd45-9d64-4e50-9994-265804f78370-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-n5xx5\" (UID: \"ab90fd45-9d64-4e50-9994-265804f78370\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-n5xx5"
Sep 17 17:02:22.894136 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:22.894110    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lmv2\" (UniqueName: \"kubernetes.io/projected/ab90fd45-9d64-4e50-9994-265804f78370-kube-api-access-6lmv2\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-n5xx5\" (UID: \"ab90fd45-9d64-4e50-9994-265804f78370\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-n5xx5"
Sep 17 17:02:23.169767 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:23.169735    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 17 17:02:23.177011 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:23.176986    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fa3f78d-cd66-468c-b73c-6c83e138f099-tls-certs\") pod \"auth-enabled-test-kserve-8cb6d4fb5-z2rrt\" (UID: \"7fa3f78d-cd66-468c-b73c-6c83e138f099\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt"
Sep 17 17:02:23.179471 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:23.179423    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab90fd45-9d64-4e50-9994-265804f78370-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-n5xx5\" (UID: \"ab90fd45-9d64-4e50-9994-265804f78370\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-n5xx5"
Sep 17 17:02:23.239693 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:23.239655    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt"
Sep 17 17:02:23.285246 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:23.285210    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-n5xx5"
Sep 17 17:02:23.534450 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:23.534360    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt"]
Sep 17 17:02:23.539543 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:02:23.539516    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa3f78d_cd66_468c_b73c_6c83e138f099.slice/crio-aeeaa801cbec0485ec9c380a385e03511d7adcf02da8607497258549526c1594 WatchSource:0}: Error finding container aeeaa801cbec0485ec9c380a385e03511d7adcf02da8607497258549526c1594: Status 404 returned error can't find the container with id aeeaa801cbec0485ec9c380a385e03511d7adcf02da8607497258549526c1594
Sep 17 17:02:23.596722 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:02:23.596697    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab90fd45_9d64_4e50_9994_265804f78370.slice/crio-ea57b785e4045578d2bd87d1891d77f70e13760324d2b85215b5f418bbba11c6 WatchSource:0}: Error finding container ea57b785e4045578d2bd87d1891d77f70e13760324d2b85215b5f418bbba11c6: Status 404 returned error can't find the container with id ea57b785e4045578d2bd87d1891d77f70e13760324d2b85215b5f418bbba11c6
Sep 17 17:02:23.605596 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:23.605574    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-n5xx5"]
Sep 17 17:02:23.727826 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:23.727786    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-n5xx5" event={"ID":"ab90fd45-9d64-4e50-9994-265804f78370","Type":"ContainerStarted","Data":"397ce862ff5a451bae15249d8c7bb94f28f7c9ed335f7bde10d69514effde72e"}
Sep 17 17:02:23.728029 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:23.727854    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-n5xx5" event={"ID":"ab90fd45-9d64-4e50-9994-265804f78370","Type":"ContainerStarted","Data":"ea57b785e4045578d2bd87d1891d77f70e13760324d2b85215b5f418bbba11c6"}
Sep 17 17:02:23.728029 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:23.727978    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-n5xx5"
Sep 17 17:02:23.729719 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:23.729691    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt" event={"ID":"7fa3f78d-cd66-468c-b73c-6c83e138f099","Type":"ContainerStarted","Data":"2e18456a607077c6e9c0291ba5a1a8e585bb13e8c8c054851e4c0ae85096ed35"}
Sep 17 17:02:23.729831 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:23.729740    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt" event={"ID":"7fa3f78d-cd66-468c-b73c-6c83e138f099","Type":"ContainerStarted","Data":"aeeaa801cbec0485ec9c380a385e03511d7adcf02da8607497258549526c1594"}
Sep 17 17:02:23.792166 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:23.792086    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-n5xx5" podStartSLOduration=1.792064199 podStartE2EDuration="1.792064199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:02:23.751827197 +0000 UTC m=+1229.646306338" watchObservedRunningTime="2026-09-17 17:02:23.792064199 +0000 UTC m=+1229.686543339"
Sep 17 17:02:26.744961 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:26.744930    2566 generic.go:356] "Generic (PLEG): container finished" podID="7fa3f78d-cd66-468c-b73c-6c83e138f099" containerID="2e18456a607077c6e9c0291ba5a1a8e585bb13e8c8c054851e4c0ae85096ed35" exitCode=0
Sep 17 17:02:26.745398 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:26.744966    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt" event={"ID":"7fa3f78d-cd66-468c-b73c-6c83e138f099","Type":"ContainerDied","Data":"2e18456a607077c6e9c0291ba5a1a8e585bb13e8c8c054851e4c0ae85096ed35"}
Sep 17 17:02:53.919886 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:53.919847    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt" event={"ID":"7fa3f78d-cd66-468c-b73c-6c83e138f099","Type":"ContainerStarted","Data":"e30ba3779579a5eae9b2fe2a27f08a30377961035c02ebb156077bcfc25cf285"}
Sep 17 17:02:53.952475 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:53.951510    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt" podStartSLOduration=2.577533028 podStartE2EDuration="31.951490255s" totalImagesPullingTime="27.072452084s" totalInitContainerRuntime="2.301505143s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:02:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:02:53.949066976 +0000 UTC m=+1259.843546115" watchObservedRunningTime="2026-09-17 17:02:53.951490255 +0000 UTC m=+1259.845969395"
Sep 17 17:02:54.240223 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:54.240137    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt"
Sep 17 17:02:54.737027 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:02:54.736991    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-n5xx5"
Sep 17 17:03:03.240412 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:03.240364    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt"
Sep 17 17:03:03.242033 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:03.242003    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt" podUID="7fa3f78d-cd66-468c-b73c-6c83e138f099" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Sep 17 17:03:05.700804 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:05.700769    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj"]
Sep 17 17:03:05.940244 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:05.940204    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj"]
Sep 17 17:03:05.940412 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:05.940309    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj"
Sep 17 17:03:05.945225 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:05.945199    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 17 17:03:05.945225 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:05.945213    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 17 17:03:05.945626 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:05.945605    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 17 17:03:05.945704 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:05.945637    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-4d4vv\""
Sep 17 17:03:05.958459 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:05.958383    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14e3cda8-d9a1-496c-8e53-55be9e5130ee-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj\" (UID: \"14e3cda8-d9a1-496c-8e53-55be9e5130ee\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj"
Sep 17 17:03:05.958459 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:05.958411    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14e3cda8-d9a1-496c-8e53-55be9e5130ee-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj\" (UID: \"14e3cda8-d9a1-496c-8e53-55be9e5130ee\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj"
Sep 17 17:03:05.958592 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:05.958469    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14e3cda8-d9a1-496c-8e53-55be9e5130ee-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj\" (UID: \"14e3cda8-d9a1-496c-8e53-55be9e5130ee\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj"
Sep 17 17:03:05.958592 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:05.958492    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnr89\" (UniqueName: \"kubernetes.io/projected/14e3cda8-d9a1-496c-8e53-55be9e5130ee-kube-api-access-vnr89\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj\" (UID: \"14e3cda8-d9a1-496c-8e53-55be9e5130ee\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj"
Sep 17 17:03:05.958592 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:05.958512    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14e3cda8-d9a1-496c-8e53-55be9e5130ee-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj\" (UID: \"14e3cda8-d9a1-496c-8e53-55be9e5130ee\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj"
Sep 17 17:03:05.958592 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:05.958530    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14e3cda8-d9a1-496c-8e53-55be9e5130ee-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj\" (UID: \"14e3cda8-d9a1-496c-8e53-55be9e5130ee\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj"
Sep 17 17:03:06.059069 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:06.059018    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14e3cda8-d9a1-496c-8e53-55be9e5130ee-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj\" (UID: \"14e3cda8-d9a1-496c-8e53-55be9e5130ee\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj"
Sep 17 17:03:06.059069 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:06.059059    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14e3cda8-d9a1-496c-8e53-55be9e5130ee-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj\" (UID: \"14e3cda8-d9a1-496c-8e53-55be9e5130ee\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj"
Sep 17 17:03:06.059452 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:06.059406    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14e3cda8-d9a1-496c-8e53-55be9e5130ee-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj\" (UID: \"14e3cda8-d9a1-496c-8e53-55be9e5130ee\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj"
Sep 17 17:03:06.059546 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:06.059481    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnr89\" (UniqueName: \"kubernetes.io/projected/14e3cda8-d9a1-496c-8e53-55be9e5130ee-kube-api-access-vnr89\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj\" (UID: \"14e3cda8-d9a1-496c-8e53-55be9e5130ee\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj"
Sep 17 17:03:06.059546 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:06.059518    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14e3cda8-d9a1-496c-8e53-55be9e5130ee-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj\" (UID: \"14e3cda8-d9a1-496c-8e53-55be9e5130ee\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj"
Sep 17 17:03:06.059667 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:06.059548    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14e3cda8-d9a1-496c-8e53-55be9e5130ee-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj\" (UID: \"14e3cda8-d9a1-496c-8e53-55be9e5130ee\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj"
Sep 17 17:03:06.059667 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:06.059484    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14e3cda8-d9a1-496c-8e53-55be9e5130ee-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj\" (UID: \"14e3cda8-d9a1-496c-8e53-55be9e5130ee\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj"
Sep 17 17:03:06.059988 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:06.059963    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14e3cda8-d9a1-496c-8e53-55be9e5130ee-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj\" (UID: \"14e3cda8-d9a1-496c-8e53-55be9e5130ee\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj"
Sep 17 17:03:06.060095 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:06.059971    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14e3cda8-d9a1-496c-8e53-55be9e5130ee-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj\" (UID: \"14e3cda8-d9a1-496c-8e53-55be9e5130ee\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj"
Sep 17 17:03:06.061842 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:06.061816    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14e3cda8-d9a1-496c-8e53-55be9e5130ee-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj\" (UID: \"14e3cda8-d9a1-496c-8e53-55be9e5130ee\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj"
Sep 17 17:03:06.061994 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:06.061973    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14e3cda8-d9a1-496c-8e53-55be9e5130ee-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj\" (UID: \"14e3cda8-d9a1-496c-8e53-55be9e5130ee\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj"
Sep 17 17:03:06.086398 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:06.086364    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnr89\" (UniqueName: \"kubernetes.io/projected/14e3cda8-d9a1-496c-8e53-55be9e5130ee-kube-api-access-vnr89\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj\" (UID: \"14e3cda8-d9a1-496c-8e53-55be9e5130ee\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj"
Sep 17 17:03:06.251161 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:06.251078    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj"
Sep 17 17:03:06.552137 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:06.552108    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj"]
Sep 17 17:03:06.554534 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:03:06.554504    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14e3cda8_d9a1_496c_8e53_55be9e5130ee.slice/crio-043db90159e17cb039a90c764bd5f70d469f286b92522fb1148daaae9093d5fe WatchSource:0}: Error finding container 043db90159e17cb039a90c764bd5f70d469f286b92522fb1148daaae9093d5fe: Status 404 returned error can't find the container with id 043db90159e17cb039a90c764bd5f70d469f286b92522fb1148daaae9093d5fe
Sep 17 17:03:06.993501 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:06.993370    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj" event={"ID":"14e3cda8-d9a1-496c-8e53-55be9e5130ee","Type":"ContainerStarted","Data":"0d01af9f441c803ea9272961f5a19297fe693a38a3745b837903896aaf9cdb7b"}
Sep 17 17:03:06.993501 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:06.993450    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj" event={"ID":"14e3cda8-d9a1-496c-8e53-55be9e5130ee","Type":"ContainerStarted","Data":"043db90159e17cb039a90c764bd5f70d469f286b92522fb1148daaae9093d5fe"}
Sep 17 17:03:07.251664 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:07.251573    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj"
Sep 17 17:03:13.240026 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:13.239978    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt" podUID="7fa3f78d-cd66-468c-b73c-6c83e138f099" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Sep 17 17:03:16.251500 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:16.251463    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj"
Sep 17 17:03:16.265420 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:16.265389    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj"
Sep 17 17:03:16.290262 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:16.290213    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj" podStartSLOduration=11.290198245 podStartE2EDuration="11.290198245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:03:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:03:07.063222602 +0000 UTC m=+1272.957701742" watchObservedRunningTime="2026-09-17 17:03:16.290198245 +0000 UTC m=+1282.184677384"
Sep 17 17:03:17.059209 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:17.059178    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj"
Sep 17 17:03:23.240901 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:23.240854    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt" podUID="7fa3f78d-cd66-468c-b73c-6c83e138f099" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Sep 17 17:03:33.240345 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:33.240305    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt" podUID="7fa3f78d-cd66-468c-b73c-6c83e138f099" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Sep 17 17:03:40.651610 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:40.651573    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj"]
Sep 17 17:03:40.661902 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:03:40.661862    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:03:40.661902 ip-10-0-141-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:03:40.661902 ip-10-0-141-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:03:40.661902 ip-10-0-141-66 kubenswrapper[2566]:  > containerID="0d01af9f441c803ea9272961f5a19297fe693a38a3745b837903896aaf9cdb7b" cmd=["/bin/sleep","15"]
Sep 17 17:03:40.662212 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:40.661924    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj" message="" err=<
Sep 17 17:03:40.662212 ip-10-0-141-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:03:40.662212 ip-10-0-141-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:03:40.662212 ip-10-0-141-66 kubenswrapper[2566]:  >
Sep 17 17:03:40.662212 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:03:40.661941    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:03:40.662212 ip-10-0-141-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:03:40.662212 ip-10-0-141-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:03:40.662212 ip-10-0-141-66 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj" podUID="14e3cda8-d9a1-496c-8e53-55be9e5130ee" containerName="main" containerID="cri-o://0d01af9f441c803ea9272961f5a19297fe693a38a3745b837903896aaf9cdb7b"
Sep 17 17:03:40.662212 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:40.661996    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj" podUID="14e3cda8-d9a1-496c-8e53-55be9e5130ee" containerName="main" containerID="cri-o://0d01af9f441c803ea9272961f5a19297fe693a38a3745b837903896aaf9cdb7b" gracePeriod=60
Sep 17 17:03:41.011800 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:41.011771    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj"
Sep 17 17:03:41.173465 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:41.173419    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vnr89\" (UniqueName: \"kubernetes.io/projected/14e3cda8-d9a1-496c-8e53-55be9e5130ee-kube-api-access-vnr89\") pod \"14e3cda8-d9a1-496c-8e53-55be9e5130ee\" (UID: \"14e3cda8-d9a1-496c-8e53-55be9e5130ee\") "
Sep 17 17:03:41.173654 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:41.173486    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14e3cda8-d9a1-496c-8e53-55be9e5130ee-tls-certs\") pod \"14e3cda8-d9a1-496c-8e53-55be9e5130ee\" (UID: \"14e3cda8-d9a1-496c-8e53-55be9e5130ee\") "
Sep 17 17:03:41.173654 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:41.173531    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14e3cda8-d9a1-496c-8e53-55be9e5130ee-tmp-dir\") pod \"14e3cda8-d9a1-496c-8e53-55be9e5130ee\" (UID: \"14e3cda8-d9a1-496c-8e53-55be9e5130ee\") "
Sep 17 17:03:41.173654 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:41.173561    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14e3cda8-d9a1-496c-8e53-55be9e5130ee-dshm\") pod \"14e3cda8-d9a1-496c-8e53-55be9e5130ee\" (UID: \"14e3cda8-d9a1-496c-8e53-55be9e5130ee\") "
Sep 17 17:03:41.173654 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:41.173602    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14e3cda8-d9a1-496c-8e53-55be9e5130ee-home\") pod \"14e3cda8-d9a1-496c-8e53-55be9e5130ee\" (UID: \"14e3cda8-d9a1-496c-8e53-55be9e5130ee\") "
Sep 17 17:03:41.173654 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:41.173636    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14e3cda8-d9a1-496c-8e53-55be9e5130ee-model-cache\") pod \"14e3cda8-d9a1-496c-8e53-55be9e5130ee\" (UID: \"14e3cda8-d9a1-496c-8e53-55be9e5130ee\") "
Sep 17 17:03:41.173930 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:41.173821    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14e3cda8-d9a1-496c-8e53-55be9e5130ee-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "14e3cda8-d9a1-496c-8e53-55be9e5130ee" (UID: "14e3cda8-d9a1-496c-8e53-55be9e5130ee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:03:41.174074 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:41.174027    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14e3cda8-d9a1-496c-8e53-55be9e5130ee-home" (OuterVolumeSpecName: "home") pod "14e3cda8-d9a1-496c-8e53-55be9e5130ee" (UID: "14e3cda8-d9a1-496c-8e53-55be9e5130ee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:03:41.174259 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:41.174086    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14e3cda8-d9a1-496c-8e53-55be9e5130ee-model-cache" (OuterVolumeSpecName: "model-cache") pod "14e3cda8-d9a1-496c-8e53-55be9e5130ee" (UID: "14e3cda8-d9a1-496c-8e53-55be9e5130ee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:03:41.176100 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:41.176054    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14e3cda8-d9a1-496c-8e53-55be9e5130ee-kube-api-access-vnr89" (OuterVolumeSpecName: "kube-api-access-vnr89") pod "14e3cda8-d9a1-496c-8e53-55be9e5130ee" (UID: "14e3cda8-d9a1-496c-8e53-55be9e5130ee"). InnerVolumeSpecName "kube-api-access-vnr89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:03:41.176394 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:41.176366    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14e3cda8-d9a1-496c-8e53-55be9e5130ee-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "14e3cda8-d9a1-496c-8e53-55be9e5130ee" (UID: "14e3cda8-d9a1-496c-8e53-55be9e5130ee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:03:41.176515 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:41.176380    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14e3cda8-d9a1-496c-8e53-55be9e5130ee-dshm" (OuterVolumeSpecName: "dshm") pod "14e3cda8-d9a1-496c-8e53-55be9e5130ee" (UID: "14e3cda8-d9a1-496c-8e53-55be9e5130ee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:03:41.179041 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:41.178975    2566 generic.go:356] "Generic (PLEG): container finished" podID="14e3cda8-d9a1-496c-8e53-55be9e5130ee" containerID="0d01af9f441c803ea9272961f5a19297fe693a38a3745b837903896aaf9cdb7b" exitCode=0
Sep 17 17:03:41.179041 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:41.179023    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj"
Sep 17 17:03:41.179041 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:41.179023    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj" event={"ID":"14e3cda8-d9a1-496c-8e53-55be9e5130ee","Type":"ContainerDied","Data":"0d01af9f441c803ea9272961f5a19297fe693a38a3745b837903896aaf9cdb7b"}
Sep 17 17:03:41.179245 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:41.179097    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj" event={"ID":"14e3cda8-d9a1-496c-8e53-55be9e5130ee","Type":"ContainerDied","Data":"043db90159e17cb039a90c764bd5f70d469f286b92522fb1148daaae9093d5fe"}
Sep 17 17:03:41.179245 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:41.179131    2566 scope.go:117] "RemoveContainer" containerID="0d01af9f441c803ea9272961f5a19297fe693a38a3745b837903896aaf9cdb7b"
Sep 17 17:03:41.195299 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:41.195277    2566 scope.go:117] "RemoveContainer" containerID="0d01af9f441c803ea9272961f5a19297fe693a38a3745b837903896aaf9cdb7b"
Sep 17 17:03:41.195631 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:03:41.195611    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d01af9f441c803ea9272961f5a19297fe693a38a3745b837903896aaf9cdb7b\": container with ID starting with 0d01af9f441c803ea9272961f5a19297fe693a38a3745b837903896aaf9cdb7b not found: ID does not exist" containerID="0d01af9f441c803ea9272961f5a19297fe693a38a3745b837903896aaf9cdb7b"
Sep 17 17:03:41.195699 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:41.195639    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d01af9f441c803ea9272961f5a19297fe693a38a3745b837903896aaf9cdb7b"} err="failed to get container status \"0d01af9f441c803ea9272961f5a19297fe693a38a3745b837903896aaf9cdb7b\": rpc error: code = NotFound desc = could not find container \"0d01af9f441c803ea9272961f5a19297fe693a38a3745b837903896aaf9cdb7b\": container with ID starting with 0d01af9f441c803ea9272961f5a19297fe693a38a3745b837903896aaf9cdb7b not found: ID does not exist"
Sep 17 17:03:41.208685 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:41.208665    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj"]
Sep 17 17:03:41.218994 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:41.218975    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444zv5zj"]
Sep 17 17:03:41.274974 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:41.274953    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14e3cda8-d9a1-496c-8e53-55be9e5130ee-dshm\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:03:41.274974 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:41.274974    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14e3cda8-d9a1-496c-8e53-55be9e5130ee-home\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:03:41.275133 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:41.274982    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14e3cda8-d9a1-496c-8e53-55be9e5130ee-model-cache\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:03:41.275133 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:41.274991    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnr89\" (UniqueName: \"kubernetes.io/projected/14e3cda8-d9a1-496c-8e53-55be9e5130ee-kube-api-access-vnr89\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:03:41.275133 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:41.275000    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14e3cda8-d9a1-496c-8e53-55be9e5130ee-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:03:41.275133 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:41.275010    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14e3cda8-d9a1-496c-8e53-55be9e5130ee-tmp-dir\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:03:42.346759 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:42.346718    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14e3cda8-d9a1-496c-8e53-55be9e5130ee" path="/var/lib/kubelet/pods/14e3cda8-d9a1-496c-8e53-55be9e5130ee/volumes"
Sep 17 17:03:43.240296 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:43.240259    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt" podUID="7fa3f78d-cd66-468c-b73c-6c83e138f099" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Sep 17 17:03:53.240263 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:03:53.240222    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt" podUID="7fa3f78d-cd66-468c-b73c-6c83e138f099" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Sep 17 17:04:03.240958 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:03.240908    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt" podUID="7fa3f78d-cd66-468c-b73c-6c83e138f099" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Sep 17 17:04:13.240459 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:13.240331    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt" podUID="7fa3f78d-cd66-468c-b73c-6c83e138f099" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Sep 17 17:04:23.240850 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:23.240800    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt" podUID="7fa3f78d-cd66-468c-b73c-6c83e138f099" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Sep 17 17:04:33.240977 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:33.240926    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt" podUID="7fa3f78d-cd66-468c-b73c-6c83e138f099" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Sep 17 17:04:36.933753 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:36.933720    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf"]
Sep 17 17:04:36.934221 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:36.934207    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14e3cda8-d9a1-496c-8e53-55be9e5130ee" containerName="main"
Sep 17 17:04:36.934293 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:36.934226    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="14e3cda8-d9a1-496c-8e53-55be9e5130ee" containerName="main"
Sep 17 17:04:36.934352 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:36.934336    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="14e3cda8-d9a1-496c-8e53-55be9e5130ee" containerName="main"
Sep 17 17:04:36.939154 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:36.939124    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf"
Sep 17 17:04:36.940792 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:36.940764    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-home\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf\" (UID: \"d10ab205-053b-4dc9-9b14-ff7fdb8ae617\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf"
Sep 17 17:04:36.940792 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:36.940796    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-dshm\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf\" (UID: \"d10ab205-053b-4dc9-9b14-ff7fdb8ae617\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf"
Sep 17 17:04:36.941017 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:36.940816    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf\" (UID: \"d10ab205-053b-4dc9-9b14-ff7fdb8ae617\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf"
Sep 17 17:04:36.941017 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:36.940886    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf\" (UID: \"d10ab205-053b-4dc9-9b14-ff7fdb8ae617\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf"
Sep 17 17:04:36.941017 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:36.940917    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf\" (UID: \"d10ab205-053b-4dc9-9b14-ff7fdb8ae617\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf"
Sep 17 17:04:36.941017 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:36.940948    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-model-cache\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf\" (UID: \"d10ab205-053b-4dc9-9b14-ff7fdb8ae617\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf"
Sep 17 17:04:36.941017 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:36.940976    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf\" (UID: \"d10ab205-053b-4dc9-9b14-ff7fdb8ae617\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf"
Sep 17 17:04:36.941298 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:36.941025    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rv4g\" (UniqueName: \"kubernetes.io/projected/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-kube-api-access-6rv4g\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf\" (UID: \"d10ab205-053b-4dc9-9b14-ff7fdb8ae617\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf"
Sep 17 17:04:36.941916 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:36.941892    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 17 17:04:36.943066 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:36.943042    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 17 17:04:36.943066 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:36.943058    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 17 17:04:36.943254 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:36.943108    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 17 17:04:36.943254 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:36.943122    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:04:36.943254 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:36.943193    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-24bd8\""
Sep 17 17:04:36.953293 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:36.953266    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf"]
Sep 17 17:04:37.042132 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:37.042096    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf\" (UID: \"d10ab205-053b-4dc9-9b14-ff7fdb8ae617\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf"
Sep 17 17:04:37.042132 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:37.042131    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rv4g\" (UniqueName: \"kubernetes.io/projected/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-kube-api-access-6rv4g\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf\" (UID: \"d10ab205-053b-4dc9-9b14-ff7fdb8ae617\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf"
Sep 17 17:04:37.042390 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:37.042153    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-home\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf\" (UID: \"d10ab205-053b-4dc9-9b14-ff7fdb8ae617\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf"
Sep 17 17:04:37.042390 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:37.042170    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-dshm\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf\" (UID: \"d10ab205-053b-4dc9-9b14-ff7fdb8ae617\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf"
Sep 17 17:04:37.042390 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:37.042197    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf\" (UID: \"d10ab205-053b-4dc9-9b14-ff7fdb8ae617\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf"
Sep 17 17:04:37.042390 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:37.042258    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf\" (UID: \"d10ab205-053b-4dc9-9b14-ff7fdb8ae617\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf"
Sep 17 17:04:37.042390 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:37.042289    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf\" (UID: \"d10ab205-053b-4dc9-9b14-ff7fdb8ae617\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf"
Sep 17 17:04:37.042390 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:37.042329    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-model-cache\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf\" (UID: \"d10ab205-053b-4dc9-9b14-ff7fdb8ae617\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf"
Sep 17 17:04:37.042711 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:37.042569    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf\" (UID: \"d10ab205-053b-4dc9-9b14-ff7fdb8ae617\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf"
Sep 17 17:04:37.042711 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:37.042665    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-home\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf\" (UID: \"d10ab205-053b-4dc9-9b14-ff7fdb8ae617\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf"
Sep 17 17:04:37.042711 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:37.042662    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf\" (UID: \"d10ab205-053b-4dc9-9b14-ff7fdb8ae617\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf"
Sep 17 17:04:37.042711 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:37.042707    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-model-cache\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf\" (UID: \"d10ab205-053b-4dc9-9b14-ff7fdb8ae617\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf"
Sep 17 17:04:37.043223 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:37.043198    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf\" (UID: \"d10ab205-053b-4dc9-9b14-ff7fdb8ae617\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf"
Sep 17 17:04:37.044905 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:37.044881    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-dshm\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf\" (UID: \"d10ab205-053b-4dc9-9b14-ff7fdb8ae617\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf"
Sep 17 17:04:37.045128 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:37.045102    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf\" (UID: \"d10ab205-053b-4dc9-9b14-ff7fdb8ae617\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf"
Sep 17 17:04:37.052494 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:37.052474    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rv4g\" (UniqueName: \"kubernetes.io/projected/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-kube-api-access-6rv4g\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf\" (UID: \"d10ab205-053b-4dc9-9b14-ff7fdb8ae617\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf"
Sep 17 17:04:37.251737 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:37.251643    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf"
Sep 17 17:04:37.533866 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:37.533825    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf"]
Sep 17 17:04:37.534561 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:04:37.534520    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd10ab205_053b_4dc9_9b14_ff7fdb8ae617.slice/crio-b093f751b8b8c948023982a65a1a32cc401b851a7429e6aa6c59b6cfca003534 WatchSource:0}: Error finding container b093f751b8b8c948023982a65a1a32cc401b851a7429e6aa6c59b6cfca003534: Status 404 returned error can't find the container with id b093f751b8b8c948023982a65a1a32cc401b851a7429e6aa6c59b6cfca003534
Sep 17 17:04:38.485967 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:38.485935    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf" event={"ID":"d10ab205-053b-4dc9-9b14-ff7fdb8ae617","Type":"ContainerStarted","Data":"b1ccf153a997e843e63ed9756337834509125bb7946d537d0a5d993d412e79ae"}
Sep 17 17:04:38.486351 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:38.485990    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf" event={"ID":"d10ab205-053b-4dc9-9b14-ff7fdb8ae617","Type":"ContainerStarted","Data":"b093f751b8b8c948023982a65a1a32cc401b851a7429e6aa6c59b6cfca003534"}
Sep 17 17:04:40.500082 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:40.500039    2566 generic.go:356] "Generic (PLEG): container finished" podID="d10ab205-053b-4dc9-9b14-ff7fdb8ae617" containerID="b1ccf153a997e843e63ed9756337834509125bb7946d537d0a5d993d412e79ae" exitCode=0
Sep 17 17:04:40.500717 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:40.500072    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf" event={"ID":"d10ab205-053b-4dc9-9b14-ff7fdb8ae617","Type":"ContainerDied","Data":"b1ccf153a997e843e63ed9756337834509125bb7946d537d0a5d993d412e79ae"}
Sep 17 17:04:41.508461 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:41.508405    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf" event={"ID":"d10ab205-053b-4dc9-9b14-ff7fdb8ae617","Type":"ContainerStarted","Data":"0bbb0e9bb5cce4f8bf3258a7e09479699af7d184ba79d412c291e371545b90e1"}
Sep 17 17:04:41.531684 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:41.531633    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf" podStartSLOduration=3.115972286 podStartE2EDuration="5.531617917s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.415645631s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:04:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:04:41.530284245 +0000 UTC m=+1367.424763384" watchObservedRunningTime="2026-09-17 17:04:41.531617917 +0000 UTC m=+1367.426097056"
Sep 17 17:04:42.252354 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:42.252308    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf"
Sep 17 17:04:43.249817 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:43.249789    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt"
Sep 17 17:04:43.257427 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:43.257401    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt"
Sep 17 17:04:47.252451 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:47.252395    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf"
Sep 17 17:04:47.265482 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:47.265425    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf"
Sep 17 17:04:47.551786 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:04:47.551760    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf"
Sep 17 17:05:27.453347 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:27.453297    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt"]
Sep 17 17:05:27.457074 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:27.457050    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-n5xx5"]
Sep 17 17:05:34.735926 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:34.735842    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-n5xx5" podUID="ab90fd45-9d64-4e50-9994-265804f78370" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:05:42.477589 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:42.477550    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt" podUID="7fa3f78d-cd66-468c-b73c-6c83e138f099" containerName="main" containerID="cri-o://e30ba3779579a5eae9b2fe2a27f08a30377961035c02ebb156077bcfc25cf285" gracePeriod=45
Sep 17 17:05:42.481397 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:42.481338    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-n5xx5" podUID="ab90fd45-9d64-4e50-9994-265804f78370" containerName="main" containerID="cri-o://397ce862ff5a451bae15249d8c7bb94f28f7c9ed335f7bde10d69514effde72e" gracePeriod=45
Sep 17 17:05:43.156391 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:43.156365    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt"
Sep 17 17:05:43.198632 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:43.198609    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fa3f78d-cd66-468c-b73c-6c83e138f099-tmp-dir\") pod \"7fa3f78d-cd66-468c-b73c-6c83e138f099\" (UID: \"7fa3f78d-cd66-468c-b73c-6c83e138f099\") "
Sep 17 17:05:43.198799 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:43.198638    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rns4l\" (UniqueName: \"kubernetes.io/projected/7fa3f78d-cd66-468c-b73c-6c83e138f099-kube-api-access-rns4l\") pod \"7fa3f78d-cd66-468c-b73c-6c83e138f099\" (UID: \"7fa3f78d-cd66-468c-b73c-6c83e138f099\") "
Sep 17 17:05:43.198799 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:43.198676    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fa3f78d-cd66-468c-b73c-6c83e138f099-model-cache\") pod \"7fa3f78d-cd66-468c-b73c-6c83e138f099\" (UID: \"7fa3f78d-cd66-468c-b73c-6c83e138f099\") "
Sep 17 17:05:43.198799 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:43.198704    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fa3f78d-cd66-468c-b73c-6c83e138f099-home\") pod \"7fa3f78d-cd66-468c-b73c-6c83e138f099\" (UID: \"7fa3f78d-cd66-468c-b73c-6c83e138f099\") "
Sep 17 17:05:43.198799 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:43.198728    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fa3f78d-cd66-468c-b73c-6c83e138f099-dshm\") pod \"7fa3f78d-cd66-468c-b73c-6c83e138f099\" (UID: \"7fa3f78d-cd66-468c-b73c-6c83e138f099\") "
Sep 17 17:05:43.198799 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:43.198745    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fa3f78d-cd66-468c-b73c-6c83e138f099-cabundle-cert\") pod \"7fa3f78d-cd66-468c-b73c-6c83e138f099\" (UID: \"7fa3f78d-cd66-468c-b73c-6c83e138f099\") "
Sep 17 17:05:43.198799 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:43.198789    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fa3f78d-cd66-468c-b73c-6c83e138f099-tls-certs\") pod \"7fa3f78d-cd66-468c-b73c-6c83e138f099\" (UID: \"7fa3f78d-cd66-468c-b73c-6c83e138f099\") "
Sep 17 17:05:43.199123 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:43.198829    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fa3f78d-cd66-468c-b73c-6c83e138f099-kserve-provision-location\") pod \"7fa3f78d-cd66-468c-b73c-6c83e138f099\" (UID: \"7fa3f78d-cd66-468c-b73c-6c83e138f099\") "
Sep 17 17:05:43.199123 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:43.199027    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fa3f78d-cd66-468c-b73c-6c83e138f099-model-cache" (OuterVolumeSpecName: "model-cache") pod "7fa3f78d-cd66-468c-b73c-6c83e138f099" (UID: "7fa3f78d-cd66-468c-b73c-6c83e138f099"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:05:43.199549 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:43.199499    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fa3f78d-cd66-468c-b73c-6c83e138f099-home" (OuterVolumeSpecName: "home") pod "7fa3f78d-cd66-468c-b73c-6c83e138f099" (UID: "7fa3f78d-cd66-468c-b73c-6c83e138f099"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:05:43.199734 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:43.199700    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fa3f78d-cd66-468c-b73c-6c83e138f099-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7fa3f78d-cd66-468c-b73c-6c83e138f099" (UID: "7fa3f78d-cd66-468c-b73c-6c83e138f099"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:05:43.201185 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:43.201158    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fa3f78d-cd66-468c-b73c-6c83e138f099-kube-api-access-rns4l" (OuterVolumeSpecName: "kube-api-access-rns4l") pod "7fa3f78d-cd66-468c-b73c-6c83e138f099" (UID: "7fa3f78d-cd66-468c-b73c-6c83e138f099"). InnerVolumeSpecName "kube-api-access-rns4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:05:43.201687 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:43.201655    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fa3f78d-cd66-468c-b73c-6c83e138f099-dshm" (OuterVolumeSpecName: "dshm") pod "7fa3f78d-cd66-468c-b73c-6c83e138f099" (UID: "7fa3f78d-cd66-468c-b73c-6c83e138f099"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:05:43.201895 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:43.201871    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fa3f78d-cd66-468c-b73c-6c83e138f099-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7fa3f78d-cd66-468c-b73c-6c83e138f099" (UID: "7fa3f78d-cd66-468c-b73c-6c83e138f099"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:05:43.212208 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:43.212181    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fa3f78d-cd66-468c-b73c-6c83e138f099-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7fa3f78d-cd66-468c-b73c-6c83e138f099" (UID: "7fa3f78d-cd66-468c-b73c-6c83e138f099"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:05:43.219484 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:43.219458    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fa3f78d-cd66-468c-b73c-6c83e138f099-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7fa3f78d-cd66-468c-b73c-6c83e138f099" (UID: "7fa3f78d-cd66-468c-b73c-6c83e138f099"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:05:43.299571 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:43.299539    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fa3f78d-cd66-468c-b73c-6c83e138f099-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:05:43.299571 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:43.299571    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fa3f78d-cd66-468c-b73c-6c83e138f099-kserve-provision-location\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:05:43.299750 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:43.299583    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fa3f78d-cd66-468c-b73c-6c83e138f099-tmp-dir\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:05:43.299750 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:43.299593    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rns4l\" (UniqueName: \"kubernetes.io/projected/7fa3f78d-cd66-468c-b73c-6c83e138f099-kube-api-access-rns4l\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:05:43.299750 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:43.299601    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fa3f78d-cd66-468c-b73c-6c83e138f099-model-cache\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:05:43.299750 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:43.299609    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fa3f78d-cd66-468c-b73c-6c83e138f099-home\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:05:43.299750 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:43.299615    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fa3f78d-cd66-468c-b73c-6c83e138f099-dshm\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:05:43.299750 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:43.299622    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fa3f78d-cd66-468c-b73c-6c83e138f099-cabundle-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:05:43.843025 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:43.842985    2566 generic.go:356] "Generic (PLEG): container finished" podID="7fa3f78d-cd66-468c-b73c-6c83e138f099" containerID="e30ba3779579a5eae9b2fe2a27f08a30377961035c02ebb156077bcfc25cf285" exitCode=0
Sep 17 17:05:43.843470 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:43.843036    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt"
Sep 17 17:05:43.843470 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:43.843034    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt" event={"ID":"7fa3f78d-cd66-468c-b73c-6c83e138f099","Type":"ContainerDied","Data":"e30ba3779579a5eae9b2fe2a27f08a30377961035c02ebb156077bcfc25cf285"}
Sep 17 17:05:43.843470 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:43.843168    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt" event={"ID":"7fa3f78d-cd66-468c-b73c-6c83e138f099","Type":"ContainerDied","Data":"aeeaa801cbec0485ec9c380a385e03511d7adcf02da8607497258549526c1594"}
Sep 17 17:05:43.843470 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:43.843199    2566 scope.go:117] "RemoveContainer" containerID="e30ba3779579a5eae9b2fe2a27f08a30377961035c02ebb156077bcfc25cf285"
Sep 17 17:05:43.852508 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:43.852492    2566 scope.go:117] "RemoveContainer" containerID="2e18456a607077c6e9c0291ba5a1a8e585bb13e8c8c054851e4c0ae85096ed35"
Sep 17 17:05:43.860161 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:43.860136    2566 scope.go:117] "RemoveContainer" containerID="e30ba3779579a5eae9b2fe2a27f08a30377961035c02ebb156077bcfc25cf285"
Sep 17 17:05:43.860454 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:05:43.860420    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e30ba3779579a5eae9b2fe2a27f08a30377961035c02ebb156077bcfc25cf285\": container with ID starting with e30ba3779579a5eae9b2fe2a27f08a30377961035c02ebb156077bcfc25cf285 not found: ID does not exist" containerID="e30ba3779579a5eae9b2fe2a27f08a30377961035c02ebb156077bcfc25cf285"
Sep 17 17:05:43.860503 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:43.860465    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e30ba3779579a5eae9b2fe2a27f08a30377961035c02ebb156077bcfc25cf285"} err="failed to get container status \"e30ba3779579a5eae9b2fe2a27f08a30377961035c02ebb156077bcfc25cf285\": rpc error: code = NotFound desc = could not find container \"e30ba3779579a5eae9b2fe2a27f08a30377961035c02ebb156077bcfc25cf285\": container with ID starting with e30ba3779579a5eae9b2fe2a27f08a30377961035c02ebb156077bcfc25cf285 not found: ID does not exist"
Sep 17 17:05:43.860503 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:43.860485    2566 scope.go:117] "RemoveContainer" containerID="2e18456a607077c6e9c0291ba5a1a8e585bb13e8c8c054851e4c0ae85096ed35"
Sep 17 17:05:43.860730 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:05:43.860710    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e18456a607077c6e9c0291ba5a1a8e585bb13e8c8c054851e4c0ae85096ed35\": container with ID starting with 2e18456a607077c6e9c0291ba5a1a8e585bb13e8c8c054851e4c0ae85096ed35 not found: ID does not exist" containerID="2e18456a607077c6e9c0291ba5a1a8e585bb13e8c8c054851e4c0ae85096ed35"
Sep 17 17:05:43.860773 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:43.860737    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e18456a607077c6e9c0291ba5a1a8e585bb13e8c8c054851e4c0ae85096ed35"} err="failed to get container status \"2e18456a607077c6e9c0291ba5a1a8e585bb13e8c8c054851e4c0ae85096ed35\": rpc error: code = NotFound desc = could not find container \"2e18456a607077c6e9c0291ba5a1a8e585bb13e8c8c054851e4c0ae85096ed35\": container with ID starting with 2e18456a607077c6e9c0291ba5a1a8e585bb13e8c8c054851e4c0ae85096ed35 not found: ID does not exist"
Sep 17 17:05:43.871733 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:43.871705    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt"]
Sep 17 17:05:43.880266 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:43.880243    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-8cb6d4fb5-z2rrt"]
Sep 17 17:05:44.347341 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:44.347301    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fa3f78d-cd66-468c-b73c-6c83e138f099" path="/var/lib/kubelet/pods/7fa3f78d-cd66-468c-b73c-6c83e138f099/volumes"
Sep 17 17:05:44.736760 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:44.736674    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-n5xx5" podUID="ab90fd45-9d64-4e50-9994-265804f78370" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:05:54.736111 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:54.736071    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-n5xx5" podUID="ab90fd45-9d64-4e50-9994-265804f78370" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:05:54.736537 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:05:54.736207    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-n5xx5"
Sep 17 17:06:04.413214 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:04.413181    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf"]
Sep 17 17:06:04.424378 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:06:04.424332    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:06:04.424378 ip-10-0-141-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:06:04.424378 ip-10-0-141-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:06:04.424378 ip-10-0-141-66 kubenswrapper[2566]:  > containerID="0bbb0e9bb5cce4f8bf3258a7e09479699af7d184ba79d412c291e371545b90e1" cmd=["/bin/sleep","15"]
Sep 17 17:06:04.424736 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:04.424405    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf" message="" err=<
Sep 17 17:06:04.424736 ip-10-0-141-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:06:04.424736 ip-10-0-141-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:06:04.424736 ip-10-0-141-66 kubenswrapper[2566]:  >
Sep 17 17:06:04.424736 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:06:04.424425    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:06:04.424736 ip-10-0-141-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:06:04.424736 ip-10-0-141-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:06:04.424736 ip-10-0-141-66 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf" podUID="d10ab205-053b-4dc9-9b14-ff7fdb8ae617" containerName="main" containerID="cri-o://0bbb0e9bb5cce4f8bf3258a7e09479699af7d184ba79d412c291e371545b90e1"
Sep 17 17:06:04.424736 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:04.424515    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf" podUID="d10ab205-053b-4dc9-9b14-ff7fdb8ae617" containerName="main" containerID="cri-o://0bbb0e9bb5cce4f8bf3258a7e09479699af7d184ba79d412c291e371545b90e1" gracePeriod=60
Sep 17 17:06:04.541308 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:04.541268    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf" podUID="d10ab205-053b-4dc9-9b14-ff7fdb8ae617" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Sep 17 17:06:04.736806 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:04.736767    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-n5xx5" podUID="ab90fd45-9d64-4e50-9994-265804f78370" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:06:04.778522 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:04.778492    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf"
Sep 17 17:06:04.880065 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:04.880028    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rv4g\" (UniqueName: \"kubernetes.io/projected/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-kube-api-access-6rv4g\") pod \"d10ab205-053b-4dc9-9b14-ff7fdb8ae617\" (UID: \"d10ab205-053b-4dc9-9b14-ff7fdb8ae617\") "
Sep 17 17:06:04.880256 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:04.880094    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-home\") pod \"d10ab205-053b-4dc9-9b14-ff7fdb8ae617\" (UID: \"d10ab205-053b-4dc9-9b14-ff7fdb8ae617\") "
Sep 17 17:06:04.880256 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:04.880146    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-dshm\") pod \"d10ab205-053b-4dc9-9b14-ff7fdb8ae617\" (UID: \"d10ab205-053b-4dc9-9b14-ff7fdb8ae617\") "
Sep 17 17:06:04.880256 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:04.880181    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-kserve-provision-location\") pod \"d10ab205-053b-4dc9-9b14-ff7fdb8ae617\" (UID: \"d10ab205-053b-4dc9-9b14-ff7fdb8ae617\") "
Sep 17 17:06:04.880256 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:04.880204    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-tls-certs\") pod \"d10ab205-053b-4dc9-9b14-ff7fdb8ae617\" (UID: \"d10ab205-053b-4dc9-9b14-ff7fdb8ae617\") "
Sep 17 17:06:04.880256 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:04.880223    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-cabundle-cert\") pod \"d10ab205-053b-4dc9-9b14-ff7fdb8ae617\" (UID: \"d10ab205-053b-4dc9-9b14-ff7fdb8ae617\") "
Sep 17 17:06:04.880256 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:04.880250    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-model-cache\") pod \"d10ab205-053b-4dc9-9b14-ff7fdb8ae617\" (UID: \"d10ab205-053b-4dc9-9b14-ff7fdb8ae617\") "
Sep 17 17:06:04.880623 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:04.880284    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-tmp-dir\") pod \"d10ab205-053b-4dc9-9b14-ff7fdb8ae617\" (UID: \"d10ab205-053b-4dc9-9b14-ff7fdb8ae617\") "
Sep 17 17:06:04.880623 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:04.880417    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-home" (OuterVolumeSpecName: "home") pod "d10ab205-053b-4dc9-9b14-ff7fdb8ae617" (UID: "d10ab205-053b-4dc9-9b14-ff7fdb8ae617"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:06:04.880753 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:04.880718    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d10ab205-053b-4dc9-9b14-ff7fdb8ae617" (UID: "d10ab205-053b-4dc9-9b14-ff7fdb8ae617"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:06:04.881039 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:04.880764    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d10ab205-053b-4dc9-9b14-ff7fdb8ae617" (UID: "d10ab205-053b-4dc9-9b14-ff7fdb8ae617"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:06:04.881039 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:04.880798    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-model-cache" (OuterVolumeSpecName: "model-cache") pod "d10ab205-053b-4dc9-9b14-ff7fdb8ae617" (UID: "d10ab205-053b-4dc9-9b14-ff7fdb8ae617"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:06:04.882672 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:04.882651    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-kube-api-access-6rv4g" (OuterVolumeSpecName: "kube-api-access-6rv4g") pod "d10ab205-053b-4dc9-9b14-ff7fdb8ae617" (UID: "d10ab205-053b-4dc9-9b14-ff7fdb8ae617"). InnerVolumeSpecName "kube-api-access-6rv4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:06:04.883043 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:04.883020    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d10ab205-053b-4dc9-9b14-ff7fdb8ae617" (UID: "d10ab205-053b-4dc9-9b14-ff7fdb8ae617"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:06:04.883111 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:04.883054    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-dshm" (OuterVolumeSpecName: "dshm") pod "d10ab205-053b-4dc9-9b14-ff7fdb8ae617" (UID: "d10ab205-053b-4dc9-9b14-ff7fdb8ae617"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:06:04.902137 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:04.902099    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d10ab205-053b-4dc9-9b14-ff7fdb8ae617" (UID: "d10ab205-053b-4dc9-9b14-ff7fdb8ae617"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:06:04.957289 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:04.957189    2566 generic.go:356] "Generic (PLEG): container finished" podID="d10ab205-053b-4dc9-9b14-ff7fdb8ae617" containerID="0bbb0e9bb5cce4f8bf3258a7e09479699af7d184ba79d412c291e371545b90e1" exitCode=0
Sep 17 17:06:04.957289 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:04.957233    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf" event={"ID":"d10ab205-053b-4dc9-9b14-ff7fdb8ae617","Type":"ContainerDied","Data":"0bbb0e9bb5cce4f8bf3258a7e09479699af7d184ba79d412c291e371545b90e1"}
Sep 17 17:06:04.957289 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:04.957251    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf"
Sep 17 17:06:04.957615 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:04.957293    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf" event={"ID":"d10ab205-053b-4dc9-9b14-ff7fdb8ae617","Type":"ContainerDied","Data":"b093f751b8b8c948023982a65a1a32cc401b851a7429e6aa6c59b6cfca003534"}
Sep 17 17:06:04.957615 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:04.957321    2566 scope.go:117] "RemoveContainer" containerID="0bbb0e9bb5cce4f8bf3258a7e09479699af7d184ba79d412c291e371545b90e1"
Sep 17 17:06:04.967137 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:04.967117    2566 scope.go:117] "RemoveContainer" containerID="b1ccf153a997e843e63ed9756337834509125bb7946d537d0a5d993d412e79ae"
Sep 17 17:06:04.975813 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:04.975793    2566 scope.go:117] "RemoveContainer" containerID="0bbb0e9bb5cce4f8bf3258a7e09479699af7d184ba79d412c291e371545b90e1"
Sep 17 17:06:04.976123 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:06:04.976104    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bbb0e9bb5cce4f8bf3258a7e09479699af7d184ba79d412c291e371545b90e1\": container with ID starting with 0bbb0e9bb5cce4f8bf3258a7e09479699af7d184ba79d412c291e371545b90e1 not found: ID does not exist" containerID="0bbb0e9bb5cce4f8bf3258a7e09479699af7d184ba79d412c291e371545b90e1"
Sep 17 17:06:04.976190 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:04.976132    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bbb0e9bb5cce4f8bf3258a7e09479699af7d184ba79d412c291e371545b90e1"} err="failed to get container status \"0bbb0e9bb5cce4f8bf3258a7e09479699af7d184ba79d412c291e371545b90e1\": rpc error: code = NotFound desc = could not find container \"0bbb0e9bb5cce4f8bf3258a7e09479699af7d184ba79d412c291e371545b90e1\": container with ID starting with 0bbb0e9bb5cce4f8bf3258a7e09479699af7d184ba79d412c291e371545b90e1 not found: ID does not exist"
Sep 17 17:06:04.976190 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:04.976157    2566 scope.go:117] "RemoveContainer" containerID="b1ccf153a997e843e63ed9756337834509125bb7946d537d0a5d993d412e79ae"
Sep 17 17:06:04.976423 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:06:04.976404    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1ccf153a997e843e63ed9756337834509125bb7946d537d0a5d993d412e79ae\": container with ID starting with b1ccf153a997e843e63ed9756337834509125bb7946d537d0a5d993d412e79ae not found: ID does not exist" containerID="b1ccf153a997e843e63ed9756337834509125bb7946d537d0a5d993d412e79ae"
Sep 17 17:06:04.976509 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:04.976456    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1ccf153a997e843e63ed9756337834509125bb7946d537d0a5d993d412e79ae"} err="failed to get container status \"b1ccf153a997e843e63ed9756337834509125bb7946d537d0a5d993d412e79ae\": rpc error: code = NotFound desc = could not find container \"b1ccf153a997e843e63ed9756337834509125bb7946d537d0a5d993d412e79ae\": container with ID starting with b1ccf153a997e843e63ed9756337834509125bb7946d537d0a5d993d412e79ae not found: ID does not exist"
Sep 17 17:06:04.981851 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:04.981828    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-tmp-dir\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:06:04.981915 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:04.981856    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rv4g\" (UniqueName: \"kubernetes.io/projected/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-kube-api-access-6rv4g\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:06:04.981915 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:04.981867    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-home\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:06:04.981915 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:04.981876    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-dshm\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:06:04.981915 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:04.981883    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-kserve-provision-location\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:06:04.981915 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:04.981891    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:06:04.981915 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:04.981898    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-cabundle-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:06:04.981915 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:04.981906    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d10ab205-053b-4dc9-9b14-ff7fdb8ae617-model-cache\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:06:04.990839 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:04.990809    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf"]
Sep 17 17:06:04.995939 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:04.995906    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-xhnvf"]
Sep 17 17:06:06.346662 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:06.346628    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d10ab205-053b-4dc9-9b14-ff7fdb8ae617" path="/var/lib/kubelet/pods/d10ab205-053b-4dc9-9b14-ff7fdb8ae617/volumes"
Sep 17 17:06:12.732870 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:12.732845    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-n5xx5"
Sep 17 17:06:12.849911 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:12.849879    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6lmv2\" (UniqueName: \"kubernetes.io/projected/ab90fd45-9d64-4e50-9994-265804f78370-kube-api-access-6lmv2\") pod \"ab90fd45-9d64-4e50-9994-265804f78370\" (UID: \"ab90fd45-9d64-4e50-9994-265804f78370\") "
Sep 17 17:06:12.850120 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:12.849934    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab90fd45-9d64-4e50-9994-265804f78370-tls-certs\") pod \"ab90fd45-9d64-4e50-9994-265804f78370\" (UID: \"ab90fd45-9d64-4e50-9994-265804f78370\") "
Sep 17 17:06:12.852252 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:12.852215    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab90fd45-9d64-4e50-9994-265804f78370-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ab90fd45-9d64-4e50-9994-265804f78370" (UID: "ab90fd45-9d64-4e50-9994-265804f78370"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:06:12.852362 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:12.852277    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab90fd45-9d64-4e50-9994-265804f78370-kube-api-access-6lmv2" (OuterVolumeSpecName: "kube-api-access-6lmv2") pod "ab90fd45-9d64-4e50-9994-265804f78370" (UID: "ab90fd45-9d64-4e50-9994-265804f78370"). InnerVolumeSpecName "kube-api-access-6lmv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:06:12.950803 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:12.950762    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab90fd45-9d64-4e50-9994-265804f78370-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:06:12.950803 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:12.950797    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lmv2\" (UniqueName: \"kubernetes.io/projected/ab90fd45-9d64-4e50-9994-265804f78370-kube-api-access-6lmv2\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:06:13.000871 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:13.000835    2566 generic.go:356] "Generic (PLEG): container finished" podID="ab90fd45-9d64-4e50-9994-265804f78370" containerID="397ce862ff5a451bae15249d8c7bb94f28f7c9ed335f7bde10d69514effde72e" exitCode=0
Sep 17 17:06:13.001070 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:13.000870    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-n5xx5" event={"ID":"ab90fd45-9d64-4e50-9994-265804f78370","Type":"ContainerDied","Data":"397ce862ff5a451bae15249d8c7bb94f28f7c9ed335f7bde10d69514effde72e"}
Sep 17 17:06:13.001070 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:13.000893    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-n5xx5"
Sep 17 17:06:13.001070 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:13.000939    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-n5xx5" event={"ID":"ab90fd45-9d64-4e50-9994-265804f78370","Type":"ContainerDied","Data":"ea57b785e4045578d2bd87d1891d77f70e13760324d2b85215b5f418bbba11c6"}
Sep 17 17:06:13.001070 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:13.000967    2566 scope.go:117] "RemoveContainer" containerID="397ce862ff5a451bae15249d8c7bb94f28f7c9ed335f7bde10d69514effde72e"
Sep 17 17:06:13.010593 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:13.010574    2566 scope.go:117] "RemoveContainer" containerID="397ce862ff5a451bae15249d8c7bb94f28f7c9ed335f7bde10d69514effde72e"
Sep 17 17:06:13.010893 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:06:13.010874    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"397ce862ff5a451bae15249d8c7bb94f28f7c9ed335f7bde10d69514effde72e\": container with ID starting with 397ce862ff5a451bae15249d8c7bb94f28f7c9ed335f7bde10d69514effde72e not found: ID does not exist" containerID="397ce862ff5a451bae15249d8c7bb94f28f7c9ed335f7bde10d69514effde72e"
Sep 17 17:06:13.010953 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:13.010905    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"397ce862ff5a451bae15249d8c7bb94f28f7c9ed335f7bde10d69514effde72e"} err="failed to get container status \"397ce862ff5a451bae15249d8c7bb94f28f7c9ed335f7bde10d69514effde72e\": rpc error: code = NotFound desc = could not find container \"397ce862ff5a451bae15249d8c7bb94f28f7c9ed335f7bde10d69514effde72e\": container with ID starting with 397ce862ff5a451bae15249d8c7bb94f28f7c9ed335f7bde10d69514effde72e not found: ID does not exist"
Sep 17 17:06:13.027304 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:13.027276    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-n5xx5"]
Sep 17 17:06:13.033636 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:13.033612    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-n5xx5"]
Sep 17 17:06:14.350835 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:14.350802    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab90fd45-9d64-4e50-9994-265804f78370" path="/var/lib/kubelet/pods/ab90fd45-9d64-4e50-9994-265804f78370/volumes"
Sep 17 17:06:24.367293 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.367259    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv"]
Sep 17 17:06:24.367770 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.367725    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fa3f78d-cd66-468c-b73c-6c83e138f099" containerName="storage-initializer"
Sep 17 17:06:24.367770 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.367745    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fa3f78d-cd66-468c-b73c-6c83e138f099" containerName="storage-initializer"
Sep 17 17:06:24.367770 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.367760    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d10ab205-053b-4dc9-9b14-ff7fdb8ae617" containerName="main"
Sep 17 17:06:24.367770 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.367769    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d10ab205-053b-4dc9-9b14-ff7fdb8ae617" containerName="main"
Sep 17 17:06:24.367986 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.367796    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d10ab205-053b-4dc9-9b14-ff7fdb8ae617" containerName="storage-initializer"
Sep 17 17:06:24.367986 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.367806    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d10ab205-053b-4dc9-9b14-ff7fdb8ae617" containerName="storage-initializer"
Sep 17 17:06:24.367986 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.367817    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab90fd45-9d64-4e50-9994-265804f78370" containerName="main"
Sep 17 17:06:24.367986 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.367825    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab90fd45-9d64-4e50-9994-265804f78370" containerName="main"
Sep 17 17:06:24.367986 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.367833    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fa3f78d-cd66-468c-b73c-6c83e138f099" containerName="main"
Sep 17 17:06:24.367986 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.367841    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fa3f78d-cd66-468c-b73c-6c83e138f099" containerName="main"
Sep 17 17:06:24.367986 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.367926    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d10ab205-053b-4dc9-9b14-ff7fdb8ae617" containerName="main"
Sep 17 17:06:24.367986 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.367942    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fa3f78d-cd66-468c-b73c-6c83e138f099" containerName="main"
Sep 17 17:06:24.367986 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.367954    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab90fd45-9d64-4e50-9994-265804f78370" containerName="main"
Sep 17 17:06:24.370727 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.370705    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv"
Sep 17 17:06:24.373206 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.373185    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:06:24.373353 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.373259    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 17 17:06:24.373353 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.373279    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 17 17:06:24.373598 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.373579    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Sep 17 17:06:24.373868 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.373850    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 17 17:06:24.373936 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.373896    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-c522d\""
Sep 17 17:06:24.385646 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.385621    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv"]
Sep 17 17:06:24.443625 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.443588    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-cabundle-cert\") pod \"auth-invalid-token-test-kserve-9fdbb9d5-rdtzv\" (UID: \"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv"
Sep 17 17:06:24.443819 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.443640    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-home\") pod \"auth-invalid-token-test-kserve-9fdbb9d5-rdtzv\" (UID: \"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv"
Sep 17 17:06:24.443819 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.443667    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-9fdbb9d5-rdtzv\" (UID: \"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv"
Sep 17 17:06:24.443819 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.443682    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4szsp\" (UniqueName: \"kubernetes.io/projected/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-kube-api-access-4szsp\") pod \"auth-invalid-token-test-kserve-9fdbb9d5-rdtzv\" (UID: \"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv"
Sep 17 17:06:24.443819 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.443700    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-tls-certs\") pod \"auth-invalid-token-test-kserve-9fdbb9d5-rdtzv\" (UID: \"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv"
Sep 17 17:06:24.443819 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.443758    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-dshm\") pod \"auth-invalid-token-test-kserve-9fdbb9d5-rdtzv\" (UID: \"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv"
Sep 17 17:06:24.443819 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.443792    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-tmp-dir\") pod \"auth-invalid-token-test-kserve-9fdbb9d5-rdtzv\" (UID: \"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv"
Sep 17 17:06:24.444019 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.443820    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-model-cache\") pod \"auth-invalid-token-test-kserve-9fdbb9d5-rdtzv\" (UID: \"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv"
Sep 17 17:06:24.544954 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.544914    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-dshm\") pod \"auth-invalid-token-test-kserve-9fdbb9d5-rdtzv\" (UID: \"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv"
Sep 17 17:06:24.544954 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.544952    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-tmp-dir\") pod \"auth-invalid-token-test-kserve-9fdbb9d5-rdtzv\" (UID: \"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv"
Sep 17 17:06:24.545186 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.544972    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-model-cache\") pod \"auth-invalid-token-test-kserve-9fdbb9d5-rdtzv\" (UID: \"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv"
Sep 17 17:06:24.545186 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.545009    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-cabundle-cert\") pod \"auth-invalid-token-test-kserve-9fdbb9d5-rdtzv\" (UID: \"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv"
Sep 17 17:06:24.545186 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.545116    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-home\") pod \"auth-invalid-token-test-kserve-9fdbb9d5-rdtzv\" (UID: \"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv"
Sep 17 17:06:24.545186 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.545156    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-9fdbb9d5-rdtzv\" (UID: \"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv"
Sep 17 17:06:24.545186 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.545182    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4szsp\" (UniqueName: \"kubernetes.io/projected/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-kube-api-access-4szsp\") pod \"auth-invalid-token-test-kserve-9fdbb9d5-rdtzv\" (UID: \"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv"
Sep 17 17:06:24.545430 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.545209    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-tls-certs\") pod \"auth-invalid-token-test-kserve-9fdbb9d5-rdtzv\" (UID: \"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv"
Sep 17 17:06:24.545430 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.545398    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-model-cache\") pod \"auth-invalid-token-test-kserve-9fdbb9d5-rdtzv\" (UID: \"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv"
Sep 17 17:06:24.545595 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.545528    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-tmp-dir\") pod \"auth-invalid-token-test-kserve-9fdbb9d5-rdtzv\" (UID: \"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv"
Sep 17 17:06:24.545665 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.545605    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-9fdbb9d5-rdtzv\" (UID: \"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv"
Sep 17 17:06:24.545800 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.545779    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-cabundle-cert\") pod \"auth-invalid-token-test-kserve-9fdbb9d5-rdtzv\" (UID: \"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv"
Sep 17 17:06:24.545861 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.545808    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-home\") pod \"auth-invalid-token-test-kserve-9fdbb9d5-rdtzv\" (UID: \"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv"
Sep 17 17:06:24.547382 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.547365    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-dshm\") pod \"auth-invalid-token-test-kserve-9fdbb9d5-rdtzv\" (UID: \"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv"
Sep 17 17:06:24.547733 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.547714    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-tls-certs\") pod \"auth-invalid-token-test-kserve-9fdbb9d5-rdtzv\" (UID: \"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv"
Sep 17 17:06:24.555679 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.555654    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4szsp\" (UniqueName: \"kubernetes.io/projected/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-kube-api-access-4szsp\") pod \"auth-invalid-token-test-kserve-9fdbb9d5-rdtzv\" (UID: \"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv"
Sep 17 17:06:24.681333 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.681210    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv"
Sep 17 17:06:24.986463 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:06:24.986411    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b7e93ba_3f6e_4fe2_85d4_ce5aaed3d19e.slice/crio-2fb80edfcd09b9d265564f21bedf34c8197f6d8532b6b82bcf8ed6214c9f4099 WatchSource:0}: Error finding container 2fb80edfcd09b9d265564f21bedf34c8197f6d8532b6b82bcf8ed6214c9f4099: Status 404 returned error can't find the container with id 2fb80edfcd09b9d265564f21bedf34c8197f6d8532b6b82bcf8ed6214c9f4099
Sep 17 17:06:24.987705 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:24.987682    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv"]
Sep 17 17:06:25.074306 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:25.074268    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv" event={"ID":"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e","Type":"ContainerStarted","Data":"9cfa1d1c5c3289f3ce43446b87953664e88f60d02b2a66d5b5de5b8eae6948c4"}
Sep 17 17:06:25.074547 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:25.074332    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv" event={"ID":"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e","Type":"ContainerStarted","Data":"2fb80edfcd09b9d265564f21bedf34c8197f6d8532b6b82bcf8ed6214c9f4099"}
Sep 17 17:06:28.094545 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:28.094507    2566 generic.go:356] "Generic (PLEG): container finished" podID="8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e" containerID="9cfa1d1c5c3289f3ce43446b87953664e88f60d02b2a66d5b5de5b8eae6948c4" exitCode=0
Sep 17 17:06:28.094545 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:28.094551    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv" event={"ID":"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e","Type":"ContainerDied","Data":"9cfa1d1c5c3289f3ce43446b87953664e88f60d02b2a66d5b5de5b8eae6948c4"}
Sep 17 17:06:29.103017 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:29.102978    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv" event={"ID":"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e","Type":"ContainerStarted","Data":"1fcb9893b0d1e34c51ce6fe61acc4849160ddf325ec4d68c90014e0b7c88af95"}
Sep 17 17:06:29.158891 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:29.158830    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv" podStartSLOduration=2.832555829 podStartE2EDuration="5.158813212s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.326257383s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:06:29.156834039 +0000 UTC m=+1475.051313206" watchObservedRunningTime="2026-09-17 17:06:29.158813212 +0000 UTC m=+1475.053292350"
Sep 17 17:06:29.681884 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:29.681845    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv"
Sep 17 17:06:34.682415 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:34.682369    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv"
Sep 17 17:06:34.684019 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:34.683990    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv" podUID="8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 17:06:44.682084 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:44.682026    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv" podUID="8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 17:06:54.681719 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:54.681678    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv" podUID="8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 17:06:56.380118 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:56.380087    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gblhr_91477834-6648-4e75-8530-e01bdee5e57e/ovn-acl-logging/0.log"
Sep 17 17:06:56.383506 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:06:56.383484    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gblhr_91477834-6648-4e75-8530-e01bdee5e57e/ovn-acl-logging/0.log"
Sep 17 17:07:04.682676 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:04.682581    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv" podUID="8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 17:07:07.297981 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.297945    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h"]
Sep 17 17:07:07.303035 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.303002    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h"
Sep 17 17:07:07.306968 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.306940    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 17 17:07:07.307138 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.307056    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 17 17:07:07.307403 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.307384    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 17 17:07:07.307619 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.307584    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:07:07.307685 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.307676    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-mmz5d\""
Sep 17 17:07:07.307759 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.307740    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 17 17:07:07.314636 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.314610    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h"]
Sep 17 17:07:07.400474 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.400408    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-home\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-pjx2h\" (UID: \"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h"
Sep 17 17:07:07.400665 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.400490    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c7br\" (UniqueName: \"kubernetes.io/projected/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-kube-api-access-8c7br\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-pjx2h\" (UID: \"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h"
Sep 17 17:07:07.400665 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.400517    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-pjx2h\" (UID: \"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h"
Sep 17 17:07:07.400665 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.400535    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-pjx2h\" (UID: \"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h"
Sep 17 17:07:07.400665 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.400603    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-model-cache\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-pjx2h\" (UID: \"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h"
Sep 17 17:07:07.400665 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.400632    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-pjx2h\" (UID: \"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h"
Sep 17 17:07:07.400665 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.400666    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-dshm\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-pjx2h\" (UID: \"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h"
Sep 17 17:07:07.400891 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.400691    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-tls-certs\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-pjx2h\" (UID: \"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h"
Sep 17 17:07:07.501618 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.501574    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-pjx2h\" (UID: \"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h"
Sep 17 17:07:07.501815 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.501627    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-pjx2h\" (UID: \"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h"
Sep 17 17:07:07.501815 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.501660    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-model-cache\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-pjx2h\" (UID: \"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h"
Sep 17 17:07:07.501815 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.501789    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-pjx2h\" (UID: \"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h"
Sep 17 17:07:07.501977 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.501858    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-dshm\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-pjx2h\" (UID: \"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h"
Sep 17 17:07:07.501977 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.501903    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-tls-certs\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-pjx2h\" (UID: \"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h"
Sep 17 17:07:07.501977 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.501954    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-home\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-pjx2h\" (UID: \"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h"
Sep 17 17:07:07.502139 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.502010    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c7br\" (UniqueName: \"kubernetes.io/projected/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-kube-api-access-8c7br\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-pjx2h\" (UID: \"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h"
Sep 17 17:07:07.502139 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.502046    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-pjx2h\" (UID: \"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h"
Sep 17 17:07:07.502139 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.502104    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-model-cache\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-pjx2h\" (UID: \"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h"
Sep 17 17:07:07.502364 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.502331    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-home\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-pjx2h\" (UID: \"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h"
Sep 17 17:07:07.502565 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.502545    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-pjx2h\" (UID: \"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h"
Sep 17 17:07:07.502635 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.502598    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-pjx2h\" (UID: \"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h"
Sep 17 17:07:07.504416 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.504394    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-dshm\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-pjx2h\" (UID: \"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h"
Sep 17 17:07:07.504659 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.504643    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-tls-certs\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-pjx2h\" (UID: \"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h"
Sep 17 17:07:07.512372 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.512344    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c7br\" (UniqueName: \"kubernetes.io/projected/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-kube-api-access-8c7br\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-pjx2h\" (UID: \"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h"
Sep 17 17:07:07.615220 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.615118    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h"
Sep 17 17:07:07.624243 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.624199    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785tbw2p"]
Sep 17 17:07:07.628306 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.628279    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785tbw2p"
Sep 17 17:07:07.630915 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.630889    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-epp-sa-dockercfg-hjjjh\""
Sep 17 17:07:07.641699 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.641662    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785tbw2p"]
Sep 17 17:07:07.704235 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.704014    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-776lh\" (UniqueName: \"kubernetes.io/projected/217e7d64-317c-4430-b4e8-6d3b220b73be-kube-api-access-776lh\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785tbw2p\" (UID: \"217e7d64-317c-4430-b4e8-6d3b220b73be\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785tbw2p"
Sep 17 17:07:07.704235 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.704098    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/217e7d64-317c-4430-b4e8-6d3b220b73be-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785tbw2p\" (UID: \"217e7d64-317c-4430-b4e8-6d3b220b73be\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785tbw2p"
Sep 17 17:07:07.804922 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.804888    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/217e7d64-317c-4430-b4e8-6d3b220b73be-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785tbw2p\" (UID: \"217e7d64-317c-4430-b4e8-6d3b220b73be\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785tbw2p"
Sep 17 17:07:07.805093 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.804945    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-776lh\" (UniqueName: \"kubernetes.io/projected/217e7d64-317c-4430-b4e8-6d3b220b73be-kube-api-access-776lh\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785tbw2p\" (UID: \"217e7d64-317c-4430-b4e8-6d3b220b73be\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785tbw2p"
Sep 17 17:07:07.807696 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.807668    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/217e7d64-317c-4430-b4e8-6d3b220b73be-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785tbw2p\" (UID: \"217e7d64-317c-4430-b4e8-6d3b220b73be\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785tbw2p"
Sep 17 17:07:07.817984 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.817951    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-776lh\" (UniqueName: \"kubernetes.io/projected/217e7d64-317c-4430-b4e8-6d3b220b73be-kube-api-access-776lh\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785tbw2p\" (UID: \"217e7d64-317c-4430-b4e8-6d3b220b73be\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785tbw2p"
Sep 17 17:07:07.908430 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.908399    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h"]
Sep 17 17:07:07.910862 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:07:07.910834    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea69b3f_7747_4a6e_bbae_48bb2ba1f0ef.slice/crio-01d74f4932aba9c644e56cb343d86e2087b1147abeff0cf631c95b71daf64aa7 WatchSource:0}: Error finding container 01d74f4932aba9c644e56cb343d86e2087b1147abeff0cf631c95b71daf64aa7: Status 404 returned error can't find the container with id 01d74f4932aba9c644e56cb343d86e2087b1147abeff0cf631c95b71daf64aa7
Sep 17 17:07:07.959640 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:07.959611    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785tbw2p"
Sep 17 17:07:08.265951 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:08.265923    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785tbw2p"]
Sep 17 17:07:08.269359 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:07:08.269331    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod217e7d64_317c_4430_b4e8_6d3b220b73be.slice/crio-47f7cd6c2eb3b0d0a03f11bd15578b01061c4a14f25b0e81a788aaa1fa8c0eeb WatchSource:0}: Error finding container 47f7cd6c2eb3b0d0a03f11bd15578b01061c4a14f25b0e81a788aaa1fa8c0eeb: Status 404 returned error can't find the container with id 47f7cd6c2eb3b0d0a03f11bd15578b01061c4a14f25b0e81a788aaa1fa8c0eeb
Sep 17 17:07:08.317823 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:08.317784    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h" event={"ID":"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef","Type":"ContainerStarted","Data":"0abda4c6a62cf12a8625152c5a319177739112e09184c8ad899c71f78051d827"}
Sep 17 17:07:08.318207 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:08.317855    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h" event={"ID":"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef","Type":"ContainerStarted","Data":"01d74f4932aba9c644e56cb343d86e2087b1147abeff0cf631c95b71daf64aa7"}
Sep 17 17:07:08.319310 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:08.319284    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785tbw2p" event={"ID":"217e7d64-317c-4430-b4e8-6d3b220b73be","Type":"ContainerStarted","Data":"47f7cd6c2eb3b0d0a03f11bd15578b01061c4a14f25b0e81a788aaa1fa8c0eeb"}
Sep 17 17:07:09.327418 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:09.327381    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785tbw2p" event={"ID":"217e7d64-317c-4430-b4e8-6d3b220b73be","Type":"ContainerStarted","Data":"b45c9ff9c7856b479e673c70afc3cd340e7abfa8a158b70ef81e6d9ead3953ef"}
Sep 17 17:07:09.327856 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:09.327473    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785tbw2p"
Sep 17 17:07:09.348206 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:09.348159    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785tbw2p" podStartSLOduration=2.348136108 podStartE2EDuration="2.348136108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:07:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:07:09.346837349 +0000 UTC m=+1515.241316489" watchObservedRunningTime="2026-09-17 17:07:09.348136108 +0000 UTC m=+1515.242615247"
Sep 17 17:07:11.342545 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:11.342497    2566 generic.go:356] "Generic (PLEG): container finished" podID="8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef" containerID="0abda4c6a62cf12a8625152c5a319177739112e09184c8ad899c71f78051d827" exitCode=0
Sep 17 17:07:11.342933 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:11.342572    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h" event={"ID":"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef","Type":"ContainerDied","Data":"0abda4c6a62cf12a8625152c5a319177739112e09184c8ad899c71f78051d827"}
Sep 17 17:07:12.350959 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:12.350915    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h" event={"ID":"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef","Type":"ContainerStarted","Data":"c1c6f2f025825a2095f51d1ba259034904dfa20ee7c23f164bb755e131551464"}
Sep 17 17:07:12.375010 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:12.374941    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h" podStartSLOduration=2.83321224 podStartE2EDuration="5.374918924s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.541706684s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:07:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:07:12.374103185 +0000 UTC m=+1518.268582326" watchObservedRunningTime="2026-09-17 17:07:12.374918924 +0000 UTC m=+1518.269398063"
Sep 17 17:07:12.616209 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:12.616096    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h"
Sep 17 17:07:14.682663 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:14.682618    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv" podUID="8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 17:07:17.616219 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:17.616173    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h"
Sep 17 17:07:17.629840 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:17.629809    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h"
Sep 17 17:07:18.393790 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:18.393760    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h"
Sep 17 17:07:24.681829 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:24.681783    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv" podUID="8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 17:07:34.682351 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:34.682304    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv" podUID="8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 17:07:40.337815 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:40.337779    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785tbw2p"
Sep 17 17:07:44.682629 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:44.682582    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv" podUID="8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 17:07:45.060133 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:45.060097    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h"]
Sep 17 17:07:45.069553 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:07:45.069520    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:07:45.069553 ip-10-0-141-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:07:45.069553 ip-10-0-141-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:07:45.069553 ip-10-0-141-66 kubenswrapper[2566]:  > containerID="c1c6f2f025825a2095f51d1ba259034904dfa20ee7c23f164bb755e131551464" cmd=["/bin/sleep","15"]
Sep 17 17:07:45.069882 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:45.069569    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h" message="" err=<
Sep 17 17:07:45.069882 ip-10-0-141-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:07:45.069882 ip-10-0-141-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:07:45.069882 ip-10-0-141-66 kubenswrapper[2566]:  >
Sep 17 17:07:45.069882 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:07:45.069582    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:07:45.069882 ip-10-0-141-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:07:45.069882 ip-10-0-141-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:07:45.069882 ip-10-0-141-66 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h" podUID="8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef" containerName="main" containerID="cri-o://c1c6f2f025825a2095f51d1ba259034904dfa20ee7c23f164bb755e131551464"
Sep 17 17:07:45.069882 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:45.069616    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h" podUID="8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef" containerName="main" containerID="cri-o://c1c6f2f025825a2095f51d1ba259034904dfa20ee7c23f164bb755e131551464" gracePeriod=60
Sep 17 17:07:45.086256 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:45.086221    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785tbw2p"]
Sep 17 17:07:45.382960 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:45.382858    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h" podUID="8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 17 17:07:45.528329 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:45.528291    2566 generic.go:356] "Generic (PLEG): container finished" podID="8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef" containerID="c1c6f2f025825a2095f51d1ba259034904dfa20ee7c23f164bb755e131551464" exitCode=0
Sep 17 17:07:45.528329 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:45.528318    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h" event={"ID":"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef","Type":"ContainerDied","Data":"c1c6f2f025825a2095f51d1ba259034904dfa20ee7c23f164bb755e131551464"}
Sep 17 17:07:45.841762 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:45.841735    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h"
Sep 17 17:07:45.948221 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:45.948123    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-model-cache\") pod \"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef\" (UID: \"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef\") "
Sep 17 17:07:45.948221 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:45.948194    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-tls-certs\") pod \"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef\" (UID: \"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef\") "
Sep 17 17:07:45.948485 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:45.948249    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-cabundle-cert\") pod \"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef\" (UID: \"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef\") "
Sep 17 17:07:45.948485 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:45.948275    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-home\") pod \"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef\" (UID: \"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef\") "
Sep 17 17:07:45.948485 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:45.948314    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-kserve-provision-location\") pod \"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef\" (UID: \"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef\") "
Sep 17 17:07:45.948485 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:45.948348    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8c7br\" (UniqueName: \"kubernetes.io/projected/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-kube-api-access-8c7br\") pod \"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef\" (UID: \"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef\") "
Sep 17 17:07:45.948485 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:45.948370    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-dshm\") pod \"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef\" (UID: \"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef\") "
Sep 17 17:07:45.948485 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:45.948397    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-tmp-dir\") pod \"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef\" (UID: \"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef\") "
Sep 17 17:07:45.948485 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:45.948394    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-model-cache" (OuterVolumeSpecName: "model-cache") pod "8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef" (UID: "8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:07:45.948848 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:45.948667    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-model-cache\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:07:45.948848 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:45.948665    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef" (UID: "8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:07:45.948848 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:45.948809    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef" (UID: "8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:07:45.949000 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:45.948925    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-home" (OuterVolumeSpecName: "home") pod "8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef" (UID: "8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:07:45.951335 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:45.951304    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-dshm" (OuterVolumeSpecName: "dshm") pod "8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef" (UID: "8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:07:45.951488 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:45.951343    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef" (UID: "8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:07:45.951488 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:45.951358    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-kube-api-access-8c7br" (OuterVolumeSpecName: "kube-api-access-8c7br") pod "8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef" (UID: "8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef"). InnerVolumeSpecName "kube-api-access-8c7br". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:07:45.967518 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:45.967486    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef" (UID: "8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:07:46.049395 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:46.049350    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c7br\" (UniqueName: \"kubernetes.io/projected/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-kube-api-access-8c7br\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:07:46.049395 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:46.049389    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-dshm\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:07:46.049395 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:46.049400    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-tmp-dir\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:07:46.049395 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:46.049408    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:07:46.049669 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:46.049417    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-cabundle-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:07:46.049669 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:46.049424    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-home\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:07:46.049669 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:46.049443    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef-kserve-provision-location\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:07:46.535705 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:46.535666    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h" event={"ID":"8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef","Type":"ContainerDied","Data":"01d74f4932aba9c644e56cb343d86e2087b1147abeff0cf631c95b71daf64aa7"}
Sep 17 17:07:46.535848 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:46.535736    2566 scope.go:117] "RemoveContainer" containerID="c1c6f2f025825a2095f51d1ba259034904dfa20ee7c23f164bb755e131551464"
Sep 17 17:07:46.535848 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:46.535686    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h"
Sep 17 17:07:46.546899 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:46.546743    2566 scope.go:117] "RemoveContainer" containerID="0abda4c6a62cf12a8625152c5a319177739112e09184c8ad899c71f78051d827"
Sep 17 17:07:46.563831 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:46.563802    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h"]
Sep 17 17:07:46.565763 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:46.565741    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-pjx2h"]
Sep 17 17:07:48.347622 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:48.347581    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef" path="/var/lib/kubelet/pods/8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef/volumes"
Sep 17 17:07:50.337083 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:50.337044    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785tbw2p" podUID="217e7d64-317c-4430-b4e8-6d3b220b73be" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:07:54.682567 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:07:54.682519    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv" podUID="8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 17:08:00.105621 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:00.105550    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785tbw2p" podUID="217e7d64-317c-4430-b4e8-6d3b220b73be" containerName="main" containerID="cri-o://b45c9ff9c7856b479e673c70afc3cd340e7abfa8a158b70ef81e6d9ead3953ef" gracePeriod=45
Sep 17 17:08:00.337263 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:00.337224    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785tbw2p" podUID="217e7d64-317c-4430-b4e8-6d3b220b73be" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:08:04.682228 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:04.682180    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv" podUID="8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 17:08:10.337323 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:10.337274    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785tbw2p" podUID="217e7d64-317c-4430-b4e8-6d3b220b73be" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:08:10.337731 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:10.337406    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785tbw2p"
Sep 17 17:08:14.682401 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:14.682344    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv" podUID="8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 17:08:20.337744 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:20.337703    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785tbw2p" podUID="217e7d64-317c-4430-b4e8-6d3b220b73be" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:08:24.692264 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:24.692229    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv"
Sep 17 17:08:24.700423 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:24.700396    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv"
Sep 17 17:08:26.621338 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:26.621304    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv"]
Sep 17 17:08:30.368973 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:30.368899    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785tbw2p"
Sep 17 17:08:30.422571 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:30.422536    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/217e7d64-317c-4430-b4e8-6d3b220b73be-tls-certs\") pod \"217e7d64-317c-4430-b4e8-6d3b220b73be\" (UID: \"217e7d64-317c-4430-b4e8-6d3b220b73be\") "
Sep 17 17:08:30.422709 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:30.422621    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-776lh\" (UniqueName: \"kubernetes.io/projected/217e7d64-317c-4430-b4e8-6d3b220b73be-kube-api-access-776lh\") pod \"217e7d64-317c-4430-b4e8-6d3b220b73be\" (UID: \"217e7d64-317c-4430-b4e8-6d3b220b73be\") "
Sep 17 17:08:30.424800 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:30.424767    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/217e7d64-317c-4430-b4e8-6d3b220b73be-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "217e7d64-317c-4430-b4e8-6d3b220b73be" (UID: "217e7d64-317c-4430-b4e8-6d3b220b73be"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:08:30.424924 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:30.424831    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/217e7d64-317c-4430-b4e8-6d3b220b73be-kube-api-access-776lh" (OuterVolumeSpecName: "kube-api-access-776lh") pod "217e7d64-317c-4430-b4e8-6d3b220b73be" (UID: "217e7d64-317c-4430-b4e8-6d3b220b73be"). InnerVolumeSpecName "kube-api-access-776lh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:08:30.523771 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:30.523735    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/217e7d64-317c-4430-b4e8-6d3b220b73be-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:08:30.523771 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:30.523765    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-776lh\" (UniqueName: \"kubernetes.io/projected/217e7d64-317c-4430-b4e8-6d3b220b73be-kube-api-access-776lh\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:08:30.776029 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:30.775934    2566 generic.go:356] "Generic (PLEG): container finished" podID="217e7d64-317c-4430-b4e8-6d3b220b73be" containerID="b45c9ff9c7856b479e673c70afc3cd340e7abfa8a158b70ef81e6d9ead3953ef" exitCode=0
Sep 17 17:08:30.776029 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:30.775986    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785tbw2p"
Sep 17 17:08:30.776029 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:30.776010    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785tbw2p" event={"ID":"217e7d64-317c-4430-b4e8-6d3b220b73be","Type":"ContainerDied","Data":"b45c9ff9c7856b479e673c70afc3cd340e7abfa8a158b70ef81e6d9ead3953ef"}
Sep 17 17:08:30.776278 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:30.776095    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785tbw2p" event={"ID":"217e7d64-317c-4430-b4e8-6d3b220b73be","Type":"ContainerDied","Data":"47f7cd6c2eb3b0d0a03f11bd15578b01061c4a14f25b0e81a788aaa1fa8c0eeb"}
Sep 17 17:08:30.776278 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:30.776125    2566 scope.go:117] "RemoveContainer" containerID="b45c9ff9c7856b479e673c70afc3cd340e7abfa8a158b70ef81e6d9ead3953ef"
Sep 17 17:08:30.785783 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:30.785760    2566 scope.go:117] "RemoveContainer" containerID="b45c9ff9c7856b479e673c70afc3cd340e7abfa8a158b70ef81e6d9ead3953ef"
Sep 17 17:08:30.786026 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:08:30.786005    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b45c9ff9c7856b479e673c70afc3cd340e7abfa8a158b70ef81e6d9ead3953ef\": container with ID starting with b45c9ff9c7856b479e673c70afc3cd340e7abfa8a158b70ef81e6d9ead3953ef not found: ID does not exist" containerID="b45c9ff9c7856b479e673c70afc3cd340e7abfa8a158b70ef81e6d9ead3953ef"
Sep 17 17:08:30.786073 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:30.786035    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b45c9ff9c7856b479e673c70afc3cd340e7abfa8a158b70ef81e6d9ead3953ef"} err="failed to get container status \"b45c9ff9c7856b479e673c70afc3cd340e7abfa8a158b70ef81e6d9ead3953ef\": rpc error: code = NotFound desc = could not find container \"b45c9ff9c7856b479e673c70afc3cd340e7abfa8a158b70ef81e6d9ead3953ef\": container with ID starting with b45c9ff9c7856b479e673c70afc3cd340e7abfa8a158b70ef81e6d9ead3953ef not found: ID does not exist"
Sep 17 17:08:30.800294 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:30.800269    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785tbw2p"]
Sep 17 17:08:30.804949 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:30.804925    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785tbw2p"]
Sep 17 17:08:31.336322 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:31.336277    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785tbw2p" podUID="217e7d64-317c-4430-b4e8-6d3b220b73be" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.133.0.51:9003\" within 1s: context deadline exceeded"
Sep 17 17:08:31.336539 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:08:31.336407    2566 logging.go:55] [core] [Channel #197 SubChannel #198]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.51:9003", ServerName: "10.133.0.51:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.51:9003: operation was canceled"
Sep 17 17:08:32.346372 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:32.346335    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="217e7d64-317c-4430-b4e8-6d3b220b73be" path="/var/lib/kubelet/pods/217e7d64-317c-4430-b4e8-6d3b220b73be/volumes"
Sep 17 17:08:41.644267 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:41.644165    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv" podUID="8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e" containerName="main" containerID="cri-o://1fcb9893b0d1e34c51ce6fe61acc4849160ddf325ec4d68c90014e0b7c88af95" gracePeriod=45
Sep 17 17:08:41.693015 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:41.692960    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv" podUID="8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 17:08:42.393897 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:42.393870    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv"
Sep 17 17:08:42.424482 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:42.424374    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-tmp-dir\") pod \"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e\" (UID: \"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e\") "
Sep 17 17:08:42.424728 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:42.424709    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-home\") pod \"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e\" (UID: \"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e\") "
Sep 17 17:08:42.424849 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:42.424837    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-dshm\") pod \"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e\" (UID: \"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e\") "
Sep 17 17:08:42.424971 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:42.424959    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-model-cache\") pod \"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e\" (UID: \"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e\") "
Sep 17 17:08:42.425098 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:42.425086    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4szsp\" (UniqueName: \"kubernetes.io/projected/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-kube-api-access-4szsp\") pod \"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e\" (UID: \"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e\") "
Sep 17 17:08:42.425228 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:42.425215    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-cabundle-cert\") pod \"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e\" (UID: \"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e\") "
Sep 17 17:08:42.425321 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:42.425310    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-tls-certs\") pod \"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e\" (UID: \"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e\") "
Sep 17 17:08:42.425416 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:42.425404    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-kserve-provision-location\") pod \"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e\" (UID: \"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e\") "
Sep 17 17:08:42.426128 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:42.426099    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-model-cache" (OuterVolumeSpecName: "model-cache") pod "8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e" (UID: "8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:08:42.426908 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:42.426855    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e" (UID: "8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:08:42.427420 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:42.427393    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-home" (OuterVolumeSpecName: "home") pod "8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e" (UID: "8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:08:42.429107 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:42.429062    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e" (UID: "8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:08:42.429217 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:42.429158    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-dshm" (OuterVolumeSpecName: "dshm") pod "8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e" (UID: "8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:08:42.429217 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:42.429178    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-kube-api-access-4szsp" (OuterVolumeSpecName: "kube-api-access-4szsp") pod "8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e" (UID: "8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e"). InnerVolumeSpecName "kube-api-access-4szsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:08:42.440162 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:42.440132    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e" (UID: "8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:08:42.450914 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:42.450888    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e" (UID: "8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:08:42.526500 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:42.526463    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4szsp\" (UniqueName: \"kubernetes.io/projected/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-kube-api-access-4szsp\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:08:42.526669 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:42.526516    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-cabundle-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:08:42.526669 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:42.526527    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:08:42.526669 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:42.526535    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-kserve-provision-location\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:08:42.526669 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:42.526544    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-tmp-dir\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:08:42.526669 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:42.526553    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-home\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:08:42.526669 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:42.526560    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-dshm\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:08:42.526669 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:42.526567    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e-model-cache\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:08:42.843423 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:42.843380    2566 generic.go:356] "Generic (PLEG): container finished" podID="8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e" containerID="1fcb9893b0d1e34c51ce6fe61acc4849160ddf325ec4d68c90014e0b7c88af95" exitCode=0
Sep 17 17:08:42.843876 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:42.843425    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv" event={"ID":"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e","Type":"ContainerDied","Data":"1fcb9893b0d1e34c51ce6fe61acc4849160ddf325ec4d68c90014e0b7c88af95"}
Sep 17 17:08:42.843876 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:42.843473    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv"
Sep 17 17:08:42.843876 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:42.843548    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv" event={"ID":"8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e","Type":"ContainerDied","Data":"2fb80edfcd09b9d265564f21bedf34c8197f6d8532b6b82bcf8ed6214c9f4099"}
Sep 17 17:08:42.843876 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:42.843575    2566 scope.go:117] "RemoveContainer" containerID="1fcb9893b0d1e34c51ce6fe61acc4849160ddf325ec4d68c90014e0b7c88af95"
Sep 17 17:08:42.853032 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:42.853013    2566 scope.go:117] "RemoveContainer" containerID="9cfa1d1c5c3289f3ce43446b87953664e88f60d02b2a66d5b5de5b8eae6948c4"
Sep 17 17:08:42.861055 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:42.861037    2566 scope.go:117] "RemoveContainer" containerID="1fcb9893b0d1e34c51ce6fe61acc4849160ddf325ec4d68c90014e0b7c88af95"
Sep 17 17:08:42.861341 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:08:42.861320    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fcb9893b0d1e34c51ce6fe61acc4849160ddf325ec4d68c90014e0b7c88af95\": container with ID starting with 1fcb9893b0d1e34c51ce6fe61acc4849160ddf325ec4d68c90014e0b7c88af95 not found: ID does not exist" containerID="1fcb9893b0d1e34c51ce6fe61acc4849160ddf325ec4d68c90014e0b7c88af95"
Sep 17 17:08:42.861476 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:42.861348    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fcb9893b0d1e34c51ce6fe61acc4849160ddf325ec4d68c90014e0b7c88af95"} err="failed to get container status \"1fcb9893b0d1e34c51ce6fe61acc4849160ddf325ec4d68c90014e0b7c88af95\": rpc error: code = NotFound desc = could not find container \"1fcb9893b0d1e34c51ce6fe61acc4849160ddf325ec4d68c90014e0b7c88af95\": container with ID starting with 1fcb9893b0d1e34c51ce6fe61acc4849160ddf325ec4d68c90014e0b7c88af95 not found: ID does not exist"
Sep 17 17:08:42.861476 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:42.861364    2566 scope.go:117] "RemoveContainer" containerID="9cfa1d1c5c3289f3ce43446b87953664e88f60d02b2a66d5b5de5b8eae6948c4"
Sep 17 17:08:42.861669 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:08:42.861650    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cfa1d1c5c3289f3ce43446b87953664e88f60d02b2a66d5b5de5b8eae6948c4\": container with ID starting with 9cfa1d1c5c3289f3ce43446b87953664e88f60d02b2a66d5b5de5b8eae6948c4 not found: ID does not exist" containerID="9cfa1d1c5c3289f3ce43446b87953664e88f60d02b2a66d5b5de5b8eae6948c4"
Sep 17 17:08:42.861713 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:42.861675    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cfa1d1c5c3289f3ce43446b87953664e88f60d02b2a66d5b5de5b8eae6948c4"} err="failed to get container status \"9cfa1d1c5c3289f3ce43446b87953664e88f60d02b2a66d5b5de5b8eae6948c4\": rpc error: code = NotFound desc = could not find container \"9cfa1d1c5c3289f3ce43446b87953664e88f60d02b2a66d5b5de5b8eae6948c4\": container with ID starting with 9cfa1d1c5c3289f3ce43446b87953664e88f60d02b2a66d5b5de5b8eae6948c4 not found: ID does not exist"
Sep 17 17:08:42.868278 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:42.868253    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv"]
Sep 17 17:08:42.873124 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:42.873098    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-9fdbb9d5-rdtzv"]
Sep 17 17:08:44.346538 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:44.346501    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e" path="/var/lib/kubelet/pods/8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e/volumes"
Sep 17 17:08:46.402449 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.402404    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9"]
Sep 17 17:08:46.402912 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.402781    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="217e7d64-317c-4430-b4e8-6d3b220b73be" containerName="main"
Sep 17 17:08:46.402912 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.402794    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="217e7d64-317c-4430-b4e8-6d3b220b73be" containerName="main"
Sep 17 17:08:46.402912 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.402802    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e" containerName="storage-initializer"
Sep 17 17:08:46.402912 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.402808    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e" containerName="storage-initializer"
Sep 17 17:08:46.402912 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.402815    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef" containerName="storage-initializer"
Sep 17 17:08:46.402912 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.402821    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef" containerName="storage-initializer"
Sep 17 17:08:46.402912 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.402832    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e" containerName="main"
Sep 17 17:08:46.402912 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.402838    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e" containerName="main"
Sep 17 17:08:46.402912 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.402844    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef" containerName="main"
Sep 17 17:08:46.402912 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.402849    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef" containerName="main"
Sep 17 17:08:46.402912 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.402896    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ea69b3f-7747-4a6e-bbae-48bb2ba1f0ef" containerName="main"
Sep 17 17:08:46.402912 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.402902    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b7e93ba-3f6e-4fe2-85d4-ce5aaed3d19e" containerName="main"
Sep 17 17:08:46.402912 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.402910    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="217e7d64-317c-4430-b4e8-6d3b220b73be" containerName="main"
Sep 17 17:08:46.406018 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.406001    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9"
Sep 17 17:08:46.408534 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.408516    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:08:46.408739 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.408722    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 17 17:08:46.409171 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.409151    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 17 17:08:46.409307 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.409287    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 17 17:08:46.409423 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.409288    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-fsnqb\""
Sep 17 17:08:46.409423 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.409412    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 17 17:08:46.421418 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.421391    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9"]
Sep 17 17:08:46.460354 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.460315    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/054ab3b8-2b6b-4ad2-861e-be5d8591e137-home\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-k4hb9\" (UID: \"054ab3b8-2b6b-4ad2-861e-be5d8591e137\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9"
Sep 17 17:08:46.460568 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.460375    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58lcc\" (UniqueName: \"kubernetes.io/projected/054ab3b8-2b6b-4ad2-861e-be5d8591e137-kube-api-access-58lcc\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-k4hb9\" (UID: \"054ab3b8-2b6b-4ad2-861e-be5d8591e137\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9"
Sep 17 17:08:46.460568 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.460419    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/054ab3b8-2b6b-4ad2-861e-be5d8591e137-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-k4hb9\" (UID: \"054ab3b8-2b6b-4ad2-861e-be5d8591e137\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9"
Sep 17 17:08:46.460568 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.460470    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/054ab3b8-2b6b-4ad2-861e-be5d8591e137-model-cache\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-k4hb9\" (UID: \"054ab3b8-2b6b-4ad2-861e-be5d8591e137\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9"
Sep 17 17:08:46.460568 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.460491    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/054ab3b8-2b6b-4ad2-861e-be5d8591e137-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-k4hb9\" (UID: \"054ab3b8-2b6b-4ad2-861e-be5d8591e137\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9"
Sep 17 17:08:46.460568 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.460512    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/054ab3b8-2b6b-4ad2-861e-be5d8591e137-tls-certs\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-k4hb9\" (UID: \"054ab3b8-2b6b-4ad2-861e-be5d8591e137\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9"
Sep 17 17:08:46.460568 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.460544    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/054ab3b8-2b6b-4ad2-861e-be5d8591e137-dshm\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-k4hb9\" (UID: \"054ab3b8-2b6b-4ad2-861e-be5d8591e137\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9"
Sep 17 17:08:46.460799 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.460607    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/054ab3b8-2b6b-4ad2-861e-be5d8591e137-tmp-dir\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-k4hb9\" (UID: \"054ab3b8-2b6b-4ad2-861e-be5d8591e137\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9"
Sep 17 17:08:46.561415 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.561374    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58lcc\" (UniqueName: \"kubernetes.io/projected/054ab3b8-2b6b-4ad2-861e-be5d8591e137-kube-api-access-58lcc\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-k4hb9\" (UID: \"054ab3b8-2b6b-4ad2-861e-be5d8591e137\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9"
Sep 17 17:08:46.561415 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.561417    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/054ab3b8-2b6b-4ad2-861e-be5d8591e137-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-k4hb9\" (UID: \"054ab3b8-2b6b-4ad2-861e-be5d8591e137\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9"
Sep 17 17:08:46.561676 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.561469    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/054ab3b8-2b6b-4ad2-861e-be5d8591e137-model-cache\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-k4hb9\" (UID: \"054ab3b8-2b6b-4ad2-861e-be5d8591e137\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9"
Sep 17 17:08:46.561676 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.561497    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/054ab3b8-2b6b-4ad2-861e-be5d8591e137-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-k4hb9\" (UID: \"054ab3b8-2b6b-4ad2-861e-be5d8591e137\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9"
Sep 17 17:08:46.561676 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.561528    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/054ab3b8-2b6b-4ad2-861e-be5d8591e137-tls-certs\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-k4hb9\" (UID: \"054ab3b8-2b6b-4ad2-861e-be5d8591e137\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9"
Sep 17 17:08:46.561676 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.561567    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/054ab3b8-2b6b-4ad2-861e-be5d8591e137-dshm\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-k4hb9\" (UID: \"054ab3b8-2b6b-4ad2-861e-be5d8591e137\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9"
Sep 17 17:08:46.561676 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.561618    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/054ab3b8-2b6b-4ad2-861e-be5d8591e137-tmp-dir\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-k4hb9\" (UID: \"054ab3b8-2b6b-4ad2-861e-be5d8591e137\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9"
Sep 17 17:08:46.561676 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.561652    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/054ab3b8-2b6b-4ad2-861e-be5d8591e137-home\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-k4hb9\" (UID: \"054ab3b8-2b6b-4ad2-861e-be5d8591e137\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9"
Sep 17 17:08:46.562019 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.561991    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/054ab3b8-2b6b-4ad2-861e-be5d8591e137-model-cache\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-k4hb9\" (UID: \"054ab3b8-2b6b-4ad2-861e-be5d8591e137\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9"
Sep 17 17:08:46.562019 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.562009    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/054ab3b8-2b6b-4ad2-861e-be5d8591e137-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-k4hb9\" (UID: \"054ab3b8-2b6b-4ad2-861e-be5d8591e137\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9"
Sep 17 17:08:46.562133 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.562073    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/054ab3b8-2b6b-4ad2-861e-be5d8591e137-tmp-dir\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-k4hb9\" (UID: \"054ab3b8-2b6b-4ad2-861e-be5d8591e137\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9"
Sep 17 17:08:46.562133 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.562086    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/054ab3b8-2b6b-4ad2-861e-be5d8591e137-home\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-k4hb9\" (UID: \"054ab3b8-2b6b-4ad2-861e-be5d8591e137\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9"
Sep 17 17:08:46.562246 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.562230    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/054ab3b8-2b6b-4ad2-861e-be5d8591e137-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-k4hb9\" (UID: \"054ab3b8-2b6b-4ad2-861e-be5d8591e137\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9"
Sep 17 17:08:46.564117 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.564095    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/054ab3b8-2b6b-4ad2-861e-be5d8591e137-dshm\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-k4hb9\" (UID: \"054ab3b8-2b6b-4ad2-861e-be5d8591e137\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9"
Sep 17 17:08:46.564309 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.564293    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/054ab3b8-2b6b-4ad2-861e-be5d8591e137-tls-certs\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-k4hb9\" (UID: \"054ab3b8-2b6b-4ad2-861e-be5d8591e137\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9"
Sep 17 17:08:46.571828 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.571804    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58lcc\" (UniqueName: \"kubernetes.io/projected/054ab3b8-2b6b-4ad2-861e-be5d8591e137-kube-api-access-58lcc\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-k4hb9\" (UID: \"054ab3b8-2b6b-4ad2-861e-be5d8591e137\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9"
Sep 17 17:08:46.716345 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.716247    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9"
Sep 17 17:08:46.741331 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.741298    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d5q5qf"]
Sep 17 17:08:46.744835 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.744813    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d5q5qf"
Sep 17 17:08:46.747775 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.747753    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-epp-sa-dockercfg-j5pv2\""
Sep 17 17:08:46.759654 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.759627    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d5q5qf"]
Sep 17 17:08:46.864626 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.864588    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhtjb\" (UniqueName: \"kubernetes.io/projected/71f1a84d-cc85-4301-8318-ab383bbc36d9-kube-api-access-hhtjb\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7d5q5qf\" (UID: \"71f1a84d-cc85-4301-8318-ab383bbc36d9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d5q5qf"
Sep 17 17:08:46.864807 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.864650    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71f1a84d-cc85-4301-8318-ab383bbc36d9-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7d5q5qf\" (UID: \"71f1a84d-cc85-4301-8318-ab383bbc36d9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d5q5qf"
Sep 17 17:08:46.965785 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.965745    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71f1a84d-cc85-4301-8318-ab383bbc36d9-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7d5q5qf\" (UID: \"71f1a84d-cc85-4301-8318-ab383bbc36d9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d5q5qf"
Sep 17 17:08:46.966005 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.965813    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhtjb\" (UniqueName: \"kubernetes.io/projected/71f1a84d-cc85-4301-8318-ab383bbc36d9-kube-api-access-hhtjb\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7d5q5qf\" (UID: \"71f1a84d-cc85-4301-8318-ab383bbc36d9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d5q5qf"
Sep 17 17:08:46.968623 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.968561    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71f1a84d-cc85-4301-8318-ab383bbc36d9-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7d5q5qf\" (UID: \"71f1a84d-cc85-4301-8318-ab383bbc36d9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d5q5qf"
Sep 17 17:08:46.979784 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:46.979756    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhtjb\" (UniqueName: \"kubernetes.io/projected/71f1a84d-cc85-4301-8318-ab383bbc36d9-kube-api-access-hhtjb\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7d5q5qf\" (UID: \"71f1a84d-cc85-4301-8318-ab383bbc36d9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d5q5qf"
Sep 17 17:08:47.013338 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:47.013309    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9"]
Sep 17 17:08:47.014813 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:08:47.014787    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod054ab3b8_2b6b_4ad2_861e_be5d8591e137.slice/crio-039ed1c486a122aa213148e65c04be945d58ed6c909539da6a6d3f555e0efc11 WatchSource:0}: Error finding container 039ed1c486a122aa213148e65c04be945d58ed6c909539da6a6d3f555e0efc11: Status 404 returned error can't find the container with id 039ed1c486a122aa213148e65c04be945d58ed6c909539da6a6d3f555e0efc11
Sep 17 17:08:47.057783 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:47.057750    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d5q5qf"
Sep 17 17:08:47.342966 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:47.342790    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d5q5qf"]
Sep 17 17:08:47.345062 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:08:47.345035    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71f1a84d_cc85_4301_8318_ab383bbc36d9.slice/crio-5d15f5d037d4790a5522e8afa9747783bf50736d9d21d27170097224893d9481 WatchSource:0}: Error finding container 5d15f5d037d4790a5522e8afa9747783bf50736d9d21d27170097224893d9481: Status 404 returned error can't find the container with id 5d15f5d037d4790a5522e8afa9747783bf50736d9d21d27170097224893d9481
Sep 17 17:08:47.872904 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:47.872861    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d5q5qf" event={"ID":"71f1a84d-cc85-4301-8318-ab383bbc36d9","Type":"ContainerStarted","Data":"7e1fbbcc87cf618d2f661d067c8f959f91bf03ba9223ba2f251b180c3e2606f0"}
Sep 17 17:08:47.873367 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:47.872962    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d5q5qf" event={"ID":"71f1a84d-cc85-4301-8318-ab383bbc36d9","Type":"ContainerStarted","Data":"5d15f5d037d4790a5522e8afa9747783bf50736d9d21d27170097224893d9481"}
Sep 17 17:08:47.873367 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:47.872996    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d5q5qf"
Sep 17 17:08:47.874544 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:47.874513    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9" event={"ID":"054ab3b8-2b6b-4ad2-861e-be5d8591e137","Type":"ContainerStarted","Data":"89f2149249fc86855f78a14215d7e690f448635e78d7aace7888bc78deaae540"}
Sep 17 17:08:47.874691 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:47.874559    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9" event={"ID":"054ab3b8-2b6b-4ad2-861e-be5d8591e137","Type":"ContainerStarted","Data":"039ed1c486a122aa213148e65c04be945d58ed6c909539da6a6d3f555e0efc11"}
Sep 17 17:08:47.895120 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:47.895065    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d5q5qf" podStartSLOduration=1.89504874 podStartE2EDuration="1.89504874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:08:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:08:47.892772828 +0000 UTC m=+1613.787251970" watchObservedRunningTime="2026-09-17 17:08:47.89504874 +0000 UTC m=+1613.789527871"
Sep 17 17:08:49.886786 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:49.886754    2566 generic.go:356] "Generic (PLEG): container finished" podID="054ab3b8-2b6b-4ad2-861e-be5d8591e137" containerID="89f2149249fc86855f78a14215d7e690f448635e78d7aace7888bc78deaae540" exitCode=0
Sep 17 17:08:49.887178 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:49.886839    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9" event={"ID":"054ab3b8-2b6b-4ad2-861e-be5d8591e137","Type":"ContainerDied","Data":"89f2149249fc86855f78a14215d7e690f448635e78d7aace7888bc78deaae540"}
Sep 17 17:08:50.894043 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:50.894009    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9" event={"ID":"054ab3b8-2b6b-4ad2-861e-be5d8591e137","Type":"ContainerStarted","Data":"f5e4f6baee00c641084502352571051f73539881a712ab2c845d33ba6dfca7e6"}
Sep 17 17:08:50.917228 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:50.917152    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9" podStartSLOduration=2.486155622 podStartE2EDuration="4.917134213s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.430978591s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:08:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:08:50.915424477 +0000 UTC m=+1616.809903607" watchObservedRunningTime="2026-09-17 17:08:50.917134213 +0000 UTC m=+1616.811613351"
Sep 17 17:08:51.716499 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:51.716457    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9"
Sep 17 17:08:56.716489 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:56.716427    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9"
Sep 17 17:08:56.733153 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:56.733072    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9"
Sep 17 17:08:56.936455 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:08:56.936403    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9"
Sep 17 17:09:18.881683 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:18.881650    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d5q5qf"
Sep 17 17:09:25.396241 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:25.396196    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx"]
Sep 17 17:09:25.399965 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:25.399941    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx"
Sep 17 17:09:25.404181 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:25.404155    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:09:25.404310 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:25.404215    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 17 17:09:25.404475 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:25.404457    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-88rxs\""
Sep 17 17:09:25.404563 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:25.404547    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 17 17:09:25.404737 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:25.404724    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 17 17:09:25.404804 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:25.404786    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Sep 17 17:09:25.418508 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:25.418483    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx"]
Sep 17 17:09:25.490899 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:25.490865    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-kserve-provision-location\") pod \"auth-disabled-test-kserve-6d5c585b7d-dp4dx\" (UID: \"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx"
Sep 17 17:09:25.490899 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:25.490897    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-cabundle-cert\") pod \"auth-disabled-test-kserve-6d5c585b7d-dp4dx\" (UID: \"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx"
Sep 17 17:09:25.491156 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:25.490920    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t457\" (UniqueName: \"kubernetes.io/projected/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-kube-api-access-7t457\") pod \"auth-disabled-test-kserve-6d5c585b7d-dp4dx\" (UID: \"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx"
Sep 17 17:09:25.491156 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:25.490985    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-home\") pod \"auth-disabled-test-kserve-6d5c585b7d-dp4dx\" (UID: \"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx"
Sep 17 17:09:25.491156 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:25.491033    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-model-cache\") pod \"auth-disabled-test-kserve-6d5c585b7d-dp4dx\" (UID: \"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx"
Sep 17 17:09:25.491156 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:25.491057    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-tmp-dir\") pod \"auth-disabled-test-kserve-6d5c585b7d-dp4dx\" (UID: \"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx"
Sep 17 17:09:25.491156 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:25.491106    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-tls-certs\") pod \"auth-disabled-test-kserve-6d5c585b7d-dp4dx\" (UID: \"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx"
Sep 17 17:09:25.491156 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:25.491151    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-dshm\") pod \"auth-disabled-test-kserve-6d5c585b7d-dp4dx\" (UID: \"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx"
Sep 17 17:09:25.591742 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:25.591705    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-tmp-dir\") pod \"auth-disabled-test-kserve-6d5c585b7d-dp4dx\" (UID: \"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx"
Sep 17 17:09:25.591927 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:25.591754    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-tls-certs\") pod \"auth-disabled-test-kserve-6d5c585b7d-dp4dx\" (UID: \"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx"
Sep 17 17:09:25.591927 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:25.591792    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-dshm\") pod \"auth-disabled-test-kserve-6d5c585b7d-dp4dx\" (UID: \"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx"
Sep 17 17:09:25.591927 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:25.591831    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-kserve-provision-location\") pod \"auth-disabled-test-kserve-6d5c585b7d-dp4dx\" (UID: \"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx"
Sep 17 17:09:25.591927 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:25.591858    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-cabundle-cert\") pod \"auth-disabled-test-kserve-6d5c585b7d-dp4dx\" (UID: \"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx"
Sep 17 17:09:25.592143 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:25.591992    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t457\" (UniqueName: \"kubernetes.io/projected/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-kube-api-access-7t457\") pod \"auth-disabled-test-kserve-6d5c585b7d-dp4dx\" (UID: \"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx"
Sep 17 17:09:25.592143 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:25.592038    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-home\") pod \"auth-disabled-test-kserve-6d5c585b7d-dp4dx\" (UID: \"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx"
Sep 17 17:09:25.592260 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:25.592184    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-model-cache\") pod \"auth-disabled-test-kserve-6d5c585b7d-dp4dx\" (UID: \"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx"
Sep 17 17:09:25.592260 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:25.592213    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-tmp-dir\") pod \"auth-disabled-test-kserve-6d5c585b7d-dp4dx\" (UID: \"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx"
Sep 17 17:09:25.592370 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:25.592337    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-home\") pod \"auth-disabled-test-kserve-6d5c585b7d-dp4dx\" (UID: \"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx"
Sep 17 17:09:25.592524 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:25.592501    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-kserve-provision-location\") pod \"auth-disabled-test-kserve-6d5c585b7d-dp4dx\" (UID: \"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx"
Sep 17 17:09:25.592764 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:25.592568    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-model-cache\") pod \"auth-disabled-test-kserve-6d5c585b7d-dp4dx\" (UID: \"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx"
Sep 17 17:09:25.592828 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:25.592789    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-cabundle-cert\") pod \"auth-disabled-test-kserve-6d5c585b7d-dp4dx\" (UID: \"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx"
Sep 17 17:09:25.594284 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:25.594262    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-dshm\") pod \"auth-disabled-test-kserve-6d5c585b7d-dp4dx\" (UID: \"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx"
Sep 17 17:09:25.594557 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:25.594540    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-tls-certs\") pod \"auth-disabled-test-kserve-6d5c585b7d-dp4dx\" (UID: \"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx"
Sep 17 17:09:25.607344 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:25.607325    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t457\" (UniqueName: \"kubernetes.io/projected/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-kube-api-access-7t457\") pod \"auth-disabled-test-kserve-6d5c585b7d-dp4dx\" (UID: \"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx"
Sep 17 17:09:25.710665 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:25.710578    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx"
Sep 17 17:09:25.997623 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:25.997582    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx"]
Sep 17 17:09:25.998803 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:09:25.998772    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91d3bcbd_ac1b_48c6_89c8_66f1ce71eb91.slice/crio-f3aecd7a9ef775781c0fc82b807af01d7ec66bf1292fc27cac47655b311fcf3b WatchSource:0}: Error finding container f3aecd7a9ef775781c0fc82b807af01d7ec66bf1292fc27cac47655b311fcf3b: Status 404 returned error can't find the container with id f3aecd7a9ef775781c0fc82b807af01d7ec66bf1292fc27cac47655b311fcf3b
Sep 17 17:09:26.084838 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:26.084810    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx" event={"ID":"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91","Type":"ContainerStarted","Data":"a51a6c9f23fd078da7b8aa1e11fd10a1366f28689fe149342ca89dda4edd61b1"}
Sep 17 17:09:26.084952 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:26.084886    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx" event={"ID":"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91","Type":"ContainerStarted","Data":"f3aecd7a9ef775781c0fc82b807af01d7ec66bf1292fc27cac47655b311fcf3b"}
Sep 17 17:09:29.104498 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:29.104458    2566 generic.go:356] "Generic (PLEG): container finished" podID="91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91" containerID="a51a6c9f23fd078da7b8aa1e11fd10a1366f28689fe149342ca89dda4edd61b1" exitCode=0
Sep 17 17:09:29.104880 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:29.104467    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx" event={"ID":"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91","Type":"ContainerDied","Data":"a51a6c9f23fd078da7b8aa1e11fd10a1366f28689fe149342ca89dda4edd61b1"}
Sep 17 17:09:30.112511 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:30.112470    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx" event={"ID":"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91","Type":"ContainerStarted","Data":"ccd88dfb172d55e7ed84734b2f42b93692d1cd7396154bbb0bcb116008220b33"}
Sep 17 17:09:30.138288 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:30.138223    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx" podStartSLOduration=2.938513221 podStartE2EDuration="5.138200266s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.199687045s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:09:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:09:30.137366857 +0000 UTC m=+1656.031845998" watchObservedRunningTime="2026-09-17 17:09:30.138200266 +0000 UTC m=+1656.032679406"
Sep 17 17:09:30.711217 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:30.711177    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx"
Sep 17 17:09:35.711566 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:35.711516    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx"
Sep 17 17:09:35.713164 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:35.713131    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx" podUID="91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 17 17:09:45.711052 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:45.710999    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx" podUID="91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 17 17:09:47.465960 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:47.465919    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9"]
Sep 17 17:09:47.475844 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:09:47.475805    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:09:47.475844 ip-10-0-141-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:09:47.475844 ip-10-0-141-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:09:47.475844 ip-10-0-141-66 kubenswrapper[2566]:  > containerID="f5e4f6baee00c641084502352571051f73539881a712ab2c845d33ba6dfca7e6" cmd=["/bin/sleep","15"]
Sep 17 17:09:47.476166 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:47.475863    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9" message="" err=<
Sep 17 17:09:47.476166 ip-10-0-141-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:09:47.476166 ip-10-0-141-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:09:47.476166 ip-10-0-141-66 kubenswrapper[2566]:  >
Sep 17 17:09:47.476166 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:09:47.475879    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:09:47.476166 ip-10-0-141-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:09:47.476166 ip-10-0-141-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:09:47.476166 ip-10-0-141-66 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9" podUID="054ab3b8-2b6b-4ad2-861e-be5d8591e137" containerName="main" containerID="cri-o://f5e4f6baee00c641084502352571051f73539881a712ab2c845d33ba6dfca7e6"
Sep 17 17:09:47.476166 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:47.475924    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9" podUID="054ab3b8-2b6b-4ad2-861e-be5d8591e137" containerName="main" containerID="cri-o://f5e4f6baee00c641084502352571051f73539881a712ab2c845d33ba6dfca7e6" gracePeriod=60
Sep 17 17:09:47.489935 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:47.489907    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d5q5qf"]
Sep 17 17:09:47.840363 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:47.840337    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9"
Sep 17 17:09:47.993305 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:47.993273    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/054ab3b8-2b6b-4ad2-861e-be5d8591e137-tls-certs\") pod \"054ab3b8-2b6b-4ad2-861e-be5d8591e137\" (UID: \"054ab3b8-2b6b-4ad2-861e-be5d8591e137\") "
Sep 17 17:09:47.993305 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:47.993311    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/054ab3b8-2b6b-4ad2-861e-be5d8591e137-dshm\") pod \"054ab3b8-2b6b-4ad2-861e-be5d8591e137\" (UID: \"054ab3b8-2b6b-4ad2-861e-be5d8591e137\") "
Sep 17 17:09:47.993590 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:47.993341    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58lcc\" (UniqueName: \"kubernetes.io/projected/054ab3b8-2b6b-4ad2-861e-be5d8591e137-kube-api-access-58lcc\") pod \"054ab3b8-2b6b-4ad2-861e-be5d8591e137\" (UID: \"054ab3b8-2b6b-4ad2-861e-be5d8591e137\") "
Sep 17 17:09:47.993590 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:47.993375    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/054ab3b8-2b6b-4ad2-861e-be5d8591e137-kserve-provision-location\") pod \"054ab3b8-2b6b-4ad2-861e-be5d8591e137\" (UID: \"054ab3b8-2b6b-4ad2-861e-be5d8591e137\") "
Sep 17 17:09:47.993590 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:47.993414    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/054ab3b8-2b6b-4ad2-861e-be5d8591e137-home\") pod \"054ab3b8-2b6b-4ad2-861e-be5d8591e137\" (UID: \"054ab3b8-2b6b-4ad2-861e-be5d8591e137\") "
Sep 17 17:09:47.993590 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:47.993486    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/054ab3b8-2b6b-4ad2-861e-be5d8591e137-model-cache\") pod \"054ab3b8-2b6b-4ad2-861e-be5d8591e137\" (UID: \"054ab3b8-2b6b-4ad2-861e-be5d8591e137\") "
Sep 17 17:09:47.993590 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:47.993520    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/054ab3b8-2b6b-4ad2-861e-be5d8591e137-tmp-dir\") pod \"054ab3b8-2b6b-4ad2-861e-be5d8591e137\" (UID: \"054ab3b8-2b6b-4ad2-861e-be5d8591e137\") "
Sep 17 17:09:47.993590 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:47.993570    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/054ab3b8-2b6b-4ad2-861e-be5d8591e137-cabundle-cert\") pod \"054ab3b8-2b6b-4ad2-861e-be5d8591e137\" (UID: \"054ab3b8-2b6b-4ad2-861e-be5d8591e137\") "
Sep 17 17:09:47.993894 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:47.993769    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/054ab3b8-2b6b-4ad2-861e-be5d8591e137-home" (OuterVolumeSpecName: "home") pod "054ab3b8-2b6b-4ad2-861e-be5d8591e137" (UID: "054ab3b8-2b6b-4ad2-861e-be5d8591e137"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:09:47.993956 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:47.993923    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/054ab3b8-2b6b-4ad2-861e-be5d8591e137-model-cache" (OuterVolumeSpecName: "model-cache") pod "054ab3b8-2b6b-4ad2-861e-be5d8591e137" (UID: "054ab3b8-2b6b-4ad2-861e-be5d8591e137"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:09:47.994244 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:47.994161    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/054ab3b8-2b6b-4ad2-861e-be5d8591e137-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "054ab3b8-2b6b-4ad2-861e-be5d8591e137" (UID: "054ab3b8-2b6b-4ad2-861e-be5d8591e137"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:09:47.994391 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:47.994277    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/054ab3b8-2b6b-4ad2-861e-be5d8591e137-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "054ab3b8-2b6b-4ad2-861e-be5d8591e137" (UID: "054ab3b8-2b6b-4ad2-861e-be5d8591e137"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:09:47.996212 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:47.996144    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/054ab3b8-2b6b-4ad2-861e-be5d8591e137-dshm" (OuterVolumeSpecName: "dshm") pod "054ab3b8-2b6b-4ad2-861e-be5d8591e137" (UID: "054ab3b8-2b6b-4ad2-861e-be5d8591e137"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:09:47.996212 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:47.996146    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/054ab3b8-2b6b-4ad2-861e-be5d8591e137-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "054ab3b8-2b6b-4ad2-861e-be5d8591e137" (UID: "054ab3b8-2b6b-4ad2-861e-be5d8591e137"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:09:47.996350 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:47.996224    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/054ab3b8-2b6b-4ad2-861e-be5d8591e137-kube-api-access-58lcc" (OuterVolumeSpecName: "kube-api-access-58lcc") pod "054ab3b8-2b6b-4ad2-861e-be5d8591e137" (UID: "054ab3b8-2b6b-4ad2-861e-be5d8591e137"). InnerVolumeSpecName "kube-api-access-58lcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:09:48.017502 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:48.017429    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/054ab3b8-2b6b-4ad2-861e-be5d8591e137-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "054ab3b8-2b6b-4ad2-861e-be5d8591e137" (UID: "054ab3b8-2b6b-4ad2-861e-be5d8591e137"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:09:48.094855 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:48.094812    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/054ab3b8-2b6b-4ad2-861e-be5d8591e137-model-cache\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:09:48.094855 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:48.094850    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/054ab3b8-2b6b-4ad2-861e-be5d8591e137-tmp-dir\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:09:48.094855 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:48.094862    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/054ab3b8-2b6b-4ad2-861e-be5d8591e137-cabundle-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:09:48.095109 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:48.094872    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/054ab3b8-2b6b-4ad2-861e-be5d8591e137-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:09:48.095109 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:48.094882    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/054ab3b8-2b6b-4ad2-861e-be5d8591e137-dshm\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:09:48.095109 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:48.094895    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58lcc\" (UniqueName: \"kubernetes.io/projected/054ab3b8-2b6b-4ad2-861e-be5d8591e137-kube-api-access-58lcc\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:09:48.095109 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:48.094907    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/054ab3b8-2b6b-4ad2-861e-be5d8591e137-kserve-provision-location\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:09:48.095109 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:48.094917    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/054ab3b8-2b6b-4ad2-861e-be5d8591e137-home\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:09:48.211590 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:48.211552    2566 generic.go:356] "Generic (PLEG): container finished" podID="054ab3b8-2b6b-4ad2-861e-be5d8591e137" containerID="f5e4f6baee00c641084502352571051f73539881a712ab2c845d33ba6dfca7e6" exitCode=0
Sep 17 17:09:48.211796 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:48.211619    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9" event={"ID":"054ab3b8-2b6b-4ad2-861e-be5d8591e137","Type":"ContainerDied","Data":"f5e4f6baee00c641084502352571051f73539881a712ab2c845d33ba6dfca7e6"}
Sep 17 17:09:48.211796 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:48.211622    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9"
Sep 17 17:09:48.211796 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:48.211667    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9" event={"ID":"054ab3b8-2b6b-4ad2-861e-be5d8591e137","Type":"ContainerDied","Data":"039ed1c486a122aa213148e65c04be945d58ed6c909539da6a6d3f555e0efc11"}
Sep 17 17:09:48.211796 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:48.211696    2566 scope.go:117] "RemoveContainer" containerID="f5e4f6baee00c641084502352571051f73539881a712ab2c845d33ba6dfca7e6"
Sep 17 17:09:48.221863 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:48.221843    2566 scope.go:117] "RemoveContainer" containerID="89f2149249fc86855f78a14215d7e690f448635e78d7aace7888bc78deaae540"
Sep 17 17:09:48.232085 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:48.232063    2566 scope.go:117] "RemoveContainer" containerID="f5e4f6baee00c641084502352571051f73539881a712ab2c845d33ba6dfca7e6"
Sep 17 17:09:48.232513 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:09:48.232418    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5e4f6baee00c641084502352571051f73539881a712ab2c845d33ba6dfca7e6\": container with ID starting with f5e4f6baee00c641084502352571051f73539881a712ab2c845d33ba6dfca7e6 not found: ID does not exist" containerID="f5e4f6baee00c641084502352571051f73539881a712ab2c845d33ba6dfca7e6"
Sep 17 17:09:48.232599 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:48.232522    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5e4f6baee00c641084502352571051f73539881a712ab2c845d33ba6dfca7e6"} err="failed to get container status \"f5e4f6baee00c641084502352571051f73539881a712ab2c845d33ba6dfca7e6\": rpc error: code = NotFound desc = could not find container \"f5e4f6baee00c641084502352571051f73539881a712ab2c845d33ba6dfca7e6\": container with ID starting with f5e4f6baee00c641084502352571051f73539881a712ab2c845d33ba6dfca7e6 not found: ID does not exist"
Sep 17 17:09:48.232599 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:48.232539    2566 scope.go:117] "RemoveContainer" containerID="89f2149249fc86855f78a14215d7e690f448635e78d7aace7888bc78deaae540"
Sep 17 17:09:48.232841 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:09:48.232815    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89f2149249fc86855f78a14215d7e690f448635e78d7aace7888bc78deaae540\": container with ID starting with 89f2149249fc86855f78a14215d7e690f448635e78d7aace7888bc78deaae540 not found: ID does not exist" containerID="89f2149249fc86855f78a14215d7e690f448635e78d7aace7888bc78deaae540"
Sep 17 17:09:48.232891 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:48.232851    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89f2149249fc86855f78a14215d7e690f448635e78d7aace7888bc78deaae540"} err="failed to get container status \"89f2149249fc86855f78a14215d7e690f448635e78d7aace7888bc78deaae540\": rpc error: code = NotFound desc = could not find container \"89f2149249fc86855f78a14215d7e690f448635e78d7aace7888bc78deaae540\": container with ID starting with 89f2149249fc86855f78a14215d7e690f448635e78d7aace7888bc78deaae540 not found: ID does not exist"
Sep 17 17:09:48.239414 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:48.239383    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9"]
Sep 17 17:09:48.247073 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:48.247013    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-k4hb9"]
Sep 17 17:09:48.347337 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:48.347301    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="054ab3b8-2b6b-4ad2-861e-be5d8591e137" path="/var/lib/kubelet/pods/054ab3b8-2b6b-4ad2-861e-be5d8591e137/volumes"
Sep 17 17:09:48.882024 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:48.881989    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d5q5qf" podUID="71f1a84d-cc85-4301-8318-ab383bbc36d9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:09:55.711050 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:55.711004    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx" podUID="91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 17 17:09:58.882603 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:09:58.882558    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d5q5qf" podUID="71f1a84d-cc85-4301-8318-ab383bbc36d9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:10:02.509927 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:10:02.509875    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d5q5qf" podUID="71f1a84d-cc85-4301-8318-ab383bbc36d9" containerName="main" containerID="cri-o://7e1fbbcc87cf618d2f661d067c8f959f91bf03ba9223ba2f251b180c3e2606f0" gracePeriod=45
Sep 17 17:10:05.711365 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:10:05.711274    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx" podUID="91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 17 17:10:08.882272 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:10:08.882233    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d5q5qf" podUID="71f1a84d-cc85-4301-8318-ab383bbc36d9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:10:08.882668 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:10:08.882338    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d5q5qf"
Sep 17 17:10:15.711661 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:10:15.711613    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx" podUID="91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 17 17:10:18.881806 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:10:18.881768    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d5q5qf" podUID="71f1a84d-cc85-4301-8318-ab383bbc36d9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:10:25.711612 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:10:25.711570    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx" podUID="91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 17 17:10:28.882169 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:10:28.882132    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d5q5qf" podUID="71f1a84d-cc85-4301-8318-ab383bbc36d9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:10:32.765486 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:10:32.765458    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d5q5qf"
Sep 17 17:10:32.884559 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:10:32.884523    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhtjb\" (UniqueName: \"kubernetes.io/projected/71f1a84d-cc85-4301-8318-ab383bbc36d9-kube-api-access-hhtjb\") pod \"71f1a84d-cc85-4301-8318-ab383bbc36d9\" (UID: \"71f1a84d-cc85-4301-8318-ab383bbc36d9\") "
Sep 17 17:10:32.884753 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:10:32.884595    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71f1a84d-cc85-4301-8318-ab383bbc36d9-tls-certs\") pod \"71f1a84d-cc85-4301-8318-ab383bbc36d9\" (UID: \"71f1a84d-cc85-4301-8318-ab383bbc36d9\") "
Sep 17 17:10:32.886825 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:10:32.886797    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71f1a84d-cc85-4301-8318-ab383bbc36d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "71f1a84d-cc85-4301-8318-ab383bbc36d9" (UID: "71f1a84d-cc85-4301-8318-ab383bbc36d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:10:32.886943 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:10:32.886857    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71f1a84d-cc85-4301-8318-ab383bbc36d9-kube-api-access-hhtjb" (OuterVolumeSpecName: "kube-api-access-hhtjb") pod "71f1a84d-cc85-4301-8318-ab383bbc36d9" (UID: "71f1a84d-cc85-4301-8318-ab383bbc36d9"). InnerVolumeSpecName "kube-api-access-hhtjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:10:32.985355 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:10:32.985306    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhtjb\" (UniqueName: \"kubernetes.io/projected/71f1a84d-cc85-4301-8318-ab383bbc36d9-kube-api-access-hhtjb\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:10:32.985355 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:10:32.985350    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71f1a84d-cc85-4301-8318-ab383bbc36d9-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:10:33.460907 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:10:33.460875    2566 generic.go:356] "Generic (PLEG): container finished" podID="71f1a84d-cc85-4301-8318-ab383bbc36d9" containerID="7e1fbbcc87cf618d2f661d067c8f959f91bf03ba9223ba2f251b180c3e2606f0" exitCode=0
Sep 17 17:10:33.461100 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:10:33.460925    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d5q5qf"
Sep 17 17:10:33.461100 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:10:33.460962    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d5q5qf" event={"ID":"71f1a84d-cc85-4301-8318-ab383bbc36d9","Type":"ContainerDied","Data":"7e1fbbcc87cf618d2f661d067c8f959f91bf03ba9223ba2f251b180c3e2606f0"}
Sep 17 17:10:33.461100 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:10:33.461017    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d5q5qf" event={"ID":"71f1a84d-cc85-4301-8318-ab383bbc36d9","Type":"ContainerDied","Data":"5d15f5d037d4790a5522e8afa9747783bf50736d9d21d27170097224893d9481"}
Sep 17 17:10:33.461100 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:10:33.461039    2566 scope.go:117] "RemoveContainer" containerID="7e1fbbcc87cf618d2f661d067c8f959f91bf03ba9223ba2f251b180c3e2606f0"
Sep 17 17:10:33.470385 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:10:33.470368    2566 scope.go:117] "RemoveContainer" containerID="7e1fbbcc87cf618d2f661d067c8f959f91bf03ba9223ba2f251b180c3e2606f0"
Sep 17 17:10:33.470698 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:10:33.470674    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e1fbbcc87cf618d2f661d067c8f959f91bf03ba9223ba2f251b180c3e2606f0\": container with ID starting with 7e1fbbcc87cf618d2f661d067c8f959f91bf03ba9223ba2f251b180c3e2606f0 not found: ID does not exist" containerID="7e1fbbcc87cf618d2f661d067c8f959f91bf03ba9223ba2f251b180c3e2606f0"
Sep 17 17:10:33.470806 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:10:33.470703    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e1fbbcc87cf618d2f661d067c8f959f91bf03ba9223ba2f251b180c3e2606f0"} err="failed to get container status \"7e1fbbcc87cf618d2f661d067c8f959f91bf03ba9223ba2f251b180c3e2606f0\": rpc error: code = NotFound desc = could not find container \"7e1fbbcc87cf618d2f661d067c8f959f91bf03ba9223ba2f251b180c3e2606f0\": container with ID starting with 7e1fbbcc87cf618d2f661d067c8f959f91bf03ba9223ba2f251b180c3e2606f0 not found: ID does not exist"
Sep 17 17:10:33.486739 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:10:33.486704    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d5q5qf"]
Sep 17 17:10:33.488660 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:10:33.488637    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d5q5qf"]
Sep 17 17:10:34.346293 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:10:34.346263    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71f1a84d-cc85-4301-8318-ab383bbc36d9" path="/var/lib/kubelet/pods/71f1a84d-cc85-4301-8318-ab383bbc36d9/volumes"
Sep 17 17:10:35.711171 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:10:35.711132    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx" podUID="91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 17 17:10:45.711576 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:10:45.711522    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx" podUID="91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 17 17:10:55.711614 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:10:55.711542    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx" podUID="91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 17 17:11:01.761226 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:01.761187    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9xt7m"]
Sep 17 17:11:01.761786 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:01.761761    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="054ab3b8-2b6b-4ad2-861e-be5d8591e137" containerName="storage-initializer"
Sep 17 17:11:01.761830 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:01.761792    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="054ab3b8-2b6b-4ad2-861e-be5d8591e137" containerName="storage-initializer"
Sep 17 17:11:01.761830 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:01.761810    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="054ab3b8-2b6b-4ad2-861e-be5d8591e137" containerName="main"
Sep 17 17:11:01.761830 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:01.761820    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="054ab3b8-2b6b-4ad2-861e-be5d8591e137" containerName="main"
Sep 17 17:11:01.761931 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:01.761851    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71f1a84d-cc85-4301-8318-ab383bbc36d9" containerName="main"
Sep 17 17:11:01.761931 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:01.761861    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="71f1a84d-cc85-4301-8318-ab383bbc36d9" containerName="main"
Sep 17 17:11:01.761994 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:01.761953    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="054ab3b8-2b6b-4ad2-861e-be5d8591e137" containerName="main"
Sep 17 17:11:01.761994 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:01.761965    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="71f1a84d-cc85-4301-8318-ab383bbc36d9" containerName="main"
Sep 17 17:11:01.770186 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:01.770162    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9xt7m"
Sep 17 17:11:01.773641 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:01.773618    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-fwg4d\""
Sep 17 17:11:01.773833 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:01.773811    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-d88dz\""
Sep 17 17:11:01.773948 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:01.773638    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 17 17:11:01.773948 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:01.773636    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 17 17:11:01.773948 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:01.773710    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 17 17:11:01.774638 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:01.774616    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9xt7m"]
Sep 17 17:11:01.822848 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:01.822812    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32cc4ea0-04c7-4567-a719-2178f6daa2b1-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9xt7m\" (UID: \"32cc4ea0-04c7-4567-a719-2178f6daa2b1\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9xt7m"
Sep 17 17:11:01.822998 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:01.822881    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj7tx\" (UniqueName: \"kubernetes.io/projected/32cc4ea0-04c7-4567-a719-2178f6daa2b1-kube-api-access-xj7tx\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9xt7m\" (UID: \"32cc4ea0-04c7-4567-a719-2178f6daa2b1\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9xt7m"
Sep 17 17:11:01.924116 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:01.924080    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj7tx\" (UniqueName: \"kubernetes.io/projected/32cc4ea0-04c7-4567-a719-2178f6daa2b1-kube-api-access-xj7tx\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9xt7m\" (UID: \"32cc4ea0-04c7-4567-a719-2178f6daa2b1\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9xt7m"
Sep 17 17:11:01.924297 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:01.924134    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32cc4ea0-04c7-4567-a719-2178f6daa2b1-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9xt7m\" (UID: \"32cc4ea0-04c7-4567-a719-2178f6daa2b1\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9xt7m"
Sep 17 17:11:01.926984 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:01.926961    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32cc4ea0-04c7-4567-a719-2178f6daa2b1-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9xt7m\" (UID: \"32cc4ea0-04c7-4567-a719-2178f6daa2b1\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9xt7m"
Sep 17 17:11:01.934958 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:01.934937    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj7tx\" (UniqueName: \"kubernetes.io/projected/32cc4ea0-04c7-4567-a719-2178f6daa2b1-kube-api-access-xj7tx\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9xt7m\" (UID: \"32cc4ea0-04c7-4567-a719-2178f6daa2b1\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9xt7m"
Sep 17 17:11:02.082378 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:02.082345    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9xt7m"
Sep 17 17:11:02.362907 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:02.362823    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9xt7m"]
Sep 17 17:11:02.367307 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:11:02.367280    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32cc4ea0_04c7_4567_a719_2178f6daa2b1.slice/crio-49f13b388033cd1ecd923a82b82e94d75c345cc33f262bdb43b73d40cb1fd036 WatchSource:0}: Error finding container 49f13b388033cd1ecd923a82b82e94d75c345cc33f262bdb43b73d40cb1fd036: Status 404 returned error can't find the container with id 49f13b388033cd1ecd923a82b82e94d75c345cc33f262bdb43b73d40cb1fd036
Sep 17 17:11:02.616320 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:02.616220    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9xt7m" event={"ID":"32cc4ea0-04c7-4567-a719-2178f6daa2b1","Type":"ContainerStarted","Data":"695bbbbdb30c3e9f88d48624cd62d609562315d0440b793ca36b66e1ea031e01"}
Sep 17 17:11:02.616320 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:02.616280    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9xt7m" event={"ID":"32cc4ea0-04c7-4567-a719-2178f6daa2b1","Type":"ContainerStarted","Data":"49f13b388033cd1ecd923a82b82e94d75c345cc33f262bdb43b73d40cb1fd036"}
Sep 17 17:11:02.616320 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:02.616300    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9xt7m"
Sep 17 17:11:02.643478 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:02.643398    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9xt7m" podStartSLOduration=1.6433840659999999 podStartE2EDuration="1.643384066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:11:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:11:02.641124632 +0000 UTC m=+1748.535603772" watchObservedRunningTime="2026-09-17 17:11:02.643384066 +0000 UTC m=+1748.537863205"
Sep 17 17:11:05.711126 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:05.711075    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx" podUID="91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 17 17:11:15.711571 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:15.711524    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx" podUID="91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 17 17:11:25.721223 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:25.721189    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx"
Sep 17 17:11:25.729122 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:25.729096    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx"
Sep 17 17:11:30.826781 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:30.826749    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx"]
Sep 17 17:11:33.622758 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:33.622725    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9xt7m"
Sep 17 17:11:35.133545 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:35.133465    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9xt7m"]
Sep 17 17:11:43.623499 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:43.623449    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9xt7m" podUID="32cc4ea0-04c7-4567-a719-2178f6daa2b1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:11:45.860141 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:45.860080    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx" podUID="91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91" containerName="main" containerID="cri-o://ccd88dfb172d55e7ed84734b2f42b93692d1cd7396154bbb0bcb116008220b33" gracePeriod=45
Sep 17 17:11:46.597215 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:46.597184    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx"
Sep 17 17:11:46.709318 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:46.709290    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-cabundle-cert\") pod \"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91\" (UID: \"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91\") "
Sep 17 17:11:46.709541 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:46.709325    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-home\") pod \"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91\" (UID: \"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91\") "
Sep 17 17:11:46.709541 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:46.709353    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-tmp-dir\") pod \"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91\" (UID: \"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91\") "
Sep 17 17:11:46.709541 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:46.709378    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-dshm\") pod \"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91\" (UID: \"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91\") "
Sep 17 17:11:46.709541 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:46.709406    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-kserve-provision-location\") pod \"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91\" (UID: \"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91\") "
Sep 17 17:11:46.709541 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:46.709515    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-model-cache\") pod \"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91\" (UID: \"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91\") "
Sep 17 17:11:46.709846 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:46.709546    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7t457\" (UniqueName: \"kubernetes.io/projected/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-kube-api-access-7t457\") pod \"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91\" (UID: \"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91\") "
Sep 17 17:11:46.709846 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:46.709591    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-tls-certs\") pod \"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91\" (UID: \"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91\") "
Sep 17 17:11:46.709846 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:46.709696    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91" (UID: "91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:11:46.710019 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:46.709991    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-home" (OuterVolumeSpecName: "home") pod "91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91" (UID: "91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:11:46.710128 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:46.710096    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-model-cache" (OuterVolumeSpecName: "model-cache") pod "91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91" (UID: "91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:11:46.710527 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:46.710405    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-cabundle-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:11:46.712168 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:46.712106    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-kube-api-access-7t457" (OuterVolumeSpecName: "kube-api-access-7t457") pod "91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91" (UID: "91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91"). InnerVolumeSpecName "kube-api-access-7t457". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:11:46.712290 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:46.712193    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-dshm" (OuterVolumeSpecName: "dshm") pod "91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91" (UID: "91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:11:46.712529 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:46.712507    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91" (UID: "91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:11:46.728869 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:46.728842    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91" (UID: "91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:11:46.738813 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:46.738786    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91" (UID: "91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:11:46.811471 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:46.811408    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-model-cache\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:11:46.811471 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:46.811471    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t457\" (UniqueName: \"kubernetes.io/projected/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-kube-api-access-7t457\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:11:46.811471 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:46.811482    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:11:46.811676 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:46.811495    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-home\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:11:46.811676 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:46.811507    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-tmp-dir\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:11:46.811676 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:46.811514    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-dshm\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:11:46.811676 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:46.811521    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91-kserve-provision-location\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:11:46.848385 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:46.848353    2566 generic.go:356] "Generic (PLEG): container finished" podID="91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91" containerID="ccd88dfb172d55e7ed84734b2f42b93692d1cd7396154bbb0bcb116008220b33" exitCode=0
Sep 17 17:11:46.848533 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:46.848412    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx"
Sep 17 17:11:46.848533 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:46.848453    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx" event={"ID":"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91","Type":"ContainerDied","Data":"ccd88dfb172d55e7ed84734b2f42b93692d1cd7396154bbb0bcb116008220b33"}
Sep 17 17:11:46.848606 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:46.848548    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx" event={"ID":"91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91","Type":"ContainerDied","Data":"f3aecd7a9ef775781c0fc82b807af01d7ec66bf1292fc27cac47655b311fcf3b"}
Sep 17 17:11:46.848606 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:46.848586    2566 scope.go:117] "RemoveContainer" containerID="ccd88dfb172d55e7ed84734b2f42b93692d1cd7396154bbb0bcb116008220b33"
Sep 17 17:11:46.857970 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:46.857956    2566 scope.go:117] "RemoveContainer" containerID="a51a6c9f23fd078da7b8aa1e11fd10a1366f28689fe149342ca89dda4edd61b1"
Sep 17 17:11:46.865681 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:46.865521    2566 scope.go:117] "RemoveContainer" containerID="ccd88dfb172d55e7ed84734b2f42b93692d1cd7396154bbb0bcb116008220b33"
Sep 17 17:11:46.865908 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:11:46.865762    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccd88dfb172d55e7ed84734b2f42b93692d1cd7396154bbb0bcb116008220b33\": container with ID starting with ccd88dfb172d55e7ed84734b2f42b93692d1cd7396154bbb0bcb116008220b33 not found: ID does not exist" containerID="ccd88dfb172d55e7ed84734b2f42b93692d1cd7396154bbb0bcb116008220b33"
Sep 17 17:11:46.865908 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:46.865783    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccd88dfb172d55e7ed84734b2f42b93692d1cd7396154bbb0bcb116008220b33"} err="failed to get container status \"ccd88dfb172d55e7ed84734b2f42b93692d1cd7396154bbb0bcb116008220b33\": rpc error: code = NotFound desc = could not find container \"ccd88dfb172d55e7ed84734b2f42b93692d1cd7396154bbb0bcb116008220b33\": container with ID starting with ccd88dfb172d55e7ed84734b2f42b93692d1cd7396154bbb0bcb116008220b33 not found: ID does not exist"
Sep 17 17:11:46.865908 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:46.865797    2566 scope.go:117] "RemoveContainer" containerID="a51a6c9f23fd078da7b8aa1e11fd10a1366f28689fe149342ca89dda4edd61b1"
Sep 17 17:11:46.866035 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:11:46.866011    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a51a6c9f23fd078da7b8aa1e11fd10a1366f28689fe149342ca89dda4edd61b1\": container with ID starting with a51a6c9f23fd078da7b8aa1e11fd10a1366f28689fe149342ca89dda4edd61b1 not found: ID does not exist" containerID="a51a6c9f23fd078da7b8aa1e11fd10a1366f28689fe149342ca89dda4edd61b1"
Sep 17 17:11:46.866035 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:46.866026    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a51a6c9f23fd078da7b8aa1e11fd10a1366f28689fe149342ca89dda4edd61b1"} err="failed to get container status \"a51a6c9f23fd078da7b8aa1e11fd10a1366f28689fe149342ca89dda4edd61b1\": rpc error: code = NotFound desc = could not find container \"a51a6c9f23fd078da7b8aa1e11fd10a1366f28689fe149342ca89dda4edd61b1\": container with ID starting with a51a6c9f23fd078da7b8aa1e11fd10a1366f28689fe149342ca89dda4edd61b1 not found: ID does not exist"
Sep 17 17:11:46.875928 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:46.875908    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx"]
Sep 17 17:11:46.882296 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:46.882274    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d5c585b7d-dp4dx"]
Sep 17 17:11:48.346621 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:48.346586    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91" path="/var/lib/kubelet/pods/91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91/volumes"
Sep 17 17:11:50.153312 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:50.153250    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9xt7m" podUID="32cc4ea0-04c7-4567-a719-2178f6daa2b1" containerName="main" containerID="cri-o://695bbbbdb30c3e9f88d48624cd62d609562315d0440b793ca36b66e1ea031e01" gracePeriod=45
Sep 17 17:11:53.622857 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:53.622822    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9xt7m" podUID="32cc4ea0-04c7-4567-a719-2178f6daa2b1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:11:56.413988 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:56.413945    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gblhr_91477834-6648-4e75-8530-e01bdee5e57e/ovn-acl-logging/0.log"
Sep 17 17:11:56.418270 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:11:56.418250    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gblhr_91477834-6648-4e75-8530-e01bdee5e57e/ovn-acl-logging/0.log"
Sep 17 17:12:03.623274 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:03.623235    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9xt7m" podUID="32cc4ea0-04c7-4567-a719-2178f6daa2b1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:12:03.623857 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:03.623370    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9xt7m"
Sep 17 17:12:13.622912 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:13.622868    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9xt7m" podUID="32cc4ea0-04c7-4567-a719-2178f6daa2b1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:12:20.404240 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:20.404214    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9xt7m"
Sep 17 17:12:20.471328 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:20.471241    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32cc4ea0-04c7-4567-a719-2178f6daa2b1-tls-certs\") pod \"32cc4ea0-04c7-4567-a719-2178f6daa2b1\" (UID: \"32cc4ea0-04c7-4567-a719-2178f6daa2b1\") "
Sep 17 17:12:20.471501 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:20.471328    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xj7tx\" (UniqueName: \"kubernetes.io/projected/32cc4ea0-04c7-4567-a719-2178f6daa2b1-kube-api-access-xj7tx\") pod \"32cc4ea0-04c7-4567-a719-2178f6daa2b1\" (UID: \"32cc4ea0-04c7-4567-a719-2178f6daa2b1\") "
Sep 17 17:12:20.473750 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:20.473718    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32cc4ea0-04c7-4567-a719-2178f6daa2b1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "32cc4ea0-04c7-4567-a719-2178f6daa2b1" (UID: "32cc4ea0-04c7-4567-a719-2178f6daa2b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:12:20.473849 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:20.473768    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32cc4ea0-04c7-4567-a719-2178f6daa2b1-kube-api-access-xj7tx" (OuterVolumeSpecName: "kube-api-access-xj7tx") pod "32cc4ea0-04c7-4567-a719-2178f6daa2b1" (UID: "32cc4ea0-04c7-4567-a719-2178f6daa2b1"). InnerVolumeSpecName "kube-api-access-xj7tx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:12:20.572326 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:20.572278    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32cc4ea0-04c7-4567-a719-2178f6daa2b1-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:12:20.572326 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:20.572320    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj7tx\" (UniqueName: \"kubernetes.io/projected/32cc4ea0-04c7-4567-a719-2178f6daa2b1-kube-api-access-xj7tx\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:12:21.025681 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.025642    2566 generic.go:356] "Generic (PLEG): container finished" podID="32cc4ea0-04c7-4567-a719-2178f6daa2b1" containerID="695bbbbdb30c3e9f88d48624cd62d609562315d0440b793ca36b66e1ea031e01" exitCode=0
Sep 17 17:12:21.025681 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.025690    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9xt7m"
Sep 17 17:12:21.025882 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.025727    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9xt7m" event={"ID":"32cc4ea0-04c7-4567-a719-2178f6daa2b1","Type":"ContainerDied","Data":"695bbbbdb30c3e9f88d48624cd62d609562315d0440b793ca36b66e1ea031e01"}
Sep 17 17:12:21.025882 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.025785    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9xt7m" event={"ID":"32cc4ea0-04c7-4567-a719-2178f6daa2b1","Type":"ContainerDied","Data":"49f13b388033cd1ecd923a82b82e94d75c345cc33f262bdb43b73d40cb1fd036"}
Sep 17 17:12:21.025882 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.025807    2566 scope.go:117] "RemoveContainer" containerID="695bbbbdb30c3e9f88d48624cd62d609562315d0440b793ca36b66e1ea031e01"
Sep 17 17:12:21.035502 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.035481    2566 scope.go:117] "RemoveContainer" containerID="695bbbbdb30c3e9f88d48624cd62d609562315d0440b793ca36b66e1ea031e01"
Sep 17 17:12:21.035808 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:12:21.035788    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"695bbbbdb30c3e9f88d48624cd62d609562315d0440b793ca36b66e1ea031e01\": container with ID starting with 695bbbbdb30c3e9f88d48624cd62d609562315d0440b793ca36b66e1ea031e01 not found: ID does not exist" containerID="695bbbbdb30c3e9f88d48624cd62d609562315d0440b793ca36b66e1ea031e01"
Sep 17 17:12:21.035869 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.035817    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"695bbbbdb30c3e9f88d48624cd62d609562315d0440b793ca36b66e1ea031e01"} err="failed to get container status \"695bbbbdb30c3e9f88d48624cd62d609562315d0440b793ca36b66e1ea031e01\": rpc error: code = NotFound desc = could not find container \"695bbbbdb30c3e9f88d48624cd62d609562315d0440b793ca36b66e1ea031e01\": container with ID starting with 695bbbbdb30c3e9f88d48624cd62d609562315d0440b793ca36b66e1ea031e01 not found: ID does not exist"
Sep 17 17:12:21.048721 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.048686    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9xt7m"]
Sep 17 17:12:21.050860 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.050835    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9xt7m"]
Sep 17 17:12:21.213362 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.213328    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"]
Sep 17 17:12:21.213707 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.213685    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32cc4ea0-04c7-4567-a719-2178f6daa2b1" containerName="main"
Sep 17 17:12:21.213707 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.213705    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="32cc4ea0-04c7-4567-a719-2178f6daa2b1" containerName="main"
Sep 17 17:12:21.213707 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.213723    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91" containerName="storage-initializer"
Sep 17 17:12:21.213968 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.213733    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91" containerName="storage-initializer"
Sep 17 17:12:21.213968 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.213749    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91" containerName="main"
Sep 17 17:12:21.213968 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.213756    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91" containerName="main"
Sep 17 17:12:21.213968 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.213858    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="91d3bcbd-ac1b-48c6-89c8-66f1ce71eb91" containerName="main"
Sep 17 17:12:21.213968 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.213868    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="32cc4ea0-04c7-4567-a719-2178f6daa2b1" containerName="main"
Sep 17 17:12:21.218051 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.218028    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"
Sep 17 17:12:21.220793 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.220768    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 17 17:12:21.220932 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.220804    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"router-gateway-1-openshift-default-dockercfg-h5rpb\""
Sep 17 17:12:21.221218 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.221197    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 17 17:12:21.221916 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.221899    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-gw-ca-root-cert\""
Sep 17 17:12:21.229478 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.229424    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"]
Sep 17 17:12:21.278162 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.278048    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4e5318-0856-4c42-aa9c-3ea030053f77-credential-socket\") pod \"router-gateway-1-openshift-default-8cd48b867-q86cw\" (UID: \"8d4e5318-0856-4c42-aa9c-3ea030053f77\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"
Sep 17 17:12:21.278162 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.278108    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d4e5318-0856-4c42-aa9c-3ea030053f77-istio-token\") pod \"router-gateway-1-openshift-default-8cd48b867-q86cw\" (UID: \"8d4e5318-0856-4c42-aa9c-3ea030053f77\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"
Sep 17 17:12:21.278162 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.278151    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d4e5318-0856-4c42-aa9c-3ea030053f77-istio-data\") pod \"router-gateway-1-openshift-default-8cd48b867-q86cw\" (UID: \"8d4e5318-0856-4c42-aa9c-3ea030053f77\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"
Sep 17 17:12:21.278469 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.278182    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d4e5318-0856-4c42-aa9c-3ea030053f77-istio-podinfo\") pod \"router-gateway-1-openshift-default-8cd48b867-q86cw\" (UID: \"8d4e5318-0856-4c42-aa9c-3ea030053f77\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"
Sep 17 17:12:21.278469 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.278249    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4e5318-0856-4c42-aa9c-3ea030053f77-workload-socket\") pod \"router-gateway-1-openshift-default-8cd48b867-q86cw\" (UID: \"8d4e5318-0856-4c42-aa9c-3ea030053f77\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"
Sep 17 17:12:21.278469 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.278291    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d4e5318-0856-4c42-aa9c-3ea030053f77-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-8cd48b867-q86cw\" (UID: \"8d4e5318-0856-4c42-aa9c-3ea030053f77\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"
Sep 17 17:12:21.278469 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.278315    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d4e5318-0856-4c42-aa9c-3ea030053f77-workload-certs\") pod \"router-gateway-1-openshift-default-8cd48b867-q86cw\" (UID: \"8d4e5318-0856-4c42-aa9c-3ea030053f77\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"
Sep 17 17:12:21.278469 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.278334    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtrx5\" (UniqueName: \"kubernetes.io/projected/8d4e5318-0856-4c42-aa9c-3ea030053f77-kube-api-access-jtrx5\") pod \"router-gateway-1-openshift-default-8cd48b867-q86cw\" (UID: \"8d4e5318-0856-4c42-aa9c-3ea030053f77\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"
Sep 17 17:12:21.278469 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.278361    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d4e5318-0856-4c42-aa9c-3ea030053f77-istio-envoy\") pod \"router-gateway-1-openshift-default-8cd48b867-q86cw\" (UID: \"8d4e5318-0856-4c42-aa9c-3ea030053f77\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"
Sep 17 17:12:21.379386 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.379347    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d4e5318-0856-4c42-aa9c-3ea030053f77-istio-token\") pod \"router-gateway-1-openshift-default-8cd48b867-q86cw\" (UID: \"8d4e5318-0856-4c42-aa9c-3ea030053f77\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"
Sep 17 17:12:21.379386 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.379385    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d4e5318-0856-4c42-aa9c-3ea030053f77-istio-data\") pod \"router-gateway-1-openshift-default-8cd48b867-q86cw\" (UID: \"8d4e5318-0856-4c42-aa9c-3ea030053f77\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"
Sep 17 17:12:21.379634 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.379408    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d4e5318-0856-4c42-aa9c-3ea030053f77-istio-podinfo\") pod \"router-gateway-1-openshift-default-8cd48b867-q86cw\" (UID: \"8d4e5318-0856-4c42-aa9c-3ea030053f77\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"
Sep 17 17:12:21.379634 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.379480    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4e5318-0856-4c42-aa9c-3ea030053f77-workload-socket\") pod \"router-gateway-1-openshift-default-8cd48b867-q86cw\" (UID: \"8d4e5318-0856-4c42-aa9c-3ea030053f77\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"
Sep 17 17:12:21.379634 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.379505    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d4e5318-0856-4c42-aa9c-3ea030053f77-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-8cd48b867-q86cw\" (UID: \"8d4e5318-0856-4c42-aa9c-3ea030053f77\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"
Sep 17 17:12:21.379634 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.379535    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d4e5318-0856-4c42-aa9c-3ea030053f77-workload-certs\") pod \"router-gateway-1-openshift-default-8cd48b867-q86cw\" (UID: \"8d4e5318-0856-4c42-aa9c-3ea030053f77\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"
Sep 17 17:12:21.379634 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.379562    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtrx5\" (UniqueName: \"kubernetes.io/projected/8d4e5318-0856-4c42-aa9c-3ea030053f77-kube-api-access-jtrx5\") pod \"router-gateway-1-openshift-default-8cd48b867-q86cw\" (UID: \"8d4e5318-0856-4c42-aa9c-3ea030053f77\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"
Sep 17 17:12:21.379904 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.379702    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d4e5318-0856-4c42-aa9c-3ea030053f77-istio-envoy\") pod \"router-gateway-1-openshift-default-8cd48b867-q86cw\" (UID: \"8d4e5318-0856-4c42-aa9c-3ea030053f77\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"
Sep 17 17:12:21.379904 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.379786    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4e5318-0856-4c42-aa9c-3ea030053f77-credential-socket\") pod \"router-gateway-1-openshift-default-8cd48b867-q86cw\" (UID: \"8d4e5318-0856-4c42-aa9c-3ea030053f77\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"
Sep 17 17:12:21.379904 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.379870    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d4e5318-0856-4c42-aa9c-3ea030053f77-istio-data\") pod \"router-gateway-1-openshift-default-8cd48b867-q86cw\" (UID: \"8d4e5318-0856-4c42-aa9c-3ea030053f77\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"
Sep 17 17:12:21.380046 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.379926    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4e5318-0856-4c42-aa9c-3ea030053f77-workload-socket\") pod \"router-gateway-1-openshift-default-8cd48b867-q86cw\" (UID: \"8d4e5318-0856-4c42-aa9c-3ea030053f77\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"
Sep 17 17:12:21.380046 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.379956    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d4e5318-0856-4c42-aa9c-3ea030053f77-workload-certs\") pod \"router-gateway-1-openshift-default-8cd48b867-q86cw\" (UID: \"8d4e5318-0856-4c42-aa9c-3ea030053f77\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"
Sep 17 17:12:21.380230 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.380191    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4e5318-0856-4c42-aa9c-3ea030053f77-credential-socket\") pod \"router-gateway-1-openshift-default-8cd48b867-q86cw\" (UID: \"8d4e5318-0856-4c42-aa9c-3ea030053f77\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"
Sep 17 17:12:21.380364 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.380288    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d4e5318-0856-4c42-aa9c-3ea030053f77-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-8cd48b867-q86cw\" (UID: \"8d4e5318-0856-4c42-aa9c-3ea030053f77\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"
Sep 17 17:12:21.382271 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.382250    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d4e5318-0856-4c42-aa9c-3ea030053f77-istio-envoy\") pod \"router-gateway-1-openshift-default-8cd48b867-q86cw\" (UID: \"8d4e5318-0856-4c42-aa9c-3ea030053f77\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"
Sep 17 17:12:21.382395 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.382379    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d4e5318-0856-4c42-aa9c-3ea030053f77-istio-podinfo\") pod \"router-gateway-1-openshift-default-8cd48b867-q86cw\" (UID: \"8d4e5318-0856-4c42-aa9c-3ea030053f77\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"
Sep 17 17:12:21.389793 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.389765    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtrx5\" (UniqueName: \"kubernetes.io/projected/8d4e5318-0856-4c42-aa9c-3ea030053f77-kube-api-access-jtrx5\") pod \"router-gateway-1-openshift-default-8cd48b867-q86cw\" (UID: \"8d4e5318-0856-4c42-aa9c-3ea030053f77\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"
Sep 17 17:12:21.389904 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.389766    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d4e5318-0856-4c42-aa9c-3ea030053f77-istio-token\") pod \"router-gateway-1-openshift-default-8cd48b867-q86cw\" (UID: \"8d4e5318-0856-4c42-aa9c-3ea030053f77\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"
Sep 17 17:12:21.530429 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.530347    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"
Sep 17 17:12:21.810631 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.810605    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"]
Sep 17 17:12:21.812613 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:12:21.812578    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d4e5318_0856_4c42_aa9c_3ea030053f77.slice/crio-ae9921e4b1efe1f6208e9d18c4107ab454644237d23ac23f74996a001eabf839 WatchSource:0}: Error finding container ae9921e4b1efe1f6208e9d18c4107ab454644237d23ac23f74996a001eabf839: Status 404 returned error can't find the container with id ae9921e4b1efe1f6208e9d18c4107ab454644237d23ac23f74996a001eabf839
Sep 17 17:12:21.817672 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.817631    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 17:12:21.817800 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.817713    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 17:12:21.817800 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:21.817745    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 17:12:22.033315 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:22.033276    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw" event={"ID":"8d4e5318-0856-4c42-aa9c-3ea030053f77","Type":"ContainerStarted","Data":"1934b651ff5d9a0bb266bcc0b7f4d22852c3d0e02ac65534ae04f40a05c970a5"}
Sep 17 17:12:22.033545 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:22.033345    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw" event={"ID":"8d4e5318-0856-4c42-aa9c-3ea030053f77","Type":"ContainerStarted","Data":"ae9921e4b1efe1f6208e9d18c4107ab454644237d23ac23f74996a001eabf839"}
Sep 17 17:12:22.056525 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:22.056474    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw" podStartSLOduration=1.056428757 podStartE2EDuration="1.056428757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:12:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:12:22.055326072 +0000 UTC m=+1827.949805223" watchObservedRunningTime="2026-09-17 17:12:22.056428757 +0000 UTC m=+1827.950907896"
Sep 17 17:12:22.346850 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:22.346805    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32cc4ea0-04c7-4567-a719-2178f6daa2b1" path="/var/lib/kubelet/pods/32cc4ea0-04c7-4567-a719-2178f6daa2b1/volumes"
Sep 17 17:12:22.531183 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:22.531148    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"
Sep 17 17:12:22.536186 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:22.536159    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"
Sep 17 17:12:23.039023 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:23.038980    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"
Sep 17 17:12:23.040078 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:23.040056    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"
Sep 17 17:12:39.080658 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:39.080601    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw" podUID="8d4e5318-0856-4c42-aa9c-3ea030053f77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 17:12:40.139682 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:40.139592    2566 generic.go:356] "Generic (PLEG): container finished" podID="8d4e5318-0856-4c42-aa9c-3ea030053f77" containerID="1934b651ff5d9a0bb266bcc0b7f4d22852c3d0e02ac65534ae04f40a05c970a5" exitCode=0
Sep 17 17:12:40.139682 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:40.139666    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw" event={"ID":"8d4e5318-0856-4c42-aa9c-3ea030053f77","Type":"ContainerDied","Data":"1934b651ff5d9a0bb266bcc0b7f4d22852c3d0e02ac65534ae04f40a05c970a5"}
Sep 17 17:12:40.140123 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:40.140059    2566 scope.go:117] "RemoveContainer" containerID="1934b651ff5d9a0bb266bcc0b7f4d22852c3d0e02ac65534ae04f40a05c970a5"
Sep 17 17:12:40.140969 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:40.140941    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 17:12:40.141020 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:40.141011    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 17:12:40.141056 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:40.141045    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 17:12:41.146921 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:41.146892    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw" event={"ID":"8d4e5318-0856-4c42-aa9c-3ea030053f77","Type":"ContainerStarted","Data":"9ef7720e1029c725751ee15b3f427b721aa12e74cf8b676a1ea2178b4c063ff9"}
Sep 17 17:12:41.530739 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:41.530641    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"
Sep 17 17:12:41.531048 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:41.531024    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw" podUID="8d4e5318-0856-4c42-aa9c-3ea030053f77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 17 17:12:42.531346 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:42.531305    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw" podUID="8d4e5318-0856-4c42-aa9c-3ea030053f77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 17 17:12:43.530946 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:43.530901    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw" podUID="8d4e5318-0856-4c42-aa9c-3ea030053f77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 17 17:12:44.531543 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:44.531496    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw" podUID="8d4e5318-0856-4c42-aa9c-3ea030053f77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 17 17:12:45.531481 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:45.531414    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw" podUID="8d4e5318-0856-4c42-aa9c-3ea030053f77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 17 17:12:46.535072 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:46.535031    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"
Sep 17 17:12:46.535504 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:46.535298    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"
Sep 17 17:12:46.536061 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:12:46.536047    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"
Sep 17 17:13:02.622721 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:02.622684    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49"]
Sep 17 17:13:02.627194 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:02.627176    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49"
Sep 17 17:13:02.630747 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:02.630723    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-vhr9x\""
Sep 17 17:13:02.630859 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:02.630751    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 17 17:13:02.630859 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:02.630779    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 17 17:13:02.631564 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:02.631542    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 17 17:13:02.646212 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:02.646186    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49"]
Sep 17 17:13:02.728697 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:02.728665    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzzd2\" (UniqueName: \"kubernetes.io/projected/69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e-kube-api-access-wzzd2\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49\" (UID: \"69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49"
Sep 17 17:13:02.728887 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:02.728707    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49\" (UID: \"69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49"
Sep 17 17:13:02.728887 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:02.728836    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49\" (UID: \"69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49"
Sep 17 17:13:02.728887 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:02.728881    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49\" (UID: \"69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49"
Sep 17 17:13:02.729027 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:02.728919    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49\" (UID: \"69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49"
Sep 17 17:13:02.729027 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:02.728941    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49\" (UID: \"69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49"
Sep 17 17:13:02.829738 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:02.829681    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49\" (UID: \"69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49"
Sep 17 17:13:02.829738 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:02.829736    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49\" (UID: \"69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49"
Sep 17 17:13:02.830026 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:02.829773    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49\" (UID: \"69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49"
Sep 17 17:13:02.830026 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:02.829798    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49\" (UID: \"69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49"
Sep 17 17:13:02.830026 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:02.829843    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzzd2\" (UniqueName: \"kubernetes.io/projected/69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e-kube-api-access-wzzd2\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49\" (UID: \"69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49"
Sep 17 17:13:02.830026 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:02.829872    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49\" (UID: \"69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49"
Sep 17 17:13:02.830345 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:02.830306    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49\" (UID: \"69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49"
Sep 17 17:13:02.830345 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:02.830325    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49\" (UID: \"69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49"
Sep 17 17:13:02.830758 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:02.830371    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49\" (UID: \"69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49"
Sep 17 17:13:02.832475 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:02.832450    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49\" (UID: \"69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49"
Sep 17 17:13:02.832667 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:02.832648    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49\" (UID: \"69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49"
Sep 17 17:13:02.844569 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:02.844539    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzzd2\" (UniqueName: \"kubernetes.io/projected/69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e-kube-api-access-wzzd2\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49\" (UID: \"69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49"
Sep 17 17:13:02.937814 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:02.937712    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49"
Sep 17 17:13:03.024835 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:03.024804    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schemsrg2"]
Sep 17 17:13:03.028339 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:03.028318    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schemsrg2"
Sep 17 17:13:03.031236 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:03.031212    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-5j8lk\""
Sep 17 17:13:03.054870 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:03.054841    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schemsrg2"]
Sep 17 17:13:03.132072 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:03.132037    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46721183-8dff-4551-b852-64f5e8bc8b19-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schemsrg2\" (UID: \"46721183-8dff-4551-b852-64f5e8bc8b19\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schemsrg2"
Sep 17 17:13:03.132264 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:03.132103    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xfm9\" (UniqueName: \"kubernetes.io/projected/46721183-8dff-4551-b852-64f5e8bc8b19-kube-api-access-8xfm9\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schemsrg2\" (UID: \"46721183-8dff-4551-b852-64f5e8bc8b19\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schemsrg2"
Sep 17 17:13:03.230808 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:03.230776    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49"]
Sep 17 17:13:03.232747 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:03.232726    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46721183-8dff-4551-b852-64f5e8bc8b19-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schemsrg2\" (UID: \"46721183-8dff-4551-b852-64f5e8bc8b19\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schemsrg2"
Sep 17 17:13:03.232860 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:03.232766    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xfm9\" (UniqueName: \"kubernetes.io/projected/46721183-8dff-4551-b852-64f5e8bc8b19-kube-api-access-8xfm9\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schemsrg2\" (UID: \"46721183-8dff-4551-b852-64f5e8bc8b19\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schemsrg2"
Sep 17 17:13:03.233467 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:13:03.233410    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69dcedb3_4f52_45f9_8d4e_e88fba8f2c8e.slice/crio-b4750479e11f15cde664fd019d952fa8dce510ebe85ea81f1a97500192b557b2 WatchSource:0}: Error finding container b4750479e11f15cde664fd019d952fa8dce510ebe85ea81f1a97500192b557b2: Status 404 returned error can't find the container with id b4750479e11f15cde664fd019d952fa8dce510ebe85ea81f1a97500192b557b2
Sep 17 17:13:03.235629 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:03.235609    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46721183-8dff-4551-b852-64f5e8bc8b19-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schemsrg2\" (UID: \"46721183-8dff-4551-b852-64f5e8bc8b19\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schemsrg2"
Sep 17 17:13:03.256850 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:03.256823    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49" event={"ID":"69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e","Type":"ContainerStarted","Data":"b4750479e11f15cde664fd019d952fa8dce510ebe85ea81f1a97500192b557b2"}
Sep 17 17:13:03.260519 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:03.260494    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xfm9\" (UniqueName: \"kubernetes.io/projected/46721183-8dff-4551-b852-64f5e8bc8b19-kube-api-access-8xfm9\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schemsrg2\" (UID: \"46721183-8dff-4551-b852-64f5e8bc8b19\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schemsrg2"
Sep 17 17:13:03.340130 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:03.340098    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schemsrg2"
Sep 17 17:13:03.633786 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:03.633763    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schemsrg2"]
Sep 17 17:13:03.635368 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:13:03.635335    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46721183_8dff_4551_b852_64f5e8bc8b19.slice/crio-5d6871038fa43631b925a9437749347ef7452d4ad3a7d2077e139c1588a4a569 WatchSource:0}: Error finding container 5d6871038fa43631b925a9437749347ef7452d4ad3a7d2077e139c1588a4a569: Status 404 returned error can't find the container with id 5d6871038fa43631b925a9437749347ef7452d4ad3a7d2077e139c1588a4a569
Sep 17 17:13:04.269905 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:04.269860    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49" event={"ID":"69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e","Type":"ContainerStarted","Data":"e2d7a011ad365b36cbfcde61c92a2f2c41cee44b502abd3827383b1570bdeac7"}
Sep 17 17:13:04.271348 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:04.271324    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schemsrg2" event={"ID":"46721183-8dff-4551-b852-64f5e8bc8b19","Type":"ContainerStarted","Data":"da5d524ceae4ea5ca7fdaf07eaf577b6740723dd36274b33c3cc6822044a1196"}
Sep 17 17:13:04.271500 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:04.271370    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schemsrg2" event={"ID":"46721183-8dff-4551-b852-64f5e8bc8b19","Type":"ContainerStarted","Data":"5d6871038fa43631b925a9437749347ef7452d4ad3a7d2077e139c1588a4a569"}
Sep 17 17:13:04.271500 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:04.271419    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schemsrg2"
Sep 17 17:13:04.300597 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:04.300533    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49" podStartSLOduration=2.30051736 podStartE2EDuration="2.30051736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:13:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:13:04.298883414 +0000 UTC m=+1870.193362567" watchObservedRunningTime="2026-09-17 17:13:04.30051736 +0000 UTC m=+1870.194996498"
Sep 17 17:13:04.323801 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:04.323745    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schemsrg2" podStartSLOduration=2.323728409 podStartE2EDuration="2.323728409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:13:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:13:04.321993359 +0000 UTC m=+1870.216472497" watchObservedRunningTime="2026-09-17 17:13:04.323728409 +0000 UTC m=+1870.218207548"
Sep 17 17:13:04.938263 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:04.938225    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49"
Sep 17 17:13:12.937982 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:12.937943    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49"
Sep 17 17:13:12.950884 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:12.950854    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49"
Sep 17 17:13:13.328191 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:13.328160    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49"
Sep 17 17:13:35.278083 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:13:35.278050    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schemsrg2"
Sep 17 17:14:16.267111 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:14:16.267075    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"]
Sep 17 17:14:16.267605 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:14:16.267267    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw" podUID="8d4e5318-0856-4c42-aa9c-3ea030053f77" containerName="istio-proxy" containerID="cri-o://9ef7720e1029c725751ee15b3f427b721aa12e74cf8b676a1ea2178b4c063ff9" gracePeriod=30
Sep 17 17:14:16.537609 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:14:16.537514    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw" podUID="8d4e5318-0856-4c42-aa9c-3ea030053f77" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:14:21.522146 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:14:21.522105    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"
Sep 17 17:14:21.596974 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:14:21.596941    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtrx5\" (UniqueName: \"kubernetes.io/projected/8d4e5318-0856-4c42-aa9c-3ea030053f77-kube-api-access-jtrx5\") pod \"8d4e5318-0856-4c42-aa9c-3ea030053f77\" (UID: \"8d4e5318-0856-4c42-aa9c-3ea030053f77\") "
Sep 17 17:14:21.597144 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:14:21.596983    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d4e5318-0856-4c42-aa9c-3ea030053f77-istio-token\") pod \"8d4e5318-0856-4c42-aa9c-3ea030053f77\" (UID: \"8d4e5318-0856-4c42-aa9c-3ea030053f77\") "
Sep 17 17:14:21.597144 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:14:21.597030    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d4e5318-0856-4c42-aa9c-3ea030053f77-istio-data\") pod \"8d4e5318-0856-4c42-aa9c-3ea030053f77\" (UID: \"8d4e5318-0856-4c42-aa9c-3ea030053f77\") "
Sep 17 17:14:21.597144 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:14:21.597056    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4e5318-0856-4c42-aa9c-3ea030053f77-workload-socket\") pod \"8d4e5318-0856-4c42-aa9c-3ea030053f77\" (UID: \"8d4e5318-0856-4c42-aa9c-3ea030053f77\") "
Sep 17 17:14:21.597144 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:14:21.597078    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d4e5318-0856-4c42-aa9c-3ea030053f77-workload-certs\") pod \"8d4e5318-0856-4c42-aa9c-3ea030053f77\" (UID: \"8d4e5318-0856-4c42-aa9c-3ea030053f77\") "
Sep 17 17:14:21.597144 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:14:21.597106    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d4e5318-0856-4c42-aa9c-3ea030053f77-istio-podinfo\") pod \"8d4e5318-0856-4c42-aa9c-3ea030053f77\" (UID: \"8d4e5318-0856-4c42-aa9c-3ea030053f77\") "
Sep 17 17:14:21.597144 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:14:21.597137    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4e5318-0856-4c42-aa9c-3ea030053f77-credential-socket\") pod \"8d4e5318-0856-4c42-aa9c-3ea030053f77\" (UID: \"8d4e5318-0856-4c42-aa9c-3ea030053f77\") "
Sep 17 17:14:21.597550 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:14:21.597172    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d4e5318-0856-4c42-aa9c-3ea030053f77-istio-envoy\") pod \"8d4e5318-0856-4c42-aa9c-3ea030053f77\" (UID: \"8d4e5318-0856-4c42-aa9c-3ea030053f77\") "
Sep 17 17:14:21.597550 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:14:21.597236    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d4e5318-0856-4c42-aa9c-3ea030053f77-istiod-ca-cert\") pod \"8d4e5318-0856-4c42-aa9c-3ea030053f77\" (UID: \"8d4e5318-0856-4c42-aa9c-3ea030053f77\") "
Sep 17 17:14:21.597550 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:14:21.597385    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d4e5318-0856-4c42-aa9c-3ea030053f77-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8d4e5318-0856-4c42-aa9c-3ea030053f77" (UID: "8d4e5318-0856-4c42-aa9c-3ea030053f77"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:14:21.597550 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:14:21.597395    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d4e5318-0856-4c42-aa9c-3ea030053f77-istio-data" (OuterVolumeSpecName: "istio-data") pod "8d4e5318-0856-4c42-aa9c-3ea030053f77" (UID: "8d4e5318-0856-4c42-aa9c-3ea030053f77"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:14:21.597771 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:14:21.597556    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d4e5318-0856-4c42-aa9c-3ea030053f77-istio-data\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:14:21.597771 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:14:21.597575    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4e5318-0856-4c42-aa9c-3ea030053f77-workload-socket\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:14:21.597771 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:14:21.597644    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d4e5318-0856-4c42-aa9c-3ea030053f77-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8d4e5318-0856-4c42-aa9c-3ea030053f77" (UID: "8d4e5318-0856-4c42-aa9c-3ea030053f77"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:14:21.597771 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:14:21.597739    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d4e5318-0856-4c42-aa9c-3ea030053f77-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8d4e5318-0856-4c42-aa9c-3ea030053f77" (UID: "8d4e5318-0856-4c42-aa9c-3ea030053f77"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:14:21.597984 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:14:21.597879    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d4e5318-0856-4c42-aa9c-3ea030053f77-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8d4e5318-0856-4c42-aa9c-3ea030053f77" (UID: "8d4e5318-0856-4c42-aa9c-3ea030053f77"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:14:21.600079 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:14:21.599984    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d4e5318-0856-4c42-aa9c-3ea030053f77-kube-api-access-jtrx5" (OuterVolumeSpecName: "kube-api-access-jtrx5") pod "8d4e5318-0856-4c42-aa9c-3ea030053f77" (UID: "8d4e5318-0856-4c42-aa9c-3ea030053f77"). InnerVolumeSpecName "kube-api-access-jtrx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:14:21.600079 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:14:21.599991    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d4e5318-0856-4c42-aa9c-3ea030053f77-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8d4e5318-0856-4c42-aa9c-3ea030053f77" (UID: "8d4e5318-0856-4c42-aa9c-3ea030053f77"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 17:14:21.600079 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:14:21.599981    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d4e5318-0856-4c42-aa9c-3ea030053f77-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8d4e5318-0856-4c42-aa9c-3ea030053f77" (UID: "8d4e5318-0856-4c42-aa9c-3ea030053f77"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:14:21.600079 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:14:21.600023    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d4e5318-0856-4c42-aa9c-3ea030053f77-istio-token" (OuterVolumeSpecName: "istio-token") pod "8d4e5318-0856-4c42-aa9c-3ea030053f77" (UID: "8d4e5318-0856-4c42-aa9c-3ea030053f77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:14:21.672067 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:14:21.672037    2566 generic.go:356] "Generic (PLEG): container finished" podID="8d4e5318-0856-4c42-aa9c-3ea030053f77" containerID="9ef7720e1029c725751ee15b3f427b721aa12e74cf8b676a1ea2178b4c063ff9" exitCode=0
Sep 17 17:14:21.672235 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:14:21.672094    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"
Sep 17 17:14:21.672235 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:14:21.672099    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw" event={"ID":"8d4e5318-0856-4c42-aa9c-3ea030053f77","Type":"ContainerDied","Data":"9ef7720e1029c725751ee15b3f427b721aa12e74cf8b676a1ea2178b4c063ff9"}
Sep 17 17:14:21.672235 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:14:21.672143    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw" event={"ID":"8d4e5318-0856-4c42-aa9c-3ea030053f77","Type":"ContainerDied","Data":"ae9921e4b1efe1f6208e9d18c4107ab454644237d23ac23f74996a001eabf839"}
Sep 17 17:14:21.672235 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:14:21.672163    2566 scope.go:117] "RemoveContainer" containerID="9ef7720e1029c725751ee15b3f427b721aa12e74cf8b676a1ea2178b4c063ff9"
Sep 17 17:14:21.681639 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:14:21.681621    2566 scope.go:117] "RemoveContainer" containerID="1934b651ff5d9a0bb266bcc0b7f4d22852c3d0e02ac65534ae04f40a05c970a5"
Sep 17 17:14:21.689810 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:14:21.689789    2566 scope.go:117] "RemoveContainer" containerID="9ef7720e1029c725751ee15b3f427b721aa12e74cf8b676a1ea2178b4c063ff9"
Sep 17 17:14:21.690070 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:14:21.690045    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ef7720e1029c725751ee15b3f427b721aa12e74cf8b676a1ea2178b4c063ff9\": container with ID starting with 9ef7720e1029c725751ee15b3f427b721aa12e74cf8b676a1ea2178b4c063ff9 not found: ID does not exist" containerID="9ef7720e1029c725751ee15b3f427b721aa12e74cf8b676a1ea2178b4c063ff9"
Sep 17 17:14:21.690134 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:14:21.690075    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ef7720e1029c725751ee15b3f427b721aa12e74cf8b676a1ea2178b4c063ff9"} err="failed to get container status \"9ef7720e1029c725751ee15b3f427b721aa12e74cf8b676a1ea2178b4c063ff9\": rpc error: code = NotFound desc = could not find container \"9ef7720e1029c725751ee15b3f427b721aa12e74cf8b676a1ea2178b4c063ff9\": container with ID starting with 9ef7720e1029c725751ee15b3f427b721aa12e74cf8b676a1ea2178b4c063ff9 not found: ID does not exist"
Sep 17 17:14:21.690134 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:14:21.690092    2566 scope.go:117] "RemoveContainer" containerID="1934b651ff5d9a0bb266bcc0b7f4d22852c3d0e02ac65534ae04f40a05c970a5"
Sep 17 17:14:21.690355 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:14:21.690340    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1934b651ff5d9a0bb266bcc0b7f4d22852c3d0e02ac65534ae04f40a05c970a5\": container with ID starting with 1934b651ff5d9a0bb266bcc0b7f4d22852c3d0e02ac65534ae04f40a05c970a5 not found: ID does not exist" containerID="1934b651ff5d9a0bb266bcc0b7f4d22852c3d0e02ac65534ae04f40a05c970a5"
Sep 17 17:14:21.690404 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:14:21.690357    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1934b651ff5d9a0bb266bcc0b7f4d22852c3d0e02ac65534ae04f40a05c970a5"} err="failed to get container status \"1934b651ff5d9a0bb266bcc0b7f4d22852c3d0e02ac65534ae04f40a05c970a5\": rpc error: code = NotFound desc = could not find container \"1934b651ff5d9a0bb266bcc0b7f4d22852c3d0e02ac65534ae04f40a05c970a5\": container with ID starting with 1934b651ff5d9a0bb266bcc0b7f4d22852c3d0e02ac65534ae04f40a05c970a5 not found: ID does not exist"
Sep 17 17:14:21.697347 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:14:21.697327    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"]
Sep 17 17:14:21.698222 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:14:21.698204    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4e5318-0856-4c42-aa9c-3ea030053f77-credential-socket\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:14:21.698222 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:14:21.698224    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d4e5318-0856-4c42-aa9c-3ea030053f77-istio-envoy\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:14:21.698383 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:14:21.698233    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d4e5318-0856-4c42-aa9c-3ea030053f77-istiod-ca-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:14:21.698383 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:14:21.698241    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtrx5\" (UniqueName: \"kubernetes.io/projected/8d4e5318-0856-4c42-aa9c-3ea030053f77-kube-api-access-jtrx5\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:14:21.698383 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:14:21.698249    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d4e5318-0856-4c42-aa9c-3ea030053f77-istio-token\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:14:21.698383 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:14:21.698257    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d4e5318-0856-4c42-aa9c-3ea030053f77-workload-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:14:21.698383 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:14:21.698265    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d4e5318-0856-4c42-aa9c-3ea030053f77-istio-podinfo\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:14:21.705559 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:14:21.705528    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-q86cw"]
Sep 17 17:14:22.350314 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:14:22.350275    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d4e5318-0856-4c42-aa9c-3ea030053f77" path="/var/lib/kubelet/pods/8d4e5318-0856-4c42-aa9c-3ea030053f77/volumes"
Sep 17 17:15:31.754445 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:15:31.754397    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49"]
Sep 17 17:15:31.758905 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:15:31.758866    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schemsrg2"]
Sep 17 17:15:31.765130 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:15:31.765093    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:15:31.765130 ip-10-0-141-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:15:31.765130 ip-10-0-141-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:15:31.765130 ip-10-0-141-66 kubenswrapper[2566]:  > containerID="e2d7a011ad365b36cbfcde61c92a2f2c41cee44b502abd3827383b1570bdeac7" cmd=["/bin/sleep","15"]
Sep 17 17:15:31.765381 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:15:31.765150    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49" message="" err=<
Sep 17 17:15:31.765381 ip-10-0-141-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:15:31.765381 ip-10-0-141-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:15:31.765381 ip-10-0-141-66 kubenswrapper[2566]:  >
Sep 17 17:15:31.765381 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:15:31.765161    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:15:31.765381 ip-10-0-141-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:15:31.765381 ip-10-0-141-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:15:31.765381 ip-10-0-141-66 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49" podUID="69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e" containerName="main" containerID="cri-o://e2d7a011ad365b36cbfcde61c92a2f2c41cee44b502abd3827383b1570bdeac7"
Sep 17 17:15:31.765381 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:15:31.765210    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49" podUID="69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e" containerName="main" containerID="cri-o://e2d7a011ad365b36cbfcde61c92a2f2c41cee44b502abd3827383b1570bdeac7" gracePeriod=60
Sep 17 17:15:32.037340 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:15:32.037308    2566 generic.go:356] "Generic (PLEG): container finished" podID="69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e" containerID="e2d7a011ad365b36cbfcde61c92a2f2c41cee44b502abd3827383b1570bdeac7" exitCode=0
Sep 17 17:15:32.037476 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:15:32.037351    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49" event={"ID":"69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e","Type":"ContainerDied","Data":"e2d7a011ad365b36cbfcde61c92a2f2c41cee44b502abd3827383b1570bdeac7"}
Sep 17 17:15:32.111981 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:15:32.111950    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49"
Sep 17 17:15:32.247902 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:15:32.247868    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e-tmp-dir\") pod \"69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e\" (UID: \"69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e\") "
Sep 17 17:15:32.247902 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:15:32.247910    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzzd2\" (UniqueName: \"kubernetes.io/projected/69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e-kube-api-access-wzzd2\") pod \"69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e\" (UID: \"69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e\") "
Sep 17 17:15:32.248122 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:15:32.247938    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e-tls-certs\") pod \"69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e\" (UID: \"69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e\") "
Sep 17 17:15:32.248122 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:15:32.247968    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e-home\") pod \"69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e\" (UID: \"69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e\") "
Sep 17 17:15:32.248122 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:15:32.247988    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e-model-cache\") pod \"69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e\" (UID: \"69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e\") "
Sep 17 17:15:32.248122 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:15:32.248041    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e-dshm\") pod \"69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e\" (UID: \"69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e\") "
Sep 17 17:15:32.248339 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:15:32.248193    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e" (UID: "69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:15:32.248339 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:15:32.248287    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e-tmp-dir\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:15:32.248339 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:15:32.248283    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e-model-cache" (OuterVolumeSpecName: "model-cache") pod "69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e" (UID: "69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:15:32.248339 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:15:32.248300    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e-home" (OuterVolumeSpecName: "home") pod "69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e" (UID: "69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:15:32.250367 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:15:32.250336    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e-dshm" (OuterVolumeSpecName: "dshm") pod "69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e" (UID: "69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:15:32.250512 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:15:32.250379    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e" (UID: "69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:15:32.250512 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:15:32.250400    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e-kube-api-access-wzzd2" (OuterVolumeSpecName: "kube-api-access-wzzd2") pod "69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e" (UID: "69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e"). InnerVolumeSpecName "kube-api-access-wzzd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:15:32.349270 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:15:32.349238    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzzd2\" (UniqueName: \"kubernetes.io/projected/69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e-kube-api-access-wzzd2\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:15:32.349417 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:15:32.349272    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:15:32.349417 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:15:32.349286    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e-home\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:15:32.349417 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:15:32.349299    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e-model-cache\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:15:32.349417 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:15:32.349311    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e-dshm\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:15:33.043996 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:15:33.043959    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49" event={"ID":"69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e","Type":"ContainerDied","Data":"b4750479e11f15cde664fd019d952fa8dce510ebe85ea81f1a97500192b557b2"}
Sep 17 17:15:33.044520 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:15:33.044036    2566 scope.go:117] "RemoveContainer" containerID="e2d7a011ad365b36cbfcde61c92a2f2c41cee44b502abd3827383b1570bdeac7"
Sep 17 17:15:33.044520 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:15:33.043973    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49"
Sep 17 17:15:33.062690 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:15:33.062662    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49"]
Sep 17 17:15:33.067507 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:15:33.067482    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wfd49"]
Sep 17 17:15:34.346739 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:15:34.346703    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e" path="/var/lib/kubelet/pods/69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e/volumes"
Sep 17 17:15:35.278463 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:15:35.278413    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schemsrg2" podUID="46721183-8dff-4551-b852-64f5e8bc8b19" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:15:45.278162 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:15:45.278125    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schemsrg2" podUID="46721183-8dff-4551-b852-64f5e8bc8b19" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:15:46.778229 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:15:46.778185    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schemsrg2" podUID="46721183-8dff-4551-b852-64f5e8bc8b19" containerName="main" containerID="cri-o://da5d524ceae4ea5ca7fdaf07eaf577b6740723dd36274b33c3cc6822044a1196" gracePeriod=45
Sep 17 17:15:55.278336 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:15:55.278295    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schemsrg2" podUID="46721183-8dff-4551-b852-64f5e8bc8b19" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:15:55.278844 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:15:55.278408    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schemsrg2"
Sep 17 17:16:05.278557 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:05.278467    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schemsrg2" podUID="46721183-8dff-4551-b852-64f5e8bc8b19" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:16:15.278521 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:15.278479    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schemsrg2" podUID="46721183-8dff-4551-b852-64f5e8bc8b19" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:16:17.023663 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:17.023637    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schemsrg2"
Sep 17 17:16:17.103501 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:17.103467    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46721183-8dff-4551-b852-64f5e8bc8b19-tls-certs\") pod \"46721183-8dff-4551-b852-64f5e8bc8b19\" (UID: \"46721183-8dff-4551-b852-64f5e8bc8b19\") "
Sep 17 17:16:17.103501 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:17.103505    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xfm9\" (UniqueName: \"kubernetes.io/projected/46721183-8dff-4551-b852-64f5e8bc8b19-kube-api-access-8xfm9\") pod \"46721183-8dff-4551-b852-64f5e8bc8b19\" (UID: \"46721183-8dff-4551-b852-64f5e8bc8b19\") "
Sep 17 17:16:17.105760 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:17.105725    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46721183-8dff-4551-b852-64f5e8bc8b19-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "46721183-8dff-4551-b852-64f5e8bc8b19" (UID: "46721183-8dff-4551-b852-64f5e8bc8b19"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:16:17.105760 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:17.105751    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46721183-8dff-4551-b852-64f5e8bc8b19-kube-api-access-8xfm9" (OuterVolumeSpecName: "kube-api-access-8xfm9") pod "46721183-8dff-4551-b852-64f5e8bc8b19" (UID: "46721183-8dff-4551-b852-64f5e8bc8b19"). InnerVolumeSpecName "kube-api-access-8xfm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:16:17.204586 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:17.204555    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46721183-8dff-4551-b852-64f5e8bc8b19-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:16:17.204586 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:17.204581    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xfm9\" (UniqueName: \"kubernetes.io/projected/46721183-8dff-4551-b852-64f5e8bc8b19-kube-api-access-8xfm9\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:16:17.259404 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:17.259370    2566 generic.go:356] "Generic (PLEG): container finished" podID="46721183-8dff-4551-b852-64f5e8bc8b19" containerID="da5d524ceae4ea5ca7fdaf07eaf577b6740723dd36274b33c3cc6822044a1196" exitCode=0
Sep 17 17:16:17.259566 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:17.259423    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schemsrg2"
Sep 17 17:16:17.259566 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:17.259422    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schemsrg2" event={"ID":"46721183-8dff-4551-b852-64f5e8bc8b19","Type":"ContainerDied","Data":"da5d524ceae4ea5ca7fdaf07eaf577b6740723dd36274b33c3cc6822044a1196"}
Sep 17 17:16:17.259687 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:17.259564    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schemsrg2" event={"ID":"46721183-8dff-4551-b852-64f5e8bc8b19","Type":"ContainerDied","Data":"5d6871038fa43631b925a9437749347ef7452d4ad3a7d2077e139c1588a4a569"}
Sep 17 17:16:17.259687 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:17.259598    2566 scope.go:117] "RemoveContainer" containerID="da5d524ceae4ea5ca7fdaf07eaf577b6740723dd36274b33c3cc6822044a1196"
Sep 17 17:16:17.268902 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:17.268885    2566 scope.go:117] "RemoveContainer" containerID="da5d524ceae4ea5ca7fdaf07eaf577b6740723dd36274b33c3cc6822044a1196"
Sep 17 17:16:17.269166 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:16:17.269145    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da5d524ceae4ea5ca7fdaf07eaf577b6740723dd36274b33c3cc6822044a1196\": container with ID starting with da5d524ceae4ea5ca7fdaf07eaf577b6740723dd36274b33c3cc6822044a1196 not found: ID does not exist" containerID="da5d524ceae4ea5ca7fdaf07eaf577b6740723dd36274b33c3cc6822044a1196"
Sep 17 17:16:17.269231 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:17.269178    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da5d524ceae4ea5ca7fdaf07eaf577b6740723dd36274b33c3cc6822044a1196"} err="failed to get container status \"da5d524ceae4ea5ca7fdaf07eaf577b6740723dd36274b33c3cc6822044a1196\": rpc error: code = NotFound desc = could not find container \"da5d524ceae4ea5ca7fdaf07eaf577b6740723dd36274b33c3cc6822044a1196\": container with ID starting with da5d524ceae4ea5ca7fdaf07eaf577b6740723dd36274b33c3cc6822044a1196 not found: ID does not exist"
Sep 17 17:16:17.287895 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:17.287871    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schemsrg2"]
Sep 17 17:16:17.297088 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:17.297064    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schemsrg2"]
Sep 17 17:16:18.345996 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:18.345953    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46721183-8dff-4551-b852-64f5e8bc8b19" path="/var/lib/kubelet/pods/46721183-8dff-4551-b852-64f5e8bc8b19/volumes"
Sep 17 17:16:56.445312 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:56.445284    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gblhr_91477834-6648-4e75-8530-e01bdee5e57e/ovn-acl-logging/0.log"
Sep 17 17:16:56.450991 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:56.450968    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gblhr_91477834-6648-4e75-8530-e01bdee5e57e/ovn-acl-logging/0.log"
Sep 17 17:16:58.474684 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:58.474654    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9"]
Sep 17 17:16:58.475123 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:58.474986    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d4e5318-0856-4c42-aa9c-3ea030053f77" containerName="istio-proxy"
Sep 17 17:16:58.475123 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:58.474999    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d4e5318-0856-4c42-aa9c-3ea030053f77" containerName="istio-proxy"
Sep 17 17:16:58.475123 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:58.475015    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e" containerName="main"
Sep 17 17:16:58.475123 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:58.475022    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e" containerName="main"
Sep 17 17:16:58.475123 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:58.475033    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d4e5318-0856-4c42-aa9c-3ea030053f77" containerName="istio-proxy"
Sep 17 17:16:58.475123 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:58.475038    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d4e5318-0856-4c42-aa9c-3ea030053f77" containerName="istio-proxy"
Sep 17 17:16:58.475123 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:58.475045    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46721183-8dff-4551-b852-64f5e8bc8b19" containerName="main"
Sep 17 17:16:58.475123 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:58.475050    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="46721183-8dff-4551-b852-64f5e8bc8b19" containerName="main"
Sep 17 17:16:58.475123 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:58.475105    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="46721183-8dff-4551-b852-64f5e8bc8b19" containerName="main"
Sep 17 17:16:58.475123 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:58.475116    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="69dcedb3-4f52-45f9-8d4e-e88fba8f2c8e" containerName="main"
Sep 17 17:16:58.475123 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:58.475122    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d4e5318-0856-4c42-aa9c-3ea030053f77" containerName="istio-proxy"
Sep 17 17:16:58.475123 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:58.475129    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d4e5318-0856-4c42-aa9c-3ea030053f77" containerName="istio-proxy"
Sep 17 17:16:58.479175 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:58.479157    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9"
Sep 17 17:16:58.481657 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:58.481636    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 17 17:16:58.481921 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:58.481901    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 17 17:16:58.482738 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:58.482724    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 17 17:16:58.482778 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:58.482729    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-24xsr\""
Sep 17 17:16:58.490575 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:58.490549    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9"]
Sep 17 17:16:58.620494 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:58.620462    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b56469b9-022f-4f50-91d1-b4db7e41dcc6-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9\" (UID: \"b56469b9-022f-4f50-91d1-b4db7e41dcc6\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9"
Sep 17 17:16:58.620679 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:58.620502    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b56469b9-022f-4f50-91d1-b4db7e41dcc6-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9\" (UID: \"b56469b9-022f-4f50-91d1-b4db7e41dcc6\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9"
Sep 17 17:16:58.620679 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:58.620531    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-967vr\" (UniqueName: \"kubernetes.io/projected/b56469b9-022f-4f50-91d1-b4db7e41dcc6-kube-api-access-967vr\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9\" (UID: \"b56469b9-022f-4f50-91d1-b4db7e41dcc6\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9"
Sep 17 17:16:58.620679 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:58.620593    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b56469b9-022f-4f50-91d1-b4db7e41dcc6-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9\" (UID: \"b56469b9-022f-4f50-91d1-b4db7e41dcc6\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9"
Sep 17 17:16:58.620679 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:58.620646    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b56469b9-022f-4f50-91d1-b4db7e41dcc6-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9\" (UID: \"b56469b9-022f-4f50-91d1-b4db7e41dcc6\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9"
Sep 17 17:16:58.620824 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:58.620690    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b56469b9-022f-4f50-91d1-b4db7e41dcc6-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9\" (UID: \"b56469b9-022f-4f50-91d1-b4db7e41dcc6\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9"
Sep 17 17:16:58.722056 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:58.722016    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b56469b9-022f-4f50-91d1-b4db7e41dcc6-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9\" (UID: \"b56469b9-022f-4f50-91d1-b4db7e41dcc6\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9"
Sep 17 17:16:58.722056 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:58.722061    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b56469b9-022f-4f50-91d1-b4db7e41dcc6-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9\" (UID: \"b56469b9-022f-4f50-91d1-b4db7e41dcc6\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9"
Sep 17 17:16:58.722272 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:58.722083    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-967vr\" (UniqueName: \"kubernetes.io/projected/b56469b9-022f-4f50-91d1-b4db7e41dcc6-kube-api-access-967vr\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9\" (UID: \"b56469b9-022f-4f50-91d1-b4db7e41dcc6\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9"
Sep 17 17:16:58.722310 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:58.722268    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b56469b9-022f-4f50-91d1-b4db7e41dcc6-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9\" (UID: \"b56469b9-022f-4f50-91d1-b4db7e41dcc6\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9"
Sep 17 17:16:58.722310 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:58.722304    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b56469b9-022f-4f50-91d1-b4db7e41dcc6-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9\" (UID: \"b56469b9-022f-4f50-91d1-b4db7e41dcc6\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9"
Sep 17 17:16:58.722395 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:58.722353    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b56469b9-022f-4f50-91d1-b4db7e41dcc6-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9\" (UID: \"b56469b9-022f-4f50-91d1-b4db7e41dcc6\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9"
Sep 17 17:16:58.722694 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:58.722660    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b56469b9-022f-4f50-91d1-b4db7e41dcc6-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9\" (UID: \"b56469b9-022f-4f50-91d1-b4db7e41dcc6\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9"
Sep 17 17:16:58.722694 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:58.722684    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b56469b9-022f-4f50-91d1-b4db7e41dcc6-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9\" (UID: \"b56469b9-022f-4f50-91d1-b4db7e41dcc6\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9"
Sep 17 17:16:58.722857 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:58.722767    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b56469b9-022f-4f50-91d1-b4db7e41dcc6-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9\" (UID: \"b56469b9-022f-4f50-91d1-b4db7e41dcc6\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9"
Sep 17 17:16:58.724622 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:58.724604    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b56469b9-022f-4f50-91d1-b4db7e41dcc6-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9\" (UID: \"b56469b9-022f-4f50-91d1-b4db7e41dcc6\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9"
Sep 17 17:16:58.724933 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:58.724889    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b56469b9-022f-4f50-91d1-b4db7e41dcc6-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9\" (UID: \"b56469b9-022f-4f50-91d1-b4db7e41dcc6\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9"
Sep 17 17:16:58.731822 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:58.731797    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-967vr\" (UniqueName: \"kubernetes.io/projected/b56469b9-022f-4f50-91d1-b4db7e41dcc6-kube-api-access-967vr\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9\" (UID: \"b56469b9-022f-4f50-91d1-b4db7e41dcc6\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9"
Sep 17 17:16:58.792353 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:58.792309    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9"
Sep 17 17:16:58.814901 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:58.814873    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5nmcw"]
Sep 17 17:16:58.818809 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:58.818787    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5nmcw"
Sep 17 17:16:58.821685 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:58.821662    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-7jh7t\""
Sep 17 17:16:58.830591 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:58.830563    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5nmcw"]
Sep 17 17:16:58.924153 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:58.924120    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mw25\" (UniqueName: \"kubernetes.io/projected/0027b824-480c-427b-931e-c12bd93b6a0a-kube-api-access-2mw25\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5nmcw\" (UID: \"0027b824-480c-427b-931e-c12bd93b6a0a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5nmcw"
Sep 17 17:16:58.924315 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:58.924177    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0027b824-480c-427b-931e-c12bd93b6a0a-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5nmcw\" (UID: \"0027b824-480c-427b-931e-c12bd93b6a0a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5nmcw"
Sep 17 17:16:59.025692 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:59.025613    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0027b824-480c-427b-931e-c12bd93b6a0a-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5nmcw\" (UID: \"0027b824-480c-427b-931e-c12bd93b6a0a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5nmcw"
Sep 17 17:16:59.025692 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:59.025675    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mw25\" (UniqueName: \"kubernetes.io/projected/0027b824-480c-427b-931e-c12bd93b6a0a-kube-api-access-2mw25\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5nmcw\" (UID: \"0027b824-480c-427b-931e-c12bd93b6a0a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5nmcw"
Sep 17 17:16:59.028272 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:59.028243    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0027b824-480c-427b-931e-c12bd93b6a0a-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5nmcw\" (UID: \"0027b824-480c-427b-931e-c12bd93b6a0a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5nmcw"
Sep 17 17:16:59.035035 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:59.035012    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mw25\" (UniqueName: \"kubernetes.io/projected/0027b824-480c-427b-931e-c12bd93b6a0a-kube-api-access-2mw25\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5nmcw\" (UID: \"0027b824-480c-427b-931e-c12bd93b6a0a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5nmcw"
Sep 17 17:16:59.080719 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:59.080696    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9"]
Sep 17 17:16:59.082693 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:16:59.082666    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb56469b9_022f_4f50_91d1_b4db7e41dcc6.slice/crio-bd5d456492e9645ba2829cfcf02f608c5b895290e805bdc2c073c7de3a6bce70 WatchSource:0}: Error finding container bd5d456492e9645ba2829cfcf02f608c5b895290e805bdc2c073c7de3a6bce70: Status 404 returned error can't find the container with id bd5d456492e9645ba2829cfcf02f608c5b895290e805bdc2c073c7de3a6bce70
Sep 17 17:16:59.132816 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:59.132793    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5nmcw"
Sep 17 17:16:59.410145 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:59.410119    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5nmcw"]
Sep 17 17:16:59.411964 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:16:59.411938    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0027b824_480c_427b_931e_c12bd93b6a0a.slice/crio-6a352904f8fc6aceb2a72483b6667245edfb0fad1041165cf09d55d9d82cf6f8 WatchSource:0}: Error finding container 6a352904f8fc6aceb2a72483b6667245edfb0fad1041165cf09d55d9d82cf6f8: Status 404 returned error can't find the container with id 6a352904f8fc6aceb2a72483b6667245edfb0fad1041165cf09d55d9d82cf6f8
Sep 17 17:16:59.464636 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:59.464552    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9" event={"ID":"b56469b9-022f-4f50-91d1-b4db7e41dcc6","Type":"ContainerStarted","Data":"9e23b07717f5dc2c2f406b5ae5dd944b31aaaa315c570d28a16e3ef2fd837620"}
Sep 17 17:16:59.464636 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:59.464620    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9" event={"ID":"b56469b9-022f-4f50-91d1-b4db7e41dcc6","Type":"ContainerStarted","Data":"bd5d456492e9645ba2829cfcf02f608c5b895290e805bdc2c073c7de3a6bce70"}
Sep 17 17:16:59.466426 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:59.466404    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5nmcw" event={"ID":"0027b824-480c-427b-931e-c12bd93b6a0a","Type":"ContainerStarted","Data":"6a352904f8fc6aceb2a72483b6667245edfb0fad1041165cf09d55d9d82cf6f8"}
Sep 17 17:16:59.485295 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:59.485243    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9" podStartSLOduration=1.485228636 podStartE2EDuration="1.485228636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:16:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:16:59.483868934 +0000 UTC m=+2105.378348074" watchObservedRunningTime="2026-09-17 17:16:59.485228636 +0000 UTC m=+2105.379707775"
Sep 17 17:16:59.793255 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:16:59.793232    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9"
Sep 17 17:17:00.473477 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:17:00.473419    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5nmcw" event={"ID":"0027b824-480c-427b-931e-c12bd93b6a0a","Type":"ContainerStarted","Data":"2429ecb6370899805fb53a8401fc8dbf756960667a0f50da7d62e31dd2227bdb"}
Sep 17 17:17:00.473728 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:17:00.473710    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5nmcw"
Sep 17 17:17:00.493336 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:17:00.493285    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5nmcw" podStartSLOduration=2.493267175 podStartE2EDuration="2.493267175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:16:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:17:00.491163589 +0000 UTC m=+2106.385642729" watchObservedRunningTime="2026-09-17 17:17:00.493267175 +0000 UTC m=+2106.387746314"
Sep 17 17:17:08.792757 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:17:08.792703    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9"
Sep 17 17:17:08.804863 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:17:08.804825    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9"
Sep 17 17:17:09.532896 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:17:09.532867    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9"
Sep 17 17:17:31.480122 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:17:31.480089    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5nmcw"
Sep 17 17:17:36.811719 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:17:36.811633    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezmlm9"]
Sep 17 17:17:36.814847 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:17:36.814826    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezmlm9"
Sep 17 17:17:36.818116 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:17:36.818095    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-pkrjv\""
Sep 17 17:17:36.818219 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:17:36.818168    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 17 17:17:36.820554 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:17:36.820529    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/328562e9-0fa4-49b8-bcb2-35ffc6c20e5c-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezmlm9\" (UID: \"328562e9-0fa4-49b8-bcb2-35ffc6c20e5c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezmlm9"
Sep 17 17:17:36.820657 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:17:36.820622    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9svt\" (UniqueName: \"kubernetes.io/projected/328562e9-0fa4-49b8-bcb2-35ffc6c20e5c-kube-api-access-v9svt\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezmlm9\" (UID: \"328562e9-0fa4-49b8-bcb2-35ffc6c20e5c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezmlm9"
Sep 17 17:17:36.824952 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:17:36.824925    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezmlm9"]
Sep 17 17:17:36.921724 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:17:36.921690    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9svt\" (UniqueName: \"kubernetes.io/projected/328562e9-0fa4-49b8-bcb2-35ffc6c20e5c-kube-api-access-v9svt\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezmlm9\" (UID: \"328562e9-0fa4-49b8-bcb2-35ffc6c20e5c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezmlm9"
Sep 17 17:17:36.921724 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:17:36.921730    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/328562e9-0fa4-49b8-bcb2-35ffc6c20e5c-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezmlm9\" (UID: \"328562e9-0fa4-49b8-bcb2-35ffc6c20e5c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezmlm9"
Sep 17 17:17:36.924288 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:17:36.924261    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/328562e9-0fa4-49b8-bcb2-35ffc6c20e5c-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezmlm9\" (UID: \"328562e9-0fa4-49b8-bcb2-35ffc6c20e5c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezmlm9"
Sep 17 17:17:36.931523 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:17:36.931493    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9svt\" (UniqueName: \"kubernetes.io/projected/328562e9-0fa4-49b8-bcb2-35ffc6c20e5c-kube-api-access-v9svt\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezmlm9\" (UID: \"328562e9-0fa4-49b8-bcb2-35ffc6c20e5c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezmlm9"
Sep 17 17:17:37.128693 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:17:37.128605    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezmlm9"
Sep 17 17:17:37.412370 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:17:37.412258    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezmlm9"]
Sep 17 17:17:37.415297 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:17:37.415271    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod328562e9_0fa4_49b8_bcb2_35ffc6c20e5c.slice/crio-90665dbba06213a4cbd15b458c54a271c4e8f54eddfb30269b69fc7d71f1c4dd WatchSource:0}: Error finding container 90665dbba06213a4cbd15b458c54a271c4e8f54eddfb30269b69fc7d71f1c4dd: Status 404 returned error can't find the container with id 90665dbba06213a4cbd15b458c54a271c4e8f54eddfb30269b69fc7d71f1c4dd
Sep 17 17:17:37.668012 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:17:37.667908    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezmlm9" event={"ID":"328562e9-0fa4-49b8-bcb2-35ffc6c20e5c","Type":"ContainerStarted","Data":"d72d1490f38d3c4803b64d82b2b5c99d3db6e179bc0c2b707cc28220a663fc2c"}
Sep 17 17:17:37.668012 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:17:37.667969    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezmlm9" event={"ID":"328562e9-0fa4-49b8-bcb2-35ffc6c20e5c","Type":"ContainerStarted","Data":"90665dbba06213a4cbd15b458c54a271c4e8f54eddfb30269b69fc7d71f1c4dd"}
Sep 17 17:17:37.668012 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:17:37.667997    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezmlm9"
Sep 17 17:17:37.689336 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:17:37.689290    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezmlm9" podStartSLOduration=1.689274468 podStartE2EDuration="1.689274468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:17:37.688900801 +0000 UTC m=+2143.583379943" watchObservedRunningTime="2026-09-17 17:17:37.689274468 +0000 UTC m=+2143.583753607"
Sep 17 17:18:08.675376 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:18:08.675334    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezmlm9"
Sep 17 17:18:33.504646 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:18:33.504607    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezmlm9"]
Sep 17 17:18:38.675378 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:18:38.675337    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezmlm9" podUID="328562e9-0fa4-49b8-bcb2-35ffc6c20e5c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:18:48.532894 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:18:48.532829    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezmlm9" podUID="328562e9-0fa4-49b8-bcb2-35ffc6c20e5c" containerName="main" containerID="cri-o://d72d1490f38d3c4803b64d82b2b5c99d3db6e179bc0c2b707cc28220a663fc2c" gracePeriod=45
Sep 17 17:18:48.675147 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:18:48.675111    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezmlm9" podUID="328562e9-0fa4-49b8-bcb2-35ffc6c20e5c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:18:58.674566 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:18:58.674530    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezmlm9" podUID="328562e9-0fa4-49b8-bcb2-35ffc6c20e5c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:18:58.675120 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:18:58.674645    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezmlm9"
Sep 17 17:19:08.675346 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:08.675254    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezmlm9" podUID="328562e9-0fa4-49b8-bcb2-35ffc6c20e5c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:19:18.674283 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:19:18.674252    2566 logging.go:55] [core] [Channel #466 SubChannel #467]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.61:9003", ServerName: "10.133.0.61:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.61:9003: connect: connection refused"
Sep 17 17:19:18.764079 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:18.764053    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezmlm9"
Sep 17 17:19:18.819512 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:18.819473    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9svt\" (UniqueName: \"kubernetes.io/projected/328562e9-0fa4-49b8-bcb2-35ffc6c20e5c-kube-api-access-v9svt\") pod \"328562e9-0fa4-49b8-bcb2-35ffc6c20e5c\" (UID: \"328562e9-0fa4-49b8-bcb2-35ffc6c20e5c\") "
Sep 17 17:19:18.819707 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:18.819536    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/328562e9-0fa4-49b8-bcb2-35ffc6c20e5c-tls-certs\") pod \"328562e9-0fa4-49b8-bcb2-35ffc6c20e5c\" (UID: \"328562e9-0fa4-49b8-bcb2-35ffc6c20e5c\") "
Sep 17 17:19:18.821924 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:18.821898    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/328562e9-0fa4-49b8-bcb2-35ffc6c20e5c-kube-api-access-v9svt" (OuterVolumeSpecName: "kube-api-access-v9svt") pod "328562e9-0fa4-49b8-bcb2-35ffc6c20e5c" (UID: "328562e9-0fa4-49b8-bcb2-35ffc6c20e5c"). InnerVolumeSpecName "kube-api-access-v9svt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:19:18.822046 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:18.821970    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/328562e9-0fa4-49b8-bcb2-35ffc6c20e5c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "328562e9-0fa4-49b8-bcb2-35ffc6c20e5c" (UID: "328562e9-0fa4-49b8-bcb2-35ffc6c20e5c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:19:18.920341 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:18.920307    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/328562e9-0fa4-49b8-bcb2-35ffc6c20e5c-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:19:18.920341 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:18.920341    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9svt\" (UniqueName: \"kubernetes.io/projected/328562e9-0fa4-49b8-bcb2-35ffc6c20e5c-kube-api-access-v9svt\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:19:19.190084 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:19.189988    2566 generic.go:356] "Generic (PLEG): container finished" podID="328562e9-0fa4-49b8-bcb2-35ffc6c20e5c" containerID="d72d1490f38d3c4803b64d82b2b5c99d3db6e179bc0c2b707cc28220a663fc2c" exitCode=0
Sep 17 17:19:19.190084 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:19.190031    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezmlm9"
Sep 17 17:19:19.190084 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:19.190062    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezmlm9" event={"ID":"328562e9-0fa4-49b8-bcb2-35ffc6c20e5c","Type":"ContainerDied","Data":"d72d1490f38d3c4803b64d82b2b5c99d3db6e179bc0c2b707cc28220a663fc2c"}
Sep 17 17:19:19.190319 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:19.190109    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezmlm9" event={"ID":"328562e9-0fa4-49b8-bcb2-35ffc6c20e5c","Type":"ContainerDied","Data":"90665dbba06213a4cbd15b458c54a271c4e8f54eddfb30269b69fc7d71f1c4dd"}
Sep 17 17:19:19.190319 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:19.190134    2566 scope.go:117] "RemoveContainer" containerID="d72d1490f38d3c4803b64d82b2b5c99d3db6e179bc0c2b707cc28220a663fc2c"
Sep 17 17:19:19.206190 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:19.206168    2566 scope.go:117] "RemoveContainer" containerID="d72d1490f38d3c4803b64d82b2b5c99d3db6e179bc0c2b707cc28220a663fc2c"
Sep 17 17:19:19.206552 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:19:19.206531    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d72d1490f38d3c4803b64d82b2b5c99d3db6e179bc0c2b707cc28220a663fc2c\": container with ID starting with d72d1490f38d3c4803b64d82b2b5c99d3db6e179bc0c2b707cc28220a663fc2c not found: ID does not exist" containerID="d72d1490f38d3c4803b64d82b2b5c99d3db6e179bc0c2b707cc28220a663fc2c"
Sep 17 17:19:19.206625 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:19.206561    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d72d1490f38d3c4803b64d82b2b5c99d3db6e179bc0c2b707cc28220a663fc2c"} err="failed to get container status \"d72d1490f38d3c4803b64d82b2b5c99d3db6e179bc0c2b707cc28220a663fc2c\": rpc error: code = NotFound desc = could not find container \"d72d1490f38d3c4803b64d82b2b5c99d3db6e179bc0c2b707cc28220a663fc2c\": container with ID starting with d72d1490f38d3c4803b64d82b2b5c99d3db6e179bc0c2b707cc28220a663fc2c not found: ID does not exist"
Sep 17 17:19:19.217950 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:19.217918    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezmlm9"]
Sep 17 17:19:19.225135 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:19.225109    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezmlm9"]
Sep 17 17:19:19.674418 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:19.674371    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schezmlm9" podUID="328562e9-0fa4-49b8-bcb2-35ffc6c20e5c" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.133.0.61:9003\" within 1s: context deadline exceeded"
Sep 17 17:19:20.346716 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:20.346681    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="328562e9-0fa4-49b8-bcb2-35ffc6c20e5c" path="/var/lib/kubelet/pods/328562e9-0fa4-49b8-bcb2-35ffc6c20e5c/volumes"
Sep 17 17:19:21.043167 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:21.043132    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5nmcw"]
Sep 17 17:19:21.047755 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:21.047727    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9"]
Sep 17 17:19:21.056778 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:19:21.056734    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:19:21.056778 ip-10-0-141-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:19:21.056778 ip-10-0-141-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:19:21.056778 ip-10-0-141-66 kubenswrapper[2566]:  > containerID="9e23b07717f5dc2c2f406b5ae5dd944b31aaaa315c570d28a16e3ef2fd837620" cmd=["/bin/sleep","15"]
Sep 17 17:19:21.057026 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:21.056803    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9" message="" err=<
Sep 17 17:19:21.057026 ip-10-0-141-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:19:21.057026 ip-10-0-141-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:19:21.057026 ip-10-0-141-66 kubenswrapper[2566]:  >
Sep 17 17:19:21.057026 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:19:21.056822    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:19:21.057026 ip-10-0-141-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:19:21.057026 ip-10-0-141-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:19:21.057026 ip-10-0-141-66 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9" podUID="b56469b9-022f-4f50-91d1-b4db7e41dcc6" containerName="main" containerID="cri-o://9e23b07717f5dc2c2f406b5ae5dd944b31aaaa315c570d28a16e3ef2fd837620"
Sep 17 17:19:21.057026 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:21.056869    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9" podUID="b56469b9-022f-4f50-91d1-b4db7e41dcc6" containerName="main" containerID="cri-o://9e23b07717f5dc2c2f406b5ae5dd944b31aaaa315c570d28a16e3ef2fd837620" gracePeriod=60
Sep 17 17:19:21.403275 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:21.403242    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9"
Sep 17 17:19:21.479929 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:21.479888    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5nmcw" podUID="0027b824-480c-427b-931e-c12bd93b6a0a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:19:21.541974 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:21.541940    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b56469b9-022f-4f50-91d1-b4db7e41dcc6-home\") pod \"b56469b9-022f-4f50-91d1-b4db7e41dcc6\" (UID: \"b56469b9-022f-4f50-91d1-b4db7e41dcc6\") "
Sep 17 17:19:21.542149 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:21.541990    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b56469b9-022f-4f50-91d1-b4db7e41dcc6-dshm\") pod \"b56469b9-022f-4f50-91d1-b4db7e41dcc6\" (UID: \"b56469b9-022f-4f50-91d1-b4db7e41dcc6\") "
Sep 17 17:19:21.542149 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:21.542023    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b56469b9-022f-4f50-91d1-b4db7e41dcc6-tls-certs\") pod \"b56469b9-022f-4f50-91d1-b4db7e41dcc6\" (UID: \"b56469b9-022f-4f50-91d1-b4db7e41dcc6\") "
Sep 17 17:19:21.542149 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:21.542049    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-967vr\" (UniqueName: \"kubernetes.io/projected/b56469b9-022f-4f50-91d1-b4db7e41dcc6-kube-api-access-967vr\") pod \"b56469b9-022f-4f50-91d1-b4db7e41dcc6\" (UID: \"b56469b9-022f-4f50-91d1-b4db7e41dcc6\") "
Sep 17 17:19:21.542149 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:21.542082    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b56469b9-022f-4f50-91d1-b4db7e41dcc6-tmp-dir\") pod \"b56469b9-022f-4f50-91d1-b4db7e41dcc6\" (UID: \"b56469b9-022f-4f50-91d1-b4db7e41dcc6\") "
Sep 17 17:19:21.542149 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:21.542113    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b56469b9-022f-4f50-91d1-b4db7e41dcc6-model-cache\") pod \"b56469b9-022f-4f50-91d1-b4db7e41dcc6\" (UID: \"b56469b9-022f-4f50-91d1-b4db7e41dcc6\") "
Sep 17 17:19:21.542429 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:21.542235    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56469b9-022f-4f50-91d1-b4db7e41dcc6-home" (OuterVolumeSpecName: "home") pod "b56469b9-022f-4f50-91d1-b4db7e41dcc6" (UID: "b56469b9-022f-4f50-91d1-b4db7e41dcc6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:19:21.542429 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:21.542350    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56469b9-022f-4f50-91d1-b4db7e41dcc6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b56469b9-022f-4f50-91d1-b4db7e41dcc6" (UID: "b56469b9-022f-4f50-91d1-b4db7e41dcc6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:19:21.542429 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:21.542375    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56469b9-022f-4f50-91d1-b4db7e41dcc6-model-cache" (OuterVolumeSpecName: "model-cache") pod "b56469b9-022f-4f50-91d1-b4db7e41dcc6" (UID: "b56469b9-022f-4f50-91d1-b4db7e41dcc6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:19:21.544455 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:21.544415    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56469b9-022f-4f50-91d1-b4db7e41dcc6-dshm" (OuterVolumeSpecName: "dshm") pod "b56469b9-022f-4f50-91d1-b4db7e41dcc6" (UID: "b56469b9-022f-4f50-91d1-b4db7e41dcc6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:19:21.544568 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:21.544533    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b56469b9-022f-4f50-91d1-b4db7e41dcc6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b56469b9-022f-4f50-91d1-b4db7e41dcc6" (UID: "b56469b9-022f-4f50-91d1-b4db7e41dcc6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:19:21.544568 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:21.544551    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b56469b9-022f-4f50-91d1-b4db7e41dcc6-kube-api-access-967vr" (OuterVolumeSpecName: "kube-api-access-967vr") pod "b56469b9-022f-4f50-91d1-b4db7e41dcc6" (UID: "b56469b9-022f-4f50-91d1-b4db7e41dcc6"). InnerVolumeSpecName "kube-api-access-967vr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:19:21.643404 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:21.643308    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b56469b9-022f-4f50-91d1-b4db7e41dcc6-home\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:19:21.643404 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:21.643346    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b56469b9-022f-4f50-91d1-b4db7e41dcc6-dshm\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:19:21.643404 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:21.643357    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b56469b9-022f-4f50-91d1-b4db7e41dcc6-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:19:21.643404 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:21.643371    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-967vr\" (UniqueName: \"kubernetes.io/projected/b56469b9-022f-4f50-91d1-b4db7e41dcc6-kube-api-access-967vr\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:19:21.643404 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:21.643385    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b56469b9-022f-4f50-91d1-b4db7e41dcc6-tmp-dir\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:19:21.643404 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:21.643394    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b56469b9-022f-4f50-91d1-b4db7e41dcc6-model-cache\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:19:22.207542 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:22.207507    2566 generic.go:356] "Generic (PLEG): container finished" podID="b56469b9-022f-4f50-91d1-b4db7e41dcc6" containerID="9e23b07717f5dc2c2f406b5ae5dd944b31aaaa315c570d28a16e3ef2fd837620" exitCode=0
Sep 17 17:19:22.207979 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:22.207553    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9"
Sep 17 17:19:22.207979 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:22.207556    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9" event={"ID":"b56469b9-022f-4f50-91d1-b4db7e41dcc6","Type":"ContainerDied","Data":"9e23b07717f5dc2c2f406b5ae5dd944b31aaaa315c570d28a16e3ef2fd837620"}
Sep 17 17:19:22.207979 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:22.207684    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9" event={"ID":"b56469b9-022f-4f50-91d1-b4db7e41dcc6","Type":"ContainerDied","Data":"bd5d456492e9645ba2829cfcf02f608c5b895290e805bdc2c073c7de3a6bce70"}
Sep 17 17:19:22.207979 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:22.207717    2566 scope.go:117] "RemoveContainer" containerID="9e23b07717f5dc2c2f406b5ae5dd944b31aaaa315c570d28a16e3ef2fd837620"
Sep 17 17:19:22.218067 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:22.218048    2566 scope.go:117] "RemoveContainer" containerID="9e23b07717f5dc2c2f406b5ae5dd944b31aaaa315c570d28a16e3ef2fd837620"
Sep 17 17:19:22.218323 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:19:22.218303    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e23b07717f5dc2c2f406b5ae5dd944b31aaaa315c570d28a16e3ef2fd837620\": container with ID starting with 9e23b07717f5dc2c2f406b5ae5dd944b31aaaa315c570d28a16e3ef2fd837620 not found: ID does not exist" containerID="9e23b07717f5dc2c2f406b5ae5dd944b31aaaa315c570d28a16e3ef2fd837620"
Sep 17 17:19:22.218370 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:22.218331    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e23b07717f5dc2c2f406b5ae5dd944b31aaaa315c570d28a16e3ef2fd837620"} err="failed to get container status \"9e23b07717f5dc2c2f406b5ae5dd944b31aaaa315c570d28a16e3ef2fd837620\": rpc error: code = NotFound desc = could not find container \"9e23b07717f5dc2c2f406b5ae5dd944b31aaaa315c570d28a16e3ef2fd837620\": container with ID starting with 9e23b07717f5dc2c2f406b5ae5dd944b31aaaa315c570d28a16e3ef2fd837620 not found: ID does not exist"
Sep 17 17:19:22.249244 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:22.249218    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9"]
Sep 17 17:19:22.257949 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:22.257925    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679692x2z9"]
Sep 17 17:19:22.346234 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:22.346203    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b56469b9-022f-4f50-91d1-b4db7e41dcc6" path="/var/lib/kubelet/pods/b56469b9-022f-4f50-91d1-b4db7e41dcc6/volumes"
Sep 17 17:19:31.480920 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:31.480875    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5nmcw" podUID="0027b824-480c-427b-931e-c12bd93b6a0a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:19:36.065224 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:36.065162    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5nmcw" podUID="0027b824-480c-427b-931e-c12bd93b6a0a" containerName="main" containerID="cri-o://2429ecb6370899805fb53a8401fc8dbf756960667a0f50da7d62e31dd2227bdb" gracePeriod=45
Sep 17 17:19:41.480315 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:41.480272    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5nmcw" podUID="0027b824-480c-427b-931e-c12bd93b6a0a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:19:41.480796 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:41.480380    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5nmcw"
Sep 17 17:19:51.479933 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:19:51.479895    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5nmcw" podUID="0027b824-480c-427b-931e-c12bd93b6a0a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:20:01.479551 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:01.479510    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5nmcw" podUID="0027b824-480c-427b-931e-c12bd93b6a0a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:20:06.310934 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:06.310904    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5nmcw"
Sep 17 17:20:06.398758 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:06.398678    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mw25\" (UniqueName: \"kubernetes.io/projected/0027b824-480c-427b-931e-c12bd93b6a0a-kube-api-access-2mw25\") pod \"0027b824-480c-427b-931e-c12bd93b6a0a\" (UID: \"0027b824-480c-427b-931e-c12bd93b6a0a\") "
Sep 17 17:20:06.398758 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:06.398736    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0027b824-480c-427b-931e-c12bd93b6a0a-tls-certs\") pod \"0027b824-480c-427b-931e-c12bd93b6a0a\" (UID: \"0027b824-480c-427b-931e-c12bd93b6a0a\") "
Sep 17 17:20:06.400969 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:06.400945    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0027b824-480c-427b-931e-c12bd93b6a0a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0027b824-480c-427b-931e-c12bd93b6a0a" (UID: "0027b824-480c-427b-931e-c12bd93b6a0a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:20:06.401147 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:06.401126    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0027b824-480c-427b-931e-c12bd93b6a0a-kube-api-access-2mw25" (OuterVolumeSpecName: "kube-api-access-2mw25") pod "0027b824-480c-427b-931e-c12bd93b6a0a" (UID: "0027b824-480c-427b-931e-c12bd93b6a0a"). InnerVolumeSpecName "kube-api-access-2mw25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:20:06.424264 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:06.424234    2566 generic.go:356] "Generic (PLEG): container finished" podID="0027b824-480c-427b-931e-c12bd93b6a0a" containerID="2429ecb6370899805fb53a8401fc8dbf756960667a0f50da7d62e31dd2227bdb" exitCode=0
Sep 17 17:20:06.424382 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:06.424272    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5nmcw" event={"ID":"0027b824-480c-427b-931e-c12bd93b6a0a","Type":"ContainerDied","Data":"2429ecb6370899805fb53a8401fc8dbf756960667a0f50da7d62e31dd2227bdb"}
Sep 17 17:20:06.424382 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:06.424279    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5nmcw"
Sep 17 17:20:06.424382 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:06.424323    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5nmcw" event={"ID":"0027b824-480c-427b-931e-c12bd93b6a0a","Type":"ContainerDied","Data":"6a352904f8fc6aceb2a72483b6667245edfb0fad1041165cf09d55d9d82cf6f8"}
Sep 17 17:20:06.424382 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:06.424377    2566 scope.go:117] "RemoveContainer" containerID="2429ecb6370899805fb53a8401fc8dbf756960667a0f50da7d62e31dd2227bdb"
Sep 17 17:20:06.433731 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:06.433714    2566 scope.go:117] "RemoveContainer" containerID="2429ecb6370899805fb53a8401fc8dbf756960667a0f50da7d62e31dd2227bdb"
Sep 17 17:20:06.434029 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:20:06.434012    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2429ecb6370899805fb53a8401fc8dbf756960667a0f50da7d62e31dd2227bdb\": container with ID starting with 2429ecb6370899805fb53a8401fc8dbf756960667a0f50da7d62e31dd2227bdb not found: ID does not exist" containerID="2429ecb6370899805fb53a8401fc8dbf756960667a0f50da7d62e31dd2227bdb"
Sep 17 17:20:06.434077 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:06.434039    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2429ecb6370899805fb53a8401fc8dbf756960667a0f50da7d62e31dd2227bdb"} err="failed to get container status \"2429ecb6370899805fb53a8401fc8dbf756960667a0f50da7d62e31dd2227bdb\": rpc error: code = NotFound desc = could not find container \"2429ecb6370899805fb53a8401fc8dbf756960667a0f50da7d62e31dd2227bdb\": container with ID starting with 2429ecb6370899805fb53a8401fc8dbf756960667a0f50da7d62e31dd2227bdb not found: ID does not exist"
Sep 17 17:20:06.450420 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:06.450377    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5nmcw"]
Sep 17 17:20:06.452515 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:06.452491    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5nmcw"]
Sep 17 17:20:06.499331 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:06.499307    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0027b824-480c-427b-931e-c12bd93b6a0a-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:20:06.499331 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:06.499331    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mw25\" (UniqueName: \"kubernetes.io/projected/0027b824-480c-427b-931e-c12bd93b6a0a-kube-api-access-2mw25\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:20:08.346043 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:08.345997    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0027b824-480c-427b-931e-c12bd93b6a0a" path="/var/lib/kubelet/pods/0027b824-480c-427b-931e-c12bd93b6a0a/volumes"
Sep 17 17:20:13.918897 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:13.918860    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529"]
Sep 17 17:20:13.919286 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:13.919190    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b56469b9-022f-4f50-91d1-b4db7e41dcc6" containerName="main"
Sep 17 17:20:13.919286 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:13.919201    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b56469b9-022f-4f50-91d1-b4db7e41dcc6" containerName="main"
Sep 17 17:20:13.919286 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:13.919210    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="328562e9-0fa4-49b8-bcb2-35ffc6c20e5c" containerName="main"
Sep 17 17:20:13.919286 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:13.919215    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="328562e9-0fa4-49b8-bcb2-35ffc6c20e5c" containerName="main"
Sep 17 17:20:13.919286 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:13.919227    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0027b824-480c-427b-931e-c12bd93b6a0a" containerName="main"
Sep 17 17:20:13.919286 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:13.919232    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0027b824-480c-427b-931e-c12bd93b6a0a" containerName="main"
Sep 17 17:20:13.919286 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:13.919280    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0027b824-480c-427b-931e-c12bd93b6a0a" containerName="main"
Sep 17 17:20:13.919286 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:13.919287    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="328562e9-0fa4-49b8-bcb2-35ffc6c20e5c" containerName="main"
Sep 17 17:20:13.919587 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:13.919295    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b56469b9-022f-4f50-91d1-b4db7e41dcc6" containerName="main"
Sep 17 17:20:13.923798 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:13.923777    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529"
Sep 17 17:20:13.926264 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:13.926228    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 17 17:20:13.926264 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:13.926245    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Sep 17 17:20:13.926465 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:13.926248    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:20:13.926465 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:13.926290    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-qv7kv\""
Sep 17 17:20:13.926465 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:13.926229    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 17 17:20:13.926625 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:13.926567    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 17 17:20:13.945205 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:13.945176    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529"]
Sep 17 17:20:13.957674 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:13.957647    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkfdz\" (UniqueName: \"kubernetes.io/projected/639b7f94-0ea1-475b-a468-aba661f8f77d-kube-api-access-hkfdz\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529\" (UID: \"639b7f94-0ea1-475b-a468-aba661f8f77d\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529"
Sep 17 17:20:13.957790 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:13.957676    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/639b7f94-0ea1-475b-a468-aba661f8f77d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529\" (UID: \"639b7f94-0ea1-475b-a468-aba661f8f77d\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529"
Sep 17 17:20:13.957790 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:13.957699    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/639b7f94-0ea1-475b-a468-aba661f8f77d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529\" (UID: \"639b7f94-0ea1-475b-a468-aba661f8f77d\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529"
Sep 17 17:20:13.957790 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:13.957744    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/639b7f94-0ea1-475b-a468-aba661f8f77d-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529\" (UID: \"639b7f94-0ea1-475b-a468-aba661f8f77d\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529"
Sep 17 17:20:13.957790 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:13.957784    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/639b7f94-0ea1-475b-a468-aba661f8f77d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529\" (UID: \"639b7f94-0ea1-475b-a468-aba661f8f77d\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529"
Sep 17 17:20:13.957979 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:13.957832    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/639b7f94-0ea1-475b-a468-aba661f8f77d-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529\" (UID: \"639b7f94-0ea1-475b-a468-aba661f8f77d\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529"
Sep 17 17:20:13.957979 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:13.957853    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/639b7f94-0ea1-475b-a468-aba661f8f77d-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529\" (UID: \"639b7f94-0ea1-475b-a468-aba661f8f77d\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529"
Sep 17 17:20:13.957979 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:13.957882    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/639b7f94-0ea1-475b-a468-aba661f8f77d-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529\" (UID: \"639b7f94-0ea1-475b-a468-aba661f8f77d\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529"
Sep 17 17:20:14.059095 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:14.059063    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/639b7f94-0ea1-475b-a468-aba661f8f77d-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529\" (UID: \"639b7f94-0ea1-475b-a468-aba661f8f77d\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529"
Sep 17 17:20:14.059246 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:14.059102    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkfdz\" (UniqueName: \"kubernetes.io/projected/639b7f94-0ea1-475b-a468-aba661f8f77d-kube-api-access-hkfdz\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529\" (UID: \"639b7f94-0ea1-475b-a468-aba661f8f77d\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529"
Sep 17 17:20:14.059246 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:14.059127    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/639b7f94-0ea1-475b-a468-aba661f8f77d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529\" (UID: \"639b7f94-0ea1-475b-a468-aba661f8f77d\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529"
Sep 17 17:20:14.059346 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:14.059243    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/639b7f94-0ea1-475b-a468-aba661f8f77d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529\" (UID: \"639b7f94-0ea1-475b-a468-aba661f8f77d\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529"
Sep 17 17:20:14.059346 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:14.059280    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/639b7f94-0ea1-475b-a468-aba661f8f77d-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529\" (UID: \"639b7f94-0ea1-475b-a468-aba661f8f77d\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529"
Sep 17 17:20:14.059346 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:14.059322    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/639b7f94-0ea1-475b-a468-aba661f8f77d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529\" (UID: \"639b7f94-0ea1-475b-a468-aba661f8f77d\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529"
Sep 17 17:20:14.059538 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:14.059359    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/639b7f94-0ea1-475b-a468-aba661f8f77d-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529\" (UID: \"639b7f94-0ea1-475b-a468-aba661f8f77d\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529"
Sep 17 17:20:14.059538 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:14.059389    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/639b7f94-0ea1-475b-a468-aba661f8f77d-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529\" (UID: \"639b7f94-0ea1-475b-a468-aba661f8f77d\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529"
Sep 17 17:20:14.059645 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:14.059536    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/639b7f94-0ea1-475b-a468-aba661f8f77d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529\" (UID: \"639b7f94-0ea1-475b-a468-aba661f8f77d\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529"
Sep 17 17:20:14.059645 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:14.059562    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/639b7f94-0ea1-475b-a468-aba661f8f77d-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529\" (UID: \"639b7f94-0ea1-475b-a468-aba661f8f77d\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529"
Sep 17 17:20:14.059756 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:14.059685    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/639b7f94-0ea1-475b-a468-aba661f8f77d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529\" (UID: \"639b7f94-0ea1-475b-a468-aba661f8f77d\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529"
Sep 17 17:20:14.059814 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:14.059754    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/639b7f94-0ea1-475b-a468-aba661f8f77d-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529\" (UID: \"639b7f94-0ea1-475b-a468-aba661f8f77d\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529"
Sep 17 17:20:14.060055 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:14.060033    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/639b7f94-0ea1-475b-a468-aba661f8f77d-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529\" (UID: \"639b7f94-0ea1-475b-a468-aba661f8f77d\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529"
Sep 17 17:20:14.061795 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:14.061776    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/639b7f94-0ea1-475b-a468-aba661f8f77d-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529\" (UID: \"639b7f94-0ea1-475b-a468-aba661f8f77d\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529"
Sep 17 17:20:14.062153 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:14.062132    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/639b7f94-0ea1-475b-a468-aba661f8f77d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529\" (UID: \"639b7f94-0ea1-475b-a468-aba661f8f77d\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529"
Sep 17 17:20:14.078634 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:14.078609    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkfdz\" (UniqueName: \"kubernetes.io/projected/639b7f94-0ea1-475b-a468-aba661f8f77d-kube-api-access-hkfdz\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529\" (UID: \"639b7f94-0ea1-475b-a468-aba661f8f77d\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529"
Sep 17 17:20:14.234009 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:14.233927    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529"
Sep 17 17:20:14.306280 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:14.306250    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejlzlj"]
Sep 17 17:20:14.309695 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:14.309676    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejlzlj"
Sep 17 17:20:14.312114 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:14.312092    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-gxmzz\""
Sep 17 17:20:14.320704 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:14.320681    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejlzlj"]
Sep 17 17:20:14.362457 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:14.362409    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5460bddf-9eb4-4745-aa97-1e468e1f9eb4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejlzlj\" (UID: \"5460bddf-9eb4-4745-aa97-1e468e1f9eb4\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejlzlj"
Sep 17 17:20:14.362585 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:14.362464    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-779wp\" (UniqueName: \"kubernetes.io/projected/5460bddf-9eb4-4745-aa97-1e468e1f9eb4-kube-api-access-779wp\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejlzlj\" (UID: \"5460bddf-9eb4-4745-aa97-1e468e1f9eb4\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejlzlj"
Sep 17 17:20:14.462812 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:14.462776    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5460bddf-9eb4-4745-aa97-1e468e1f9eb4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejlzlj\" (UID: \"5460bddf-9eb4-4745-aa97-1e468e1f9eb4\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejlzlj"
Sep 17 17:20:14.462812 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:14.462807    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-779wp\" (UniqueName: \"kubernetes.io/projected/5460bddf-9eb4-4745-aa97-1e468e1f9eb4-kube-api-access-779wp\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejlzlj\" (UID: \"5460bddf-9eb4-4745-aa97-1e468e1f9eb4\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejlzlj"
Sep 17 17:20:14.465582 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:14.465558    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5460bddf-9eb4-4745-aa97-1e468e1f9eb4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejlzlj\" (UID: \"5460bddf-9eb4-4745-aa97-1e468e1f9eb4\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejlzlj"
Sep 17 17:20:14.473037 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:14.473019    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-779wp\" (UniqueName: \"kubernetes.io/projected/5460bddf-9eb4-4745-aa97-1e468e1f9eb4-kube-api-access-779wp\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejlzlj\" (UID: \"5460bddf-9eb4-4745-aa97-1e468e1f9eb4\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejlzlj"
Sep 17 17:20:14.515489 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:14.515464    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529"]
Sep 17 17:20:14.520360 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:20:14.520331    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod639b7f94_0ea1_475b_a468_aba661f8f77d.slice/crio-c9aa8bf27acb5010e6a5afb0aea905cddbc1ed906c16ba215e46637fd3a052c8 WatchSource:0}: Error finding container c9aa8bf27acb5010e6a5afb0aea905cddbc1ed906c16ba215e46637fd3a052c8: Status 404 returned error can't find the container with id c9aa8bf27acb5010e6a5afb0aea905cddbc1ed906c16ba215e46637fd3a052c8
Sep 17 17:20:14.623350 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:14.623309    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejlzlj"
Sep 17 17:20:14.907905 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:14.907880    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejlzlj"]
Sep 17 17:20:14.910618 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:20:14.910593    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5460bddf_9eb4_4745_aa97_1e468e1f9eb4.slice/crio-a52aeb9af25001c64e3d16d18a55039f9b4442e4ef0407ee38f3dc20c0530b4b WatchSource:0}: Error finding container a52aeb9af25001c64e3d16d18a55039f9b4442e4ef0407ee38f3dc20c0530b4b: Status 404 returned error can't find the container with id a52aeb9af25001c64e3d16d18a55039f9b4442e4ef0407ee38f3dc20c0530b4b
Sep 17 17:20:15.472453 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:15.472402    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejlzlj" event={"ID":"5460bddf-9eb4-4745-aa97-1e468e1f9eb4","Type":"ContainerStarted","Data":"a0bb56f9bed4778872f77d3c40d8e30ec2aaff58c090ec13d436f1a62df33a2e"}
Sep 17 17:20:15.472900 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:15.472604    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejlzlj"
Sep 17 17:20:15.472900 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:15.472630    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejlzlj" event={"ID":"5460bddf-9eb4-4745-aa97-1e468e1f9eb4","Type":"ContainerStarted","Data":"a52aeb9af25001c64e3d16d18a55039f9b4442e4ef0407ee38f3dc20c0530b4b"}
Sep 17 17:20:15.474004 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:15.473975    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529" event={"ID":"639b7f94-0ea1-475b-a468-aba661f8f77d","Type":"ContainerStarted","Data":"e6cb0e8faa28de7732fb1f2c5e011e804d61bd47c86a3211bc2207a4b55b6012"}
Sep 17 17:20:15.474109 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:15.474015    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529" event={"ID":"639b7f94-0ea1-475b-a468-aba661f8f77d","Type":"ContainerStarted","Data":"c9aa8bf27acb5010e6a5afb0aea905cddbc1ed906c16ba215e46637fd3a052c8"}
Sep 17 17:20:15.491614 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:15.491564    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejlzlj" podStartSLOduration=1.491550773 podStartE2EDuration="1.491550773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:20:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:20:15.489635683 +0000 UTC m=+2301.384114821" watchObservedRunningTime="2026-09-17 17:20:15.491550773 +0000 UTC m=+2301.386029912"
Sep 17 17:20:18.490364 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:18.490329    2566 generic.go:356] "Generic (PLEG): container finished" podID="639b7f94-0ea1-475b-a468-aba661f8f77d" containerID="e6cb0e8faa28de7732fb1f2c5e011e804d61bd47c86a3211bc2207a4b55b6012" exitCode=0
Sep 17 17:20:18.490769 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:18.490402    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529" event={"ID":"639b7f94-0ea1-475b-a468-aba661f8f77d","Type":"ContainerDied","Data":"e6cb0e8faa28de7732fb1f2c5e011e804d61bd47c86a3211bc2207a4b55b6012"}
Sep 17 17:20:19.498095 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:19.498057    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529" event={"ID":"639b7f94-0ea1-475b-a468-aba661f8f77d","Type":"ContainerStarted","Data":"2a4e7e212af3a56ca92ec85fe1a1638354f5fdbf6eae080832676223a7162f08"}
Sep 17 17:20:19.525921 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:19.525859    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529" podStartSLOduration=3.157003794 podStartE2EDuration="6.525842907s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.368839113s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:20:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:20:19.522816238 +0000 UTC m=+2305.417295377" watchObservedRunningTime="2026-09-17 17:20:19.525842907 +0000 UTC m=+2305.420322423"
Sep 17 17:20:20.234609 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:20.234572    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529"
Sep 17 17:20:24.235025 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:24.234983    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529"
Sep 17 17:20:24.236572 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:24.236540    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529" podUID="639b7f94-0ea1-475b-a468-aba661f8f77d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 17 17:20:34.234731 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:34.234675    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529" podUID="639b7f94-0ea1-475b-a468-aba661f8f77d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 17 17:20:44.235230 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:44.235185    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529" podUID="639b7f94-0ea1-475b-a468-aba661f8f77d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 17 17:20:46.480612 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:46.480580    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejlzlj"
Sep 17 17:20:54.234916 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:20:54.234865    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529" podUID="639b7f94-0ea1-475b-a468-aba661f8f77d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 17 17:21:04.235072 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:21:04.235013    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529" podUID="639b7f94-0ea1-475b-a468-aba661f8f77d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 17 17:21:14.235055 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:21:14.234992    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529" podUID="639b7f94-0ea1-475b-a468-aba661f8f77d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 17 17:21:24.235057 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:21:24.235000    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529" podUID="639b7f94-0ea1-475b-a468-aba661f8f77d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 17 17:21:34.244603 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:21:34.244566    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529"
Sep 17 17:21:34.252628 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:21:34.252599    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529"
Sep 17 17:21:56.484963 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:21:56.484932    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gblhr_91477834-6648-4e75-8530-e01bdee5e57e/ovn-acl-logging/0.log"
Sep 17 17:21:56.490964 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:21:56.490944    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gblhr_91477834-6648-4e75-8530-e01bdee5e57e/ovn-acl-logging/0.log"
Sep 17 17:22:01.464618 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:01.464575    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529"]
Sep 17 17:22:01.466926 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:01.466885    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejlzlj"]
Sep 17 17:22:06.481052 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:06.480961    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejlzlj" podUID="5460bddf-9eb4-4745-aa97-1e468e1f9eb4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:22:16.480957 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:16.480917    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejlzlj" podUID="5460bddf-9eb4-4745-aa97-1e468e1f9eb4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:22:16.495795 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:16.495733    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejlzlj" podUID="5460bddf-9eb4-4745-aa97-1e468e1f9eb4" containerName="main" containerID="cri-o://a0bb56f9bed4778872f77d3c40d8e30ec2aaff58c090ec13d436f1a62df33a2e" gracePeriod=45
Sep 17 17:22:16.498282 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:16.498250    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529" podUID="639b7f94-0ea1-475b-a468-aba661f8f77d" containerName="main" containerID="cri-o://2a4e7e212af3a56ca92ec85fe1a1638354f5fdbf6eae080832676223a7162f08" gracePeriod=45
Sep 17 17:22:17.127922 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:17.127890    2566 generic.go:356] "Generic (PLEG): container finished" podID="639b7f94-0ea1-475b-a468-aba661f8f77d" containerID="2a4e7e212af3a56ca92ec85fe1a1638354f5fdbf6eae080832676223a7162f08" exitCode=0
Sep 17 17:22:17.128110 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:17.127963    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529" event={"ID":"639b7f94-0ea1-475b-a468-aba661f8f77d","Type":"ContainerDied","Data":"2a4e7e212af3a56ca92ec85fe1a1638354f5fdbf6eae080832676223a7162f08"}
Sep 17 17:22:17.148621 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:17.148595    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529"
Sep 17 17:22:17.216426 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:17.216388    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/639b7f94-0ea1-475b-a468-aba661f8f77d-model-cache\") pod \"639b7f94-0ea1-475b-a468-aba661f8f77d\" (UID: \"639b7f94-0ea1-475b-a468-aba661f8f77d\") "
Sep 17 17:22:17.216626 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:17.216470    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/639b7f94-0ea1-475b-a468-aba661f8f77d-kserve-provision-location\") pod \"639b7f94-0ea1-475b-a468-aba661f8f77d\" (UID: \"639b7f94-0ea1-475b-a468-aba661f8f77d\") "
Sep 17 17:22:17.216626 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:17.216558    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/639b7f94-0ea1-475b-a468-aba661f8f77d-dshm\") pod \"639b7f94-0ea1-475b-a468-aba661f8f77d\" (UID: \"639b7f94-0ea1-475b-a468-aba661f8f77d\") "
Sep 17 17:22:17.216722 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:17.216639    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/639b7f94-0ea1-475b-a468-aba661f8f77d-cabundle-cert\") pod \"639b7f94-0ea1-475b-a468-aba661f8f77d\" (UID: \"639b7f94-0ea1-475b-a468-aba661f8f77d\") "
Sep 17 17:22:17.216722 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:17.216698    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/639b7f94-0ea1-475b-a468-aba661f8f77d-home\") pod \"639b7f94-0ea1-475b-a468-aba661f8f77d\" (UID: \"639b7f94-0ea1-475b-a468-aba661f8f77d\") "
Sep 17 17:22:17.216722 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:17.216698    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/639b7f94-0ea1-475b-a468-aba661f8f77d-model-cache" (OuterVolumeSpecName: "model-cache") pod "639b7f94-0ea1-475b-a468-aba661f8f77d" (UID: "639b7f94-0ea1-475b-a468-aba661f8f77d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:22:17.216888 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:17.216726    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/639b7f94-0ea1-475b-a468-aba661f8f77d-tmp-dir\") pod \"639b7f94-0ea1-475b-a468-aba661f8f77d\" (UID: \"639b7f94-0ea1-475b-a468-aba661f8f77d\") "
Sep 17 17:22:17.216888 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:17.216789    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/639b7f94-0ea1-475b-a468-aba661f8f77d-tls-certs\") pod \"639b7f94-0ea1-475b-a468-aba661f8f77d\" (UID: \"639b7f94-0ea1-475b-a468-aba661f8f77d\") "
Sep 17 17:22:17.216888 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:17.216829    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkfdz\" (UniqueName: \"kubernetes.io/projected/639b7f94-0ea1-475b-a468-aba661f8f77d-kube-api-access-hkfdz\") pod \"639b7f94-0ea1-475b-a468-aba661f8f77d\" (UID: \"639b7f94-0ea1-475b-a468-aba661f8f77d\") "
Sep 17 17:22:17.217085 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:17.217066    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/639b7f94-0ea1-475b-a468-aba661f8f77d-model-cache\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:22:17.217144 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:17.217065    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/639b7f94-0ea1-475b-a468-aba661f8f77d-home" (OuterVolumeSpecName: "home") pod "639b7f94-0ea1-475b-a468-aba661f8f77d" (UID: "639b7f94-0ea1-475b-a468-aba661f8f77d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:22:17.217144 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:17.217082    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/639b7f94-0ea1-475b-a468-aba661f8f77d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "639b7f94-0ea1-475b-a468-aba661f8f77d" (UID: "639b7f94-0ea1-475b-a468-aba661f8f77d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:22:17.219308 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:17.219230    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/639b7f94-0ea1-475b-a468-aba661f8f77d-kube-api-access-hkfdz" (OuterVolumeSpecName: "kube-api-access-hkfdz") pod "639b7f94-0ea1-475b-a468-aba661f8f77d" (UID: "639b7f94-0ea1-475b-a468-aba661f8f77d"). InnerVolumeSpecName "kube-api-access-hkfdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:22:17.219308 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:17.219298    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/639b7f94-0ea1-475b-a468-aba661f8f77d-dshm" (OuterVolumeSpecName: "dshm") pod "639b7f94-0ea1-475b-a468-aba661f8f77d" (UID: "639b7f94-0ea1-475b-a468-aba661f8f77d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:22:17.219502 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:17.219300    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/639b7f94-0ea1-475b-a468-aba661f8f77d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "639b7f94-0ea1-475b-a468-aba661f8f77d" (UID: "639b7f94-0ea1-475b-a468-aba661f8f77d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:22:17.234649 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:17.234611    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/639b7f94-0ea1-475b-a468-aba661f8f77d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "639b7f94-0ea1-475b-a468-aba661f8f77d" (UID: "639b7f94-0ea1-475b-a468-aba661f8f77d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:22:17.240985 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:17.240956    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/639b7f94-0ea1-475b-a468-aba661f8f77d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "639b7f94-0ea1-475b-a468-aba661f8f77d" (UID: "639b7f94-0ea1-475b-a468-aba661f8f77d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:22:17.317810 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:17.317771    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/639b7f94-0ea1-475b-a468-aba661f8f77d-kserve-provision-location\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:22:17.317810 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:17.317805    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/639b7f94-0ea1-475b-a468-aba661f8f77d-dshm\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:22:17.317810 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:17.317817    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/639b7f94-0ea1-475b-a468-aba661f8f77d-cabundle-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:22:17.318049 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:17.317826    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/639b7f94-0ea1-475b-a468-aba661f8f77d-home\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:22:17.318049 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:17.317856    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/639b7f94-0ea1-475b-a468-aba661f8f77d-tmp-dir\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:22:17.318049 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:17.317863    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/639b7f94-0ea1-475b-a468-aba661f8f77d-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:22:17.318049 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:17.317871    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkfdz\" (UniqueName: \"kubernetes.io/projected/639b7f94-0ea1-475b-a468-aba661f8f77d-kube-api-access-hkfdz\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:22:18.134464 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:18.134418    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529"
Sep 17 17:22:18.134937 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:18.134418    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529" event={"ID":"639b7f94-0ea1-475b-a468-aba661f8f77d","Type":"ContainerDied","Data":"c9aa8bf27acb5010e6a5afb0aea905cddbc1ed906c16ba215e46637fd3a052c8"}
Sep 17 17:22:18.134937 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:18.134564    2566 scope.go:117] "RemoveContainer" containerID="2a4e7e212af3a56ca92ec85fe1a1638354f5fdbf6eae080832676223a7162f08"
Sep 17 17:22:18.144331 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:18.144309    2566 scope.go:117] "RemoveContainer" containerID="e6cb0e8faa28de7732fb1f2c5e011e804d61bd47c86a3211bc2207a4b55b6012"
Sep 17 17:22:18.161076 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:18.161044    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529"]
Sep 17 17:22:18.166373 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:18.166341    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-559cfbcc-gf529"]
Sep 17 17:22:18.346788 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:18.346750    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="639b7f94-0ea1-475b-a468-aba661f8f77d" path="/var/lib/kubelet/pods/639b7f94-0ea1-475b-a468-aba661f8f77d/volumes"
Sep 17 17:22:26.480345 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:26.480304    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejlzlj" podUID="5460bddf-9eb4-4745-aa97-1e468e1f9eb4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:22:26.480924 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:26.480408    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejlzlj"
Sep 17 17:22:36.480305 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:36.480261    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejlzlj" podUID="5460bddf-9eb4-4745-aa97-1e468e1f9eb4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:22:46.480853 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:46.480809    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejlzlj" podUID="5460bddf-9eb4-4745-aa97-1e468e1f9eb4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:22:46.748323 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:46.748296    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejlzlj"
Sep 17 17:22:46.755670 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:46.755645    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5460bddf-9eb4-4745-aa97-1e468e1f9eb4-tls-certs\") pod \"5460bddf-9eb4-4745-aa97-1e468e1f9eb4\" (UID: \"5460bddf-9eb4-4745-aa97-1e468e1f9eb4\") "
Sep 17 17:22:46.755825 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:46.755689    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-779wp\" (UniqueName: \"kubernetes.io/projected/5460bddf-9eb4-4745-aa97-1e468e1f9eb4-kube-api-access-779wp\") pod \"5460bddf-9eb4-4745-aa97-1e468e1f9eb4\" (UID: \"5460bddf-9eb4-4745-aa97-1e468e1f9eb4\") "
Sep 17 17:22:46.757943 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:46.757918    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5460bddf-9eb4-4745-aa97-1e468e1f9eb4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5460bddf-9eb4-4745-aa97-1e468e1f9eb4" (UID: "5460bddf-9eb4-4745-aa97-1e468e1f9eb4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:22:46.757943 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:46.757923    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5460bddf-9eb4-4745-aa97-1e468e1f9eb4-kube-api-access-779wp" (OuterVolumeSpecName: "kube-api-access-779wp") pod "5460bddf-9eb4-4745-aa97-1e468e1f9eb4" (UID: "5460bddf-9eb4-4745-aa97-1e468e1f9eb4"). InnerVolumeSpecName "kube-api-access-779wp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:22:46.856312 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:46.856271    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5460bddf-9eb4-4745-aa97-1e468e1f9eb4-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:22:46.856312 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:46.856310    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-779wp\" (UniqueName: \"kubernetes.io/projected/5460bddf-9eb4-4745-aa97-1e468e1f9eb4-kube-api-access-779wp\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:22:47.287810 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:47.287770    2566 generic.go:356] "Generic (PLEG): container finished" podID="5460bddf-9eb4-4745-aa97-1e468e1f9eb4" containerID="a0bb56f9bed4778872f77d3c40d8e30ec2aaff58c090ec13d436f1a62df33a2e" exitCode=0
Sep 17 17:22:47.287986 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:47.287816    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejlzlj"
Sep 17 17:22:47.287986 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:47.287820    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejlzlj" event={"ID":"5460bddf-9eb4-4745-aa97-1e468e1f9eb4","Type":"ContainerDied","Data":"a0bb56f9bed4778872f77d3c40d8e30ec2aaff58c090ec13d436f1a62df33a2e"}
Sep 17 17:22:47.287986 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:47.287875    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejlzlj" event={"ID":"5460bddf-9eb4-4745-aa97-1e468e1f9eb4","Type":"ContainerDied","Data":"a52aeb9af25001c64e3d16d18a55039f9b4442e4ef0407ee38f3dc20c0530b4b"}
Sep 17 17:22:47.287986 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:47.287904    2566 scope.go:117] "RemoveContainer" containerID="a0bb56f9bed4778872f77d3c40d8e30ec2aaff58c090ec13d436f1a62df33a2e"
Sep 17 17:22:47.297332 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:47.297313    2566 scope.go:117] "RemoveContainer" containerID="a0bb56f9bed4778872f77d3c40d8e30ec2aaff58c090ec13d436f1a62df33a2e"
Sep 17 17:22:47.297654 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:22:47.297630    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0bb56f9bed4778872f77d3c40d8e30ec2aaff58c090ec13d436f1a62df33a2e\": container with ID starting with a0bb56f9bed4778872f77d3c40d8e30ec2aaff58c090ec13d436f1a62df33a2e not found: ID does not exist" containerID="a0bb56f9bed4778872f77d3c40d8e30ec2aaff58c090ec13d436f1a62df33a2e"
Sep 17 17:22:47.297702 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:47.297665    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0bb56f9bed4778872f77d3c40d8e30ec2aaff58c090ec13d436f1a62df33a2e"} err="failed to get container status \"a0bb56f9bed4778872f77d3c40d8e30ec2aaff58c090ec13d436f1a62df33a2e\": rpc error: code = NotFound desc = could not find container \"a0bb56f9bed4778872f77d3c40d8e30ec2aaff58c090ec13d436f1a62df33a2e\": container with ID starting with a0bb56f9bed4778872f77d3c40d8e30ec2aaff58c090ec13d436f1a62df33a2e not found: ID does not exist"
Sep 17 17:22:47.314330 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:47.314289    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejlzlj"]
Sep 17 17:22:47.320286 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:47.320261    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schejlzlj"]
Sep 17 17:22:48.346878 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:48.346831    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5460bddf-9eb4-4745-aa97-1e468e1f9eb4" path="/var/lib/kubelet/pods/5460bddf-9eb4-4745-aa97-1e468e1f9eb4/volumes"
Sep 17 17:22:58.112636 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.112603    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq"]
Sep 17 17:22:58.113061 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.112945    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5460bddf-9eb4-4745-aa97-1e468e1f9eb4" containerName="main"
Sep 17 17:22:58.113061 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.112955    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5460bddf-9eb4-4745-aa97-1e468e1f9eb4" containerName="main"
Sep 17 17:22:58.113061 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.112966    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="639b7f94-0ea1-475b-a468-aba661f8f77d" containerName="main"
Sep 17 17:22:58.113061 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.112972    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="639b7f94-0ea1-475b-a468-aba661f8f77d" containerName="main"
Sep 17 17:22:58.113061 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.112984    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="639b7f94-0ea1-475b-a468-aba661f8f77d" containerName="storage-initializer"
Sep 17 17:22:58.113061 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.112989    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="639b7f94-0ea1-475b-a468-aba661f8f77d" containerName="storage-initializer"
Sep 17 17:22:58.113061 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.113038    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5460bddf-9eb4-4745-aa97-1e468e1f9eb4" containerName="main"
Sep 17 17:22:58.113061 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.113046    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="639b7f94-0ea1-475b-a468-aba661f8f77d" containerName="main"
Sep 17 17:22:58.117544 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.117520    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq"
Sep 17 17:22:58.119790 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.119754    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:22:58.120083 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.119857    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 17 17:22:58.120083 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.120013    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 17 17:22:58.120253 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.120142    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Sep 17 17:22:58.120253 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.120184    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-b4hlm\""
Sep 17 17:22:58.120253 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.120202    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 17 17:22:58.131241 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.131203    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a7966bf-866e-4275-a78f-bdfa178cef24-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq\" (UID: \"7a7966bf-866e-4275-a78f-bdfa178cef24\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq"
Sep 17 17:22:58.131241 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.131248    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a7966bf-866e-4275-a78f-bdfa178cef24-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq\" (UID: \"7a7966bf-866e-4275-a78f-bdfa178cef24\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq"
Sep 17 17:22:58.131532 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.131281    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a7966bf-866e-4275-a78f-bdfa178cef24-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq\" (UID: \"7a7966bf-866e-4275-a78f-bdfa178cef24\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq"
Sep 17 17:22:58.131532 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.131334    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a7966bf-866e-4275-a78f-bdfa178cef24-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq\" (UID: \"7a7966bf-866e-4275-a78f-bdfa178cef24\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq"
Sep 17 17:22:58.131532 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.131355    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a7966bf-866e-4275-a78f-bdfa178cef24-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq\" (UID: \"7a7966bf-866e-4275-a78f-bdfa178cef24\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq"
Sep 17 17:22:58.131532 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.131411    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg9g9\" (UniqueName: \"kubernetes.io/projected/7a7966bf-866e-4275-a78f-bdfa178cef24-kube-api-access-fg9g9\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq\" (UID: \"7a7966bf-866e-4275-a78f-bdfa178cef24\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq"
Sep 17 17:22:58.131532 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.131495    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a7966bf-866e-4275-a78f-bdfa178cef24-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq\" (UID: \"7a7966bf-866e-4275-a78f-bdfa178cef24\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq"
Sep 17 17:22:58.131532 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.131522    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a7966bf-866e-4275-a78f-bdfa178cef24-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq\" (UID: \"7a7966bf-866e-4275-a78f-bdfa178cef24\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq"
Sep 17 17:22:58.134676 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.134652    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq"]
Sep 17 17:22:58.232679 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.232623    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a7966bf-866e-4275-a78f-bdfa178cef24-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq\" (UID: \"7a7966bf-866e-4275-a78f-bdfa178cef24\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq"
Sep 17 17:22:58.232679 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.232677    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a7966bf-866e-4275-a78f-bdfa178cef24-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq\" (UID: \"7a7966bf-866e-4275-a78f-bdfa178cef24\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq"
Sep 17 17:22:58.232956 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.232717    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a7966bf-866e-4275-a78f-bdfa178cef24-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq\" (UID: \"7a7966bf-866e-4275-a78f-bdfa178cef24\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq"
Sep 17 17:22:58.232956 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.232735    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a7966bf-866e-4275-a78f-bdfa178cef24-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq\" (UID: \"7a7966bf-866e-4275-a78f-bdfa178cef24\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq"
Sep 17 17:22:58.232956 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.232752    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a7966bf-866e-4275-a78f-bdfa178cef24-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq\" (UID: \"7a7966bf-866e-4275-a78f-bdfa178cef24\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq"
Sep 17 17:22:58.232956 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.232774    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg9g9\" (UniqueName: \"kubernetes.io/projected/7a7966bf-866e-4275-a78f-bdfa178cef24-kube-api-access-fg9g9\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq\" (UID: \"7a7966bf-866e-4275-a78f-bdfa178cef24\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq"
Sep 17 17:22:58.232956 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.232803    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a7966bf-866e-4275-a78f-bdfa178cef24-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq\" (UID: \"7a7966bf-866e-4275-a78f-bdfa178cef24\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq"
Sep 17 17:22:58.232956 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.232821    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a7966bf-866e-4275-a78f-bdfa178cef24-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq\" (UID: \"7a7966bf-866e-4275-a78f-bdfa178cef24\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq"
Sep 17 17:22:58.233272 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.233136    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a7966bf-866e-4275-a78f-bdfa178cef24-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq\" (UID: \"7a7966bf-866e-4275-a78f-bdfa178cef24\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq"
Sep 17 17:22:58.233272 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.233146    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a7966bf-866e-4275-a78f-bdfa178cef24-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq\" (UID: \"7a7966bf-866e-4275-a78f-bdfa178cef24\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq"
Sep 17 17:22:58.233272 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.233215    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a7966bf-866e-4275-a78f-bdfa178cef24-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq\" (UID: \"7a7966bf-866e-4275-a78f-bdfa178cef24\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq"
Sep 17 17:22:58.233402 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.233382    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a7966bf-866e-4275-a78f-bdfa178cef24-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq\" (UID: \"7a7966bf-866e-4275-a78f-bdfa178cef24\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq"
Sep 17 17:22:58.233599 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.233582    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a7966bf-866e-4275-a78f-bdfa178cef24-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq\" (UID: \"7a7966bf-866e-4275-a78f-bdfa178cef24\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq"
Sep 17 17:22:58.235284 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.235267    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a7966bf-866e-4275-a78f-bdfa178cef24-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq\" (UID: \"7a7966bf-866e-4275-a78f-bdfa178cef24\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq"
Sep 17 17:22:58.235592 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.235570    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a7966bf-866e-4275-a78f-bdfa178cef24-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq\" (UID: \"7a7966bf-866e-4275-a78f-bdfa178cef24\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq"
Sep 17 17:22:58.244146 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.244116    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg9g9\" (UniqueName: \"kubernetes.io/projected/7a7966bf-866e-4275-a78f-bdfa178cef24-kube-api-access-fg9g9\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq\" (UID: \"7a7966bf-866e-4275-a78f-bdfa178cef24\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq"
Sep 17 17:22:58.423462 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.423369    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltxft"]
Sep 17 17:22:58.427753 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.427732    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltxft"
Sep 17 17:22:58.427837 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.427796    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq"
Sep 17 17:22:58.429970 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.429944    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-dxbz8\""
Sep 17 17:22:58.433767 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.433740    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r456\" (UniqueName: \"kubernetes.io/projected/c40fc87c-c9f2-4f2c-99d5-1ef240215e1f-kube-api-access-6r456\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltxft\" (UID: \"c40fc87c-c9f2-4f2c-99d5-1ef240215e1f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltxft"
Sep 17 17:22:58.433896 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.433793    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c40fc87c-c9f2-4f2c-99d5-1ef240215e1f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltxft\" (UID: \"c40fc87c-c9f2-4f2c-99d5-1ef240215e1f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltxft"
Sep 17 17:22:58.445229 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.445197    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltxft"]
Sep 17 17:22:58.534766 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.534727    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c40fc87c-c9f2-4f2c-99d5-1ef240215e1f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltxft\" (UID: \"c40fc87c-c9f2-4f2c-99d5-1ef240215e1f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltxft"
Sep 17 17:22:58.534954 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.534818    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r456\" (UniqueName: \"kubernetes.io/projected/c40fc87c-c9f2-4f2c-99d5-1ef240215e1f-kube-api-access-6r456\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltxft\" (UID: \"c40fc87c-c9f2-4f2c-99d5-1ef240215e1f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltxft"
Sep 17 17:22:58.537511 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.537491    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c40fc87c-c9f2-4f2c-99d5-1ef240215e1f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltxft\" (UID: \"c40fc87c-c9f2-4f2c-99d5-1ef240215e1f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltxft"
Sep 17 17:22:58.544936 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.544913    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r456\" (UniqueName: \"kubernetes.io/projected/c40fc87c-c9f2-4f2c-99d5-1ef240215e1f-kube-api-access-6r456\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltxft\" (UID: \"c40fc87c-c9f2-4f2c-99d5-1ef240215e1f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltxft"
Sep 17 17:22:58.724680 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.724650    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq"]
Sep 17 17:22:58.726878 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:22:58.726849    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a7966bf_866e_4275_a78f_bdfa178cef24.slice/crio-344097a0d8e9837de0bcf3aa924bb67da492568fe096aa9e4e9de1a82b355a65 WatchSource:0}: Error finding container 344097a0d8e9837de0bcf3aa924bb67da492568fe096aa9e4e9de1a82b355a65: Status 404 returned error can't find the container with id 344097a0d8e9837de0bcf3aa924bb67da492568fe096aa9e4e9de1a82b355a65
Sep 17 17:22:58.738676 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:58.738647    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltxft"
Sep 17 17:22:59.016498 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:59.016469    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltxft"]
Sep 17 17:22:59.018349 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:22:59.018318    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc40fc87c_c9f2_4f2c_99d5_1ef240215e1f.slice/crio-81d3026eace205940ab6f810cd003f8ab277881737cc11fc1fc27ed10b2206de WatchSource:0}: Error finding container 81d3026eace205940ab6f810cd003f8ab277881737cc11fc1fc27ed10b2206de: Status 404 returned error can't find the container with id 81d3026eace205940ab6f810cd003f8ab277881737cc11fc1fc27ed10b2206de
Sep 17 17:22:59.354266 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:59.354224    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq" event={"ID":"7a7966bf-866e-4275-a78f-bdfa178cef24","Type":"ContainerStarted","Data":"24e9433d245b720a72c00d432bfe6bd75b5040c0ff8caf504b667b126bbed498"}
Sep 17 17:22:59.354804 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:59.354297    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq" event={"ID":"7a7966bf-866e-4275-a78f-bdfa178cef24","Type":"ContainerStarted","Data":"344097a0d8e9837de0bcf3aa924bb67da492568fe096aa9e4e9de1a82b355a65"}
Sep 17 17:22:59.355991 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:59.355964    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltxft" event={"ID":"c40fc87c-c9f2-4f2c-99d5-1ef240215e1f","Type":"ContainerStarted","Data":"8c7dd6dfa2156fafbdd80262fe5c6e85fabb8dad51c2a61197fb6ca3e1e6e2ed"}
Sep 17 17:22:59.356137 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:59.356011    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltxft" event={"ID":"c40fc87c-c9f2-4f2c-99d5-1ef240215e1f","Type":"ContainerStarted","Data":"81d3026eace205940ab6f810cd003f8ab277881737cc11fc1fc27ed10b2206de"}
Sep 17 17:22:59.356137 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:59.356091    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltxft"
Sep 17 17:22:59.395752 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:22:59.395599    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltxft" podStartSLOduration=1.395578878 podStartE2EDuration="1.395578878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:22:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:22:59.395017648 +0000 UTC m=+2465.289496813" watchObservedRunningTime="2026-09-17 17:22:59.395578878 +0000 UTC m=+2465.290058025"
Sep 17 17:23:02.372095 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:23:02.372060    2566 generic.go:356] "Generic (PLEG): container finished" podID="7a7966bf-866e-4275-a78f-bdfa178cef24" containerID="24e9433d245b720a72c00d432bfe6bd75b5040c0ff8caf504b667b126bbed498" exitCode=0
Sep 17 17:23:02.372545 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:23:02.372138    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq" event={"ID":"7a7966bf-866e-4275-a78f-bdfa178cef24","Type":"ContainerDied","Data":"24e9433d245b720a72c00d432bfe6bd75b5040c0ff8caf504b667b126bbed498"}
Sep 17 17:23:03.379081 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:23:03.379043    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq" event={"ID":"7a7966bf-866e-4275-a78f-bdfa178cef24","Type":"ContainerStarted","Data":"e65d540f1c56b08c62c0b47b3fd31dda757fbc542cbbbc4ff62003ee02b10edc"}
Sep 17 17:23:03.405313 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:23:03.405249    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq" podStartSLOduration=2.523428088 podStartE2EDuration="5.405232338s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.88180425s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:22:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:23:03.402347847 +0000 UTC m=+2469.296826989" watchObservedRunningTime="2026-09-17 17:23:03.405232338 +0000 UTC m=+2469.299711477"
Sep 17 17:23:03.428992 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:23:03.428940    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq"
Sep 17 17:23:08.428632 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:23:08.428577    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq"
Sep 17 17:23:08.430098 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:23:08.430066    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq" podUID="7a7966bf-866e-4275-a78f-bdfa178cef24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 17 17:23:18.429008 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:23:18.428958    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq" podUID="7a7966bf-866e-4275-a78f-bdfa178cef24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 17 17:23:28.428537 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:23:28.428490    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq" podUID="7a7966bf-866e-4275-a78f-bdfa178cef24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 17 17:23:30.362873 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:23:30.362841    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltxft"
Sep 17 17:23:38.429012 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:23:38.428907    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq" podUID="7a7966bf-866e-4275-a78f-bdfa178cef24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 17 17:23:48.428967 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:23:48.428918    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq" podUID="7a7966bf-866e-4275-a78f-bdfa178cef24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 17 17:23:58.428452 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:23:58.428401    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq" podUID="7a7966bf-866e-4275-a78f-bdfa178cef24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 17 17:24:04.136509 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:04.136464    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"]
Sep 17 17:24:04.140347 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:04.140318    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"
Sep 17 17:24:04.142915 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:04.142886    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 17 17:24:04.143050 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:04.142889    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-gw-ca-root-cert\""
Sep 17 17:24:04.143050 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:04.142890    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 17 17:24:04.143050 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:04.142891    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-gateway-1-openshift-default-dockercfg-fsbtb\""
Sep 17 17:24:04.157421 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:04.157386    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"]
Sep 17 17:24:04.196184 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:04.196147    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-istio-envoy\") pod \"router-gateway-1-openshift-default-cc7cb45dd-7xcct\" (UID: \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"
Sep 17 17:24:04.196184 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:04.196202    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-istio-podinfo\") pod \"router-gateway-1-openshift-default-cc7cb45dd-7xcct\" (UID: \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"
Sep 17 17:24:04.196518 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:04.196239    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-workload-certs\") pod \"router-gateway-1-openshift-default-cc7cb45dd-7xcct\" (UID: \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"
Sep 17 17:24:04.196518 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:04.196313    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-credential-socket\") pod \"router-gateway-1-openshift-default-cc7cb45dd-7xcct\" (UID: \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"
Sep 17 17:24:04.196518 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:04.196340    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-istio-token\") pod \"router-gateway-1-openshift-default-cc7cb45dd-7xcct\" (UID: \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"
Sep 17 17:24:04.196518 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:04.196364    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-workload-socket\") pod \"router-gateway-1-openshift-default-cc7cb45dd-7xcct\" (UID: \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"
Sep 17 17:24:04.196518 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:04.196390    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkwq5\" (UniqueName: \"kubernetes.io/projected/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-kube-api-access-mkwq5\") pod \"router-gateway-1-openshift-default-cc7cb45dd-7xcct\" (UID: \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"
Sep 17 17:24:04.196518 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:04.196460    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-istio-data\") pod \"router-gateway-1-openshift-default-cc7cb45dd-7xcct\" (UID: \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"
Sep 17 17:24:04.196518 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:04.196501    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-cc7cb45dd-7xcct\" (UID: \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"
Sep 17 17:24:04.297841 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:04.297804    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-cc7cb45dd-7xcct\" (UID: \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"
Sep 17 17:24:04.298051 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:04.297854    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-istio-envoy\") pod \"router-gateway-1-openshift-default-cc7cb45dd-7xcct\" (UID: \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"
Sep 17 17:24:04.298051 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:04.297883    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-istio-podinfo\") pod \"router-gateway-1-openshift-default-cc7cb45dd-7xcct\" (UID: \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"
Sep 17 17:24:04.298051 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:04.297900    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-workload-certs\") pod \"router-gateway-1-openshift-default-cc7cb45dd-7xcct\" (UID: \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"
Sep 17 17:24:04.298051 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:04.297929    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-credential-socket\") pod \"router-gateway-1-openshift-default-cc7cb45dd-7xcct\" (UID: \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"
Sep 17 17:24:04.298051 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:04.297952    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-istio-token\") pod \"router-gateway-1-openshift-default-cc7cb45dd-7xcct\" (UID: \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"
Sep 17 17:24:04.298051 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:04.297976    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-workload-socket\") pod \"router-gateway-1-openshift-default-cc7cb45dd-7xcct\" (UID: \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"
Sep 17 17:24:04.298051 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:04.298006    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkwq5\" (UniqueName: \"kubernetes.io/projected/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-kube-api-access-mkwq5\") pod \"router-gateway-1-openshift-default-cc7cb45dd-7xcct\" (UID: \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"
Sep 17 17:24:04.298461 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:04.298054    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-istio-data\") pod \"router-gateway-1-openshift-default-cc7cb45dd-7xcct\" (UID: \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"
Sep 17 17:24:04.298532 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:04.298513    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-istio-data\") pod \"router-gateway-1-openshift-default-cc7cb45dd-7xcct\" (UID: \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"
Sep 17 17:24:04.299299 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:04.298677    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-credential-socket\") pod \"router-gateway-1-openshift-default-cc7cb45dd-7xcct\" (UID: \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"
Sep 17 17:24:04.299299 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:04.298918    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-workload-socket\") pod \"router-gateway-1-openshift-default-cc7cb45dd-7xcct\" (UID: \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"
Sep 17 17:24:04.299299 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:04.298934    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-cc7cb45dd-7xcct\" (UID: \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"
Sep 17 17:24:04.299299 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:04.299244    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-workload-certs\") pod \"router-gateway-1-openshift-default-cc7cb45dd-7xcct\" (UID: \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"
Sep 17 17:24:04.300645 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:04.300619    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-istio-envoy\") pod \"router-gateway-1-openshift-default-cc7cb45dd-7xcct\" (UID: \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"
Sep 17 17:24:04.301170 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:04.301145    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-istio-podinfo\") pod \"router-gateway-1-openshift-default-cc7cb45dd-7xcct\" (UID: \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"
Sep 17 17:24:04.316954 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:04.316926    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-istio-token\") pod \"router-gateway-1-openshift-default-cc7cb45dd-7xcct\" (UID: \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"
Sep 17 17:24:04.317148 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:04.317123    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkwq5\" (UniqueName: \"kubernetes.io/projected/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-kube-api-access-mkwq5\") pod \"router-gateway-1-openshift-default-cc7cb45dd-7xcct\" (UID: \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"
Sep 17 17:24:04.451919 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:04.451799    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"
Sep 17 17:24:04.738321 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:04.738287    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"]
Sep 17 17:24:04.740393 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:24:04.740366    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e5e2ade_998c_4ec1_a2a9_ae7ef2dc2acd.slice/crio-7e4caf78f6963a321dbba2e20f73af51bc0511681ccf43e7c4c5636d39a77eb1 WatchSource:0}: Error finding container 7e4caf78f6963a321dbba2e20f73af51bc0511681ccf43e7c4c5636d39a77eb1: Status 404 returned error can't find the container with id 7e4caf78f6963a321dbba2e20f73af51bc0511681ccf43e7c4c5636d39a77eb1
Sep 17 17:24:04.742635 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:04.742605    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 17:24:04.742843 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:04.742689    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 17:24:04.742843 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:04.742733    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 17:24:05.726356 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:05.726309    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct" event={"ID":"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd","Type":"ContainerStarted","Data":"9c5aa51709e60ee2a48bf3ba96849079bef874a22b746612c5fc8b780f2092d2"}
Sep 17 17:24:05.726778 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:05.726388    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct" event={"ID":"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd","Type":"ContainerStarted","Data":"7e4caf78f6963a321dbba2e20f73af51bc0511681ccf43e7c4c5636d39a77eb1"}
Sep 17 17:24:05.756105 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:05.756033    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct" podStartSLOduration=1.756008791 podStartE2EDuration="1.756008791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:24:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:24:05.755099504 +0000 UTC m=+2531.649578642" watchObservedRunningTime="2026-09-17 17:24:05.756008791 +0000 UTC m=+2531.650487931"
Sep 17 17:24:06.452177 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:06.452134    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"
Sep 17 17:24:06.457321 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:06.457289    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"
Sep 17 17:24:06.732265 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:06.732160    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"
Sep 17 17:24:06.733754 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:06.733728    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"
Sep 17 17:24:08.428953 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:08.428888    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq" podUID="7a7966bf-866e-4275-a78f-bdfa178cef24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 17 17:24:16.794532 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:16.794497    2566 generic.go:356] "Generic (PLEG): container finished" podID="5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd" containerID="9c5aa51709e60ee2a48bf3ba96849079bef874a22b746612c5fc8b780f2092d2" exitCode=0
Sep 17 17:24:16.794955 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:16.794567    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct" event={"ID":"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd","Type":"ContainerDied","Data":"9c5aa51709e60ee2a48bf3ba96849079bef874a22b746612c5fc8b780f2092d2"}
Sep 17 17:24:16.794999 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:16.794978    2566 scope.go:117] "RemoveContainer" containerID="9c5aa51709e60ee2a48bf3ba96849079bef874a22b746612c5fc8b780f2092d2"
Sep 17 17:24:16.796038 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:16.796009    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 17:24:16.796106 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:16.796080    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 17:24:16.796144 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:16.796111    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 17:24:17.807071 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:17.806119    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct" event={"ID":"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd","Type":"ContainerStarted","Data":"9caa62b55901497bb0733566f22d60d02b93105534809b3f5f7a8f6f7380b50b"}
Sep 17 17:24:18.428893 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:18.428831    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq" podUID="7a7966bf-866e-4275-a78f-bdfa178cef24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 17 17:24:18.452195 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:18.452140    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"
Sep 17 17:24:18.453688 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:18.453646    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct" podUID="5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Sep 17 17:24:19.452760 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:19.452715    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct" podUID="5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Sep 17 17:24:20.452636 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:20.452588    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct" podUID="5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Sep 17 17:24:21.452810 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:21.452763    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct" podUID="5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Sep 17 17:24:21.733142 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:21.733042    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"
Sep 17 17:24:22.452999 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:22.452953    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct" podUID="5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Sep 17 17:24:23.456807 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:23.456774    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"
Sep 17 17:24:23.457942 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:23.457920    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"
Sep 17 17:24:28.439172 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:28.439140    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq"
Sep 17 17:24:28.447170 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:28.447147    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq"
Sep 17 17:24:45.262670 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:45.262631    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq"]
Sep 17 17:24:45.350374 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:45.350325    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltxft"]
Sep 17 17:24:50.363164 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:24:50.363125    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltxft" podUID="c40fc87c-c9f2-4f2c-99d5-1ef240215e1f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:25:00.286057 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:00.285997    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq" podUID="7a7966bf-866e-4275-a78f-bdfa178cef24" containerName="main" containerID="cri-o://e65d540f1c56b08c62c0b47b3fd31dda757fbc542cbbbc4ff62003ee02b10edc" gracePeriod=45
Sep 17 17:25:00.362258 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:00.362208    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltxft" podUID="c40fc87c-c9f2-4f2c-99d5-1ef240215e1f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:25:00.374395 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:00.374334    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltxft" podUID="c40fc87c-c9f2-4f2c-99d5-1ef240215e1f" containerName="main" containerID="cri-o://8c7dd6dfa2156fafbdd80262fe5c6e85fabb8dad51c2a61197fb6ca3e1e6e2ed" gracePeriod=45
Sep 17 17:25:00.440112 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:00.440060    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq" podUID="7a7966bf-866e-4275-a78f-bdfa178cef24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 17 17:25:00.935591 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:00.935562    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq"
Sep 17 17:25:00.990694 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:00.990597    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a7966bf-866e-4275-a78f-bdfa178cef24-cabundle-cert\") pod \"7a7966bf-866e-4275-a78f-bdfa178cef24\" (UID: \"7a7966bf-866e-4275-a78f-bdfa178cef24\") "
Sep 17 17:25:00.990694 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:00.990643    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fg9g9\" (UniqueName: \"kubernetes.io/projected/7a7966bf-866e-4275-a78f-bdfa178cef24-kube-api-access-fg9g9\") pod \"7a7966bf-866e-4275-a78f-bdfa178cef24\" (UID: \"7a7966bf-866e-4275-a78f-bdfa178cef24\") "
Sep 17 17:25:00.990694 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:00.990668    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a7966bf-866e-4275-a78f-bdfa178cef24-tmp-dir\") pod \"7a7966bf-866e-4275-a78f-bdfa178cef24\" (UID: \"7a7966bf-866e-4275-a78f-bdfa178cef24\") "
Sep 17 17:25:00.990694 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:00.990681    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a7966bf-866e-4275-a78f-bdfa178cef24-model-cache\") pod \"7a7966bf-866e-4275-a78f-bdfa178cef24\" (UID: \"7a7966bf-866e-4275-a78f-bdfa178cef24\") "
Sep 17 17:25:00.991045 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:00.990707    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a7966bf-866e-4275-a78f-bdfa178cef24-dshm\") pod \"7a7966bf-866e-4275-a78f-bdfa178cef24\" (UID: \"7a7966bf-866e-4275-a78f-bdfa178cef24\") "
Sep 17 17:25:00.991045 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:00.990740    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a7966bf-866e-4275-a78f-bdfa178cef24-kserve-provision-location\") pod \"7a7966bf-866e-4275-a78f-bdfa178cef24\" (UID: \"7a7966bf-866e-4275-a78f-bdfa178cef24\") "
Sep 17 17:25:00.991045 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:00.990781    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a7966bf-866e-4275-a78f-bdfa178cef24-home\") pod \"7a7966bf-866e-4275-a78f-bdfa178cef24\" (UID: \"7a7966bf-866e-4275-a78f-bdfa178cef24\") "
Sep 17 17:25:00.991045 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:00.990822    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a7966bf-866e-4275-a78f-bdfa178cef24-tls-certs\") pod \"7a7966bf-866e-4275-a78f-bdfa178cef24\" (UID: \"7a7966bf-866e-4275-a78f-bdfa178cef24\") "
Sep 17 17:25:00.991045 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:00.991010    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a7966bf-866e-4275-a78f-bdfa178cef24-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7a7966bf-866e-4275-a78f-bdfa178cef24" (UID: "7a7966bf-866e-4275-a78f-bdfa178cef24"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:25:00.991304 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:00.991258    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a7966bf-866e-4275-a78f-bdfa178cef24-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7a7966bf-866e-4275-a78f-bdfa178cef24" (UID: "7a7966bf-866e-4275-a78f-bdfa178cef24"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:25:00.991304 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:00.991263    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a7966bf-866e-4275-a78f-bdfa178cef24-home" (OuterVolumeSpecName: "home") pod "7a7966bf-866e-4275-a78f-bdfa178cef24" (UID: "7a7966bf-866e-4275-a78f-bdfa178cef24"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:25:00.991304 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:00.991277    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a7966bf-866e-4275-a78f-bdfa178cef24-model-cache" (OuterVolumeSpecName: "model-cache") pod "7a7966bf-866e-4275-a78f-bdfa178cef24" (UID: "7a7966bf-866e-4275-a78f-bdfa178cef24"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:25:00.993578 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:00.993515    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a7966bf-866e-4275-a78f-bdfa178cef24-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7a7966bf-866e-4275-a78f-bdfa178cef24" (UID: "7a7966bf-866e-4275-a78f-bdfa178cef24"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:25:00.993578 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:00.993567    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a7966bf-866e-4275-a78f-bdfa178cef24-dshm" (OuterVolumeSpecName: "dshm") pod "7a7966bf-866e-4275-a78f-bdfa178cef24" (UID: "7a7966bf-866e-4275-a78f-bdfa178cef24"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:25:00.993898 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:00.993732    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a7966bf-866e-4275-a78f-bdfa178cef24-kube-api-access-fg9g9" (OuterVolumeSpecName: "kube-api-access-fg9g9") pod "7a7966bf-866e-4275-a78f-bdfa178cef24" (UID: "7a7966bf-866e-4275-a78f-bdfa178cef24"). InnerVolumeSpecName "kube-api-access-fg9g9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:25:01.016895 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:01.016848    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a7966bf-866e-4275-a78f-bdfa178cef24-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7a7966bf-866e-4275-a78f-bdfa178cef24" (UID: "7a7966bf-866e-4275-a78f-bdfa178cef24"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:25:01.051666 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:01.051623    2566 generic.go:356] "Generic (PLEG): container finished" podID="7a7966bf-866e-4275-a78f-bdfa178cef24" containerID="e65d540f1c56b08c62c0b47b3fd31dda757fbc542cbbbc4ff62003ee02b10edc" exitCode=0
Sep 17 17:25:01.051885 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:01.051687    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq"
Sep 17 17:25:01.051885 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:01.051704    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq" event={"ID":"7a7966bf-866e-4275-a78f-bdfa178cef24","Type":"ContainerDied","Data":"e65d540f1c56b08c62c0b47b3fd31dda757fbc542cbbbc4ff62003ee02b10edc"}
Sep 17 17:25:01.051885 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:01.051765    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq" event={"ID":"7a7966bf-866e-4275-a78f-bdfa178cef24","Type":"ContainerDied","Data":"344097a0d8e9837de0bcf3aa924bb67da492568fe096aa9e4e9de1a82b355a65"}
Sep 17 17:25:01.051885 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:01.051788    2566 scope.go:117] "RemoveContainer" containerID="e65d540f1c56b08c62c0b47b3fd31dda757fbc542cbbbc4ff62003ee02b10edc"
Sep 17 17:25:01.061464 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:01.061423    2566 scope.go:117] "RemoveContainer" containerID="24e9433d245b720a72c00d432bfe6bd75b5040c0ff8caf504b667b126bbed498"
Sep 17 17:25:01.070354 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:01.070333    2566 scope.go:117] "RemoveContainer" containerID="e65d540f1c56b08c62c0b47b3fd31dda757fbc542cbbbc4ff62003ee02b10edc"
Sep 17 17:25:01.070675 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:25:01.070655    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e65d540f1c56b08c62c0b47b3fd31dda757fbc542cbbbc4ff62003ee02b10edc\": container with ID starting with e65d540f1c56b08c62c0b47b3fd31dda757fbc542cbbbc4ff62003ee02b10edc not found: ID does not exist" containerID="e65d540f1c56b08c62c0b47b3fd31dda757fbc542cbbbc4ff62003ee02b10edc"
Sep 17 17:25:01.070743 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:01.070685    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e65d540f1c56b08c62c0b47b3fd31dda757fbc542cbbbc4ff62003ee02b10edc"} err="failed to get container status \"e65d540f1c56b08c62c0b47b3fd31dda757fbc542cbbbc4ff62003ee02b10edc\": rpc error: code = NotFound desc = could not find container \"e65d540f1c56b08c62c0b47b3fd31dda757fbc542cbbbc4ff62003ee02b10edc\": container with ID starting with e65d540f1c56b08c62c0b47b3fd31dda757fbc542cbbbc4ff62003ee02b10edc not found: ID does not exist"
Sep 17 17:25:01.070743 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:01.070705    2566 scope.go:117] "RemoveContainer" containerID="24e9433d245b720a72c00d432bfe6bd75b5040c0ff8caf504b667b126bbed498"
Sep 17 17:25:01.070977 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:25:01.070957    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24e9433d245b720a72c00d432bfe6bd75b5040c0ff8caf504b667b126bbed498\": container with ID starting with 24e9433d245b720a72c00d432bfe6bd75b5040c0ff8caf504b667b126bbed498 not found: ID does not exist" containerID="24e9433d245b720a72c00d432bfe6bd75b5040c0ff8caf504b667b126bbed498"
Sep 17 17:25:01.071024 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:01.070984    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24e9433d245b720a72c00d432bfe6bd75b5040c0ff8caf504b667b126bbed498"} err="failed to get container status \"24e9433d245b720a72c00d432bfe6bd75b5040c0ff8caf504b667b126bbed498\": rpc error: code = NotFound desc = could not find container \"24e9433d245b720a72c00d432bfe6bd75b5040c0ff8caf504b667b126bbed498\": container with ID starting with 24e9433d245b720a72c00d432bfe6bd75b5040c0ff8caf504b667b126bbed498 not found: ID does not exist"
Sep 17 17:25:01.091659 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:01.091622    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a7966bf-866e-4275-a78f-bdfa178cef24-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:25:01.091659 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:01.091657    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a7966bf-866e-4275-a78f-bdfa178cef24-cabundle-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:25:01.091863 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:01.091669    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg9g9\" (UniqueName: \"kubernetes.io/projected/7a7966bf-866e-4275-a78f-bdfa178cef24-kube-api-access-fg9g9\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:25:01.091863 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:01.091682    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a7966bf-866e-4275-a78f-bdfa178cef24-tmp-dir\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:25:01.091863 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:01.091693    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a7966bf-866e-4275-a78f-bdfa178cef24-model-cache\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:25:01.091863 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:01.091704    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a7966bf-866e-4275-a78f-bdfa178cef24-dshm\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:25:01.091863 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:01.091714    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a7966bf-866e-4275-a78f-bdfa178cef24-kserve-provision-location\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:25:01.091863 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:01.091724    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a7966bf-866e-4275-a78f-bdfa178cef24-home\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:25:01.102455 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:01.102409    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq"]
Sep 17 17:25:01.109772 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:01.109742    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bf57b746-7ztrq"]
Sep 17 17:25:02.346731 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:02.346696    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a7966bf-866e-4275-a78f-bdfa178cef24" path="/var/lib/kubelet/pods/7a7966bf-866e-4275-a78f-bdfa178cef24/volumes"
Sep 17 17:25:10.362409 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:10.362313    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltxft" podUID="c40fc87c-c9f2-4f2c-99d5-1ef240215e1f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:25:10.362873 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:10.362451    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltxft"
Sep 17 17:25:20.362464 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:20.362397    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltxft" podUID="c40fc87c-c9f2-4f2c-99d5-1ef240215e1f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:25:30.363058 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:30.363021    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltxft" podUID="c40fc87c-c9f2-4f2c-99d5-1ef240215e1f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:25:30.636251 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:30.636212    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltxft"
Sep 17 17:25:30.713502 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:30.713428    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6r456\" (UniqueName: \"kubernetes.io/projected/c40fc87c-c9f2-4f2c-99d5-1ef240215e1f-kube-api-access-6r456\") pod \"c40fc87c-c9f2-4f2c-99d5-1ef240215e1f\" (UID: \"c40fc87c-c9f2-4f2c-99d5-1ef240215e1f\") "
Sep 17 17:25:30.713688 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:30.713556    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c40fc87c-c9f2-4f2c-99d5-1ef240215e1f-tls-certs\") pod \"c40fc87c-c9f2-4f2c-99d5-1ef240215e1f\" (UID: \"c40fc87c-c9f2-4f2c-99d5-1ef240215e1f\") "
Sep 17 17:25:30.715791 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:30.715752    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c40fc87c-c9f2-4f2c-99d5-1ef240215e1f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c40fc87c-c9f2-4f2c-99d5-1ef240215e1f" (UID: "c40fc87c-c9f2-4f2c-99d5-1ef240215e1f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:25:30.715791 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:30.715759    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c40fc87c-c9f2-4f2c-99d5-1ef240215e1f-kube-api-access-6r456" (OuterVolumeSpecName: "kube-api-access-6r456") pod "c40fc87c-c9f2-4f2c-99d5-1ef240215e1f" (UID: "c40fc87c-c9f2-4f2c-99d5-1ef240215e1f"). InnerVolumeSpecName "kube-api-access-6r456". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:25:30.814094 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:30.814049    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r456\" (UniqueName: \"kubernetes.io/projected/c40fc87c-c9f2-4f2c-99d5-1ef240215e1f-kube-api-access-6r456\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:25:30.814094 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:30.814084    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c40fc87c-c9f2-4f2c-99d5-1ef240215e1f-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:25:31.211354 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:31.211312    2566 generic.go:356] "Generic (PLEG): container finished" podID="c40fc87c-c9f2-4f2c-99d5-1ef240215e1f" containerID="8c7dd6dfa2156fafbdd80262fe5c6e85fabb8dad51c2a61197fb6ca3e1e6e2ed" exitCode=0
Sep 17 17:25:31.211354 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:31.211363    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltxft"
Sep 17 17:25:31.211589 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:31.211392    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltxft" event={"ID":"c40fc87c-c9f2-4f2c-99d5-1ef240215e1f","Type":"ContainerDied","Data":"8c7dd6dfa2156fafbdd80262fe5c6e85fabb8dad51c2a61197fb6ca3e1e6e2ed"}
Sep 17 17:25:31.211589 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:31.211471    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltxft" event={"ID":"c40fc87c-c9f2-4f2c-99d5-1ef240215e1f","Type":"ContainerDied","Data":"81d3026eace205940ab6f810cd003f8ab277881737cc11fc1fc27ed10b2206de"}
Sep 17 17:25:31.211589 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:31.211499    2566 scope.go:117] "RemoveContainer" containerID="8c7dd6dfa2156fafbdd80262fe5c6e85fabb8dad51c2a61197fb6ca3e1e6e2ed"
Sep 17 17:25:31.224654 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:31.224418    2566 scope.go:117] "RemoveContainer" containerID="8c7dd6dfa2156fafbdd80262fe5c6e85fabb8dad51c2a61197fb6ca3e1e6e2ed"
Sep 17 17:25:31.225171 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:25:31.225147    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c7dd6dfa2156fafbdd80262fe5c6e85fabb8dad51c2a61197fb6ca3e1e6e2ed\": container with ID starting with 8c7dd6dfa2156fafbdd80262fe5c6e85fabb8dad51c2a61197fb6ca3e1e6e2ed not found: ID does not exist" containerID="8c7dd6dfa2156fafbdd80262fe5c6e85fabb8dad51c2a61197fb6ca3e1e6e2ed"
Sep 17 17:25:31.225248 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:31.225180    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c7dd6dfa2156fafbdd80262fe5c6e85fabb8dad51c2a61197fb6ca3e1e6e2ed"} err="failed to get container status \"8c7dd6dfa2156fafbdd80262fe5c6e85fabb8dad51c2a61197fb6ca3e1e6e2ed\": rpc error: code = NotFound desc = could not find container \"8c7dd6dfa2156fafbdd80262fe5c6e85fabb8dad51c2a61197fb6ca3e1e6e2ed\": container with ID starting with 8c7dd6dfa2156fafbdd80262fe5c6e85fabb8dad51c2a61197fb6ca3e1e6e2ed not found: ID does not exist"
Sep 17 17:25:31.243486 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:31.243427    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltxft"]
Sep 17 17:25:31.251826 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:31.251789    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltxft"]
Sep 17 17:25:32.347710 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:32.347663    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c40fc87c-c9f2-4f2c-99d5-1ef240215e1f" path="/var/lib/kubelet/pods/c40fc87c-c9f2-4f2c-99d5-1ef240215e1f/volumes"
Sep 17 17:25:46.108045 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.108006    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn"]
Sep 17 17:25:46.108549 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.108338    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a7966bf-866e-4275-a78f-bdfa178cef24" containerName="main"
Sep 17 17:25:46.108549 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.108350    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a7966bf-866e-4275-a78f-bdfa178cef24" containerName="main"
Sep 17 17:25:46.108549 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.108363    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a7966bf-866e-4275-a78f-bdfa178cef24" containerName="storage-initializer"
Sep 17 17:25:46.108549 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.108369    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a7966bf-866e-4275-a78f-bdfa178cef24" containerName="storage-initializer"
Sep 17 17:25:46.108549 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.108378    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c40fc87c-c9f2-4f2c-99d5-1ef240215e1f" containerName="main"
Sep 17 17:25:46.108549 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.108384    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c40fc87c-c9f2-4f2c-99d5-1ef240215e1f" containerName="main"
Sep 17 17:25:46.108549 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.108458    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c40fc87c-c9f2-4f2c-99d5-1ef240215e1f" containerName="main"
Sep 17 17:25:46.108549 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.108468    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a7966bf-866e-4275-a78f-bdfa178cef24" containerName="main"
Sep 17 17:25:46.112865 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.112845    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn"
Sep 17 17:25:46.115683 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.115655    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 17 17:25:46.115825 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.115701    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 17 17:25:46.115958 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.115940    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 17 17:25:46.116037 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.115980    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-qfqxq\""
Sep 17 17:25:46.116037 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.116029    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:25:46.116201 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.116186    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 17 17:25:46.132204 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.132175    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn"]
Sep 17 17:25:46.241857 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.241822    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-dshm\") pod \"stop-feature-test-kserve-6cc97bc96-5hccn\" (UID: \"ff260ff7-fae7-4ef1-b4df-e028b25c48cd\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn"
Sep 17 17:25:46.242036 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.241867    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwbrx\" (UniqueName: \"kubernetes.io/projected/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-kube-api-access-pwbrx\") pod \"stop-feature-test-kserve-6cc97bc96-5hccn\" (UID: \"ff260ff7-fae7-4ef1-b4df-e028b25c48cd\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn"
Sep 17 17:25:46.242036 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.241902    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-home\") pod \"stop-feature-test-kserve-6cc97bc96-5hccn\" (UID: \"ff260ff7-fae7-4ef1-b4df-e028b25c48cd\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn"
Sep 17 17:25:46.242036 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.241955    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-kserve-provision-location\") pod \"stop-feature-test-kserve-6cc97bc96-5hccn\" (UID: \"ff260ff7-fae7-4ef1-b4df-e028b25c48cd\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn"
Sep 17 17:25:46.242036 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.241989    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-cabundle-cert\") pod \"stop-feature-test-kserve-6cc97bc96-5hccn\" (UID: \"ff260ff7-fae7-4ef1-b4df-e028b25c48cd\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn"
Sep 17 17:25:46.242036 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.242018    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-model-cache\") pod \"stop-feature-test-kserve-6cc97bc96-5hccn\" (UID: \"ff260ff7-fae7-4ef1-b4df-e028b25c48cd\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn"
Sep 17 17:25:46.242207 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.242039    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-tmp-dir\") pod \"stop-feature-test-kserve-6cc97bc96-5hccn\" (UID: \"ff260ff7-fae7-4ef1-b4df-e028b25c48cd\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn"
Sep 17 17:25:46.242207 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.242104    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-tls-certs\") pod \"stop-feature-test-kserve-6cc97bc96-5hccn\" (UID: \"ff260ff7-fae7-4ef1-b4df-e028b25c48cd\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn"
Sep 17 17:25:46.343378 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.343346    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-model-cache\") pod \"stop-feature-test-kserve-6cc97bc96-5hccn\" (UID: \"ff260ff7-fae7-4ef1-b4df-e028b25c48cd\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn"
Sep 17 17:25:46.343378 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.343379    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-tmp-dir\") pod \"stop-feature-test-kserve-6cc97bc96-5hccn\" (UID: \"ff260ff7-fae7-4ef1-b4df-e028b25c48cd\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn"
Sep 17 17:25:46.343683 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.343396    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-tls-certs\") pod \"stop-feature-test-kserve-6cc97bc96-5hccn\" (UID: \"ff260ff7-fae7-4ef1-b4df-e028b25c48cd\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn"
Sep 17 17:25:46.343683 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.343452    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-dshm\") pod \"stop-feature-test-kserve-6cc97bc96-5hccn\" (UID: \"ff260ff7-fae7-4ef1-b4df-e028b25c48cd\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn"
Sep 17 17:25:46.343683 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.343483    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwbrx\" (UniqueName: \"kubernetes.io/projected/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-kube-api-access-pwbrx\") pod \"stop-feature-test-kserve-6cc97bc96-5hccn\" (UID: \"ff260ff7-fae7-4ef1-b4df-e028b25c48cd\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn"
Sep 17 17:25:46.343683 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.343515    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-home\") pod \"stop-feature-test-kserve-6cc97bc96-5hccn\" (UID: \"ff260ff7-fae7-4ef1-b4df-e028b25c48cd\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn"
Sep 17 17:25:46.343683 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.343572    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-kserve-provision-location\") pod \"stop-feature-test-kserve-6cc97bc96-5hccn\" (UID: \"ff260ff7-fae7-4ef1-b4df-e028b25c48cd\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn"
Sep 17 17:25:46.343683 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.343604    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-cabundle-cert\") pod \"stop-feature-test-kserve-6cc97bc96-5hccn\" (UID: \"ff260ff7-fae7-4ef1-b4df-e028b25c48cd\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn"
Sep 17 17:25:46.344018 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.343851    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-model-cache\") pod \"stop-feature-test-kserve-6cc97bc96-5hccn\" (UID: \"ff260ff7-fae7-4ef1-b4df-e028b25c48cd\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn"
Sep 17 17:25:46.344018 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.343857    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-tmp-dir\") pod \"stop-feature-test-kserve-6cc97bc96-5hccn\" (UID: \"ff260ff7-fae7-4ef1-b4df-e028b25c48cd\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn"
Sep 17 17:25:46.344125 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.344098    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-home\") pod \"stop-feature-test-kserve-6cc97bc96-5hccn\" (UID: \"ff260ff7-fae7-4ef1-b4df-e028b25c48cd\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn"
Sep 17 17:25:46.344182 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.344156    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-kserve-provision-location\") pod \"stop-feature-test-kserve-6cc97bc96-5hccn\" (UID: \"ff260ff7-fae7-4ef1-b4df-e028b25c48cd\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn"
Sep 17 17:25:46.344328 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.344306    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-cabundle-cert\") pod \"stop-feature-test-kserve-6cc97bc96-5hccn\" (UID: \"ff260ff7-fae7-4ef1-b4df-e028b25c48cd\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn"
Sep 17 17:25:46.346226 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.346194    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-dshm\") pod \"stop-feature-test-kserve-6cc97bc96-5hccn\" (UID: \"ff260ff7-fae7-4ef1-b4df-e028b25c48cd\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn"
Sep 17 17:25:46.346447 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.346411    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-tls-certs\") pod \"stop-feature-test-kserve-6cc97bc96-5hccn\" (UID: \"ff260ff7-fae7-4ef1-b4df-e028b25c48cd\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn"
Sep 17 17:25:46.356858 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.356824    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwbrx\" (UniqueName: \"kubernetes.io/projected/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-kube-api-access-pwbrx\") pod \"stop-feature-test-kserve-6cc97bc96-5hccn\" (UID: \"ff260ff7-fae7-4ef1-b4df-e028b25c48cd\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn"
Sep 17 17:25:46.423582 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.423487    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn"
Sep 17 17:25:46.518552 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.518516    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nj22k"]
Sep 17 17:25:46.522109 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.522084    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nj22k"
Sep 17 17:25:46.527226 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.527203    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-g84dz\""
Sep 17 17:25:46.551963 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.551922    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nj22k"]
Sep 17 17:25:46.646746 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.646709    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb8px\" (UniqueName: \"kubernetes.io/projected/43fd6571-80e0-4f9d-9d54-0ac7bd88efe2-kube-api-access-cb8px\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nj22k\" (UID: \"43fd6571-80e0-4f9d-9d54-0ac7bd88efe2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nj22k"
Sep 17 17:25:46.646952 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.646789    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43fd6571-80e0-4f9d-9d54-0ac7bd88efe2-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nj22k\" (UID: \"43fd6571-80e0-4f9d-9d54-0ac7bd88efe2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nj22k"
Sep 17 17:25:46.721000 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.720917    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn"]
Sep 17 17:25:46.724511 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:25:46.724486    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff260ff7_fae7_4ef1_b4df_e028b25c48cd.slice/crio-1f8239226cd4dbbbd3775b837c6d13a33f60af0f330ed00173d8de169bf720b9 WatchSource:0}: Error finding container 1f8239226cd4dbbbd3775b837c6d13a33f60af0f330ed00173d8de169bf720b9: Status 404 returned error can't find the container with id 1f8239226cd4dbbbd3775b837c6d13a33f60af0f330ed00173d8de169bf720b9
Sep 17 17:25:46.747335 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.747305    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43fd6571-80e0-4f9d-9d54-0ac7bd88efe2-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nj22k\" (UID: \"43fd6571-80e0-4f9d-9d54-0ac7bd88efe2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nj22k"
Sep 17 17:25:46.747477 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.747356    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb8px\" (UniqueName: \"kubernetes.io/projected/43fd6571-80e0-4f9d-9d54-0ac7bd88efe2-kube-api-access-cb8px\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nj22k\" (UID: \"43fd6571-80e0-4f9d-9d54-0ac7bd88efe2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nj22k"
Sep 17 17:25:46.749963 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.749941    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43fd6571-80e0-4f9d-9d54-0ac7bd88efe2-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nj22k\" (UID: \"43fd6571-80e0-4f9d-9d54-0ac7bd88efe2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nj22k"
Sep 17 17:25:46.761917 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.761894    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb8px\" (UniqueName: \"kubernetes.io/projected/43fd6571-80e0-4f9d-9d54-0ac7bd88efe2-kube-api-access-cb8px\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nj22k\" (UID: \"43fd6571-80e0-4f9d-9d54-0ac7bd88efe2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nj22k"
Sep 17 17:25:46.834317 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:46.834279    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nj22k"
Sep 17 17:25:47.120333 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:47.120303    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nj22k"]
Sep 17 17:25:47.121738 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:25:47.121706    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43fd6571_80e0_4f9d_9d54_0ac7bd88efe2.slice/crio-b928625b50e82cb3e8f6cfa690e5e88153d7764af2c6fe4a7dd5497eea0582fa WatchSource:0}: Error finding container b928625b50e82cb3e8f6cfa690e5e88153d7764af2c6fe4a7dd5497eea0582fa: Status 404 returned error can't find the container with id b928625b50e82cb3e8f6cfa690e5e88153d7764af2c6fe4a7dd5497eea0582fa
Sep 17 17:25:47.294371 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:47.294332    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn" event={"ID":"ff260ff7-fae7-4ef1-b4df-e028b25c48cd","Type":"ContainerStarted","Data":"93a086172090e780d956fe5556315618e11d2b6758345d6c3fd3f732bf56742b"}
Sep 17 17:25:47.294872 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:47.294818    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn" event={"ID":"ff260ff7-fae7-4ef1-b4df-e028b25c48cd","Type":"ContainerStarted","Data":"1f8239226cd4dbbbd3775b837c6d13a33f60af0f330ed00173d8de169bf720b9"}
Sep 17 17:25:47.303271 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:47.303229    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nj22k" event={"ID":"43fd6571-80e0-4f9d-9d54-0ac7bd88efe2","Type":"ContainerStarted","Data":"7f5f67d9a0635ebf0c2114a47ca6171a84cb9d2127fcef7496d602f12e9c7112"}
Sep 17 17:25:47.303481 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:47.303332    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nj22k" event={"ID":"43fd6571-80e0-4f9d-9d54-0ac7bd88efe2","Type":"ContainerStarted","Data":"b928625b50e82cb3e8f6cfa690e5e88153d7764af2c6fe4a7dd5497eea0582fa"}
Sep 17 17:25:47.303481 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:47.303364    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nj22k"
Sep 17 17:25:49.326613 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:49.326576    2566 generic.go:356] "Generic (PLEG): container finished" podID="ff260ff7-fae7-4ef1-b4df-e028b25c48cd" containerID="93a086172090e780d956fe5556315618e11d2b6758345d6c3fd3f732bf56742b" exitCode=0
Sep 17 17:25:49.327014 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:49.326664    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn" event={"ID":"ff260ff7-fae7-4ef1-b4df-e028b25c48cd","Type":"ContainerDied","Data":"93a086172090e780d956fe5556315618e11d2b6758345d6c3fd3f732bf56742b"}
Sep 17 17:25:49.364194 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:49.364133    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nj22k" podStartSLOduration=3.364113324 podStartE2EDuration="3.364113324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:25:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:25:47.399588902 +0000 UTC m=+2633.294068053" watchObservedRunningTime="2026-09-17 17:25:49.364113324 +0000 UTC m=+2635.258592464"
Sep 17 17:25:50.335190 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:50.335156    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn" event={"ID":"ff260ff7-fae7-4ef1-b4df-e028b25c48cd","Type":"ContainerStarted","Data":"8233815aab935b2f2b9de7e7fe635f8603e1f98c057f6986d63dab0e2c18faf6"}
Sep 17 17:25:50.373353 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:50.373283    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn" podStartSLOduration=2.047362324 podStartE2EDuration="4.37325764s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.325895316s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:25:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:25:50.370540506 +0000 UTC m=+2636.265019645" watchObservedRunningTime="2026-09-17 17:25:50.37325764 +0000 UTC m=+2636.267736779"
Sep 17 17:25:50.424101 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:50.424063    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn"
Sep 17 17:25:56.424125 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:56.424080    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn"
Sep 17 17:25:56.425884 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:25:56.425850    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn" podUID="ff260ff7-fae7-4ef1-b4df-e028b25c48cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Sep 17 17:26:06.424524 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:26:06.424479    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn" podUID="ff260ff7-fae7-4ef1-b4df-e028b25c48cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Sep 17 17:26:16.424404 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:26:16.424358    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn" podUID="ff260ff7-fae7-4ef1-b4df-e028b25c48cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Sep 17 17:26:18.320045 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:26:18.320012    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nj22k"
Sep 17 17:26:26.424786 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:26:26.424739    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn" podUID="ff260ff7-fae7-4ef1-b4df-e028b25c48cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Sep 17 17:26:36.424947 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:26:36.424846    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn" podUID="ff260ff7-fae7-4ef1-b4df-e028b25c48cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Sep 17 17:26:46.424427 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:26:46.424372    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn" podUID="ff260ff7-fae7-4ef1-b4df-e028b25c48cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Sep 17 17:26:56.424411 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:26:56.424364    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn" podUID="ff260ff7-fae7-4ef1-b4df-e028b25c48cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Sep 17 17:26:56.526944 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:26:56.526914    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gblhr_91477834-6648-4e75-8530-e01bdee5e57e/ovn-acl-logging/0.log"
Sep 17 17:26:56.538348 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:26:56.538321    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gblhr_91477834-6648-4e75-8530-e01bdee5e57e/ovn-acl-logging/0.log"
Sep 17 17:27:06.424929 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:06.424878    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn" podUID="ff260ff7-fae7-4ef1-b4df-e028b25c48cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Sep 17 17:27:15.284092 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:15.284048    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"]
Sep 17 17:27:15.284603 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:15.284299    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct" podUID="5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd" containerName="istio-proxy" containerID="cri-o://9caa62b55901497bb0733566f22d60d02b93105534809b3f5f7a8f6f7380b50b" gracePeriod=30
Sep 17 17:27:15.953019 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:15.952979    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d"]
Sep 17 17:27:15.957189 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:15.957163    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d"
Sep 17 17:27:15.961750 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:15.961723    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-r6hb9\""
Sep 17 17:27:15.962015 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:15.961996    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 17 17:27:15.962141 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:15.962119    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:27:15.962682 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:15.962664    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 17 17:27:15.962852 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:15.962831    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 17 17:27:15.962931 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:15.962883    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 17 17:27:15.983711 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:15.983672    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d"]
Sep 17 17:27:16.086912 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:16.086874    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d\" (UID: \"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d"
Sep 17 17:27:16.086912 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:16.086915    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d\" (UID: \"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d"
Sep 17 17:27:16.087131 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:16.086974    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpxjn\" (UniqueName: \"kubernetes.io/projected/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-kube-api-access-wpxjn\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d\" (UID: \"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d"
Sep 17 17:27:16.087131 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:16.087047    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d\" (UID: \"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d"
Sep 17 17:27:16.087131 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:16.087077    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d\" (UID: \"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d"
Sep 17 17:27:16.087131 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:16.087102    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d\" (UID: \"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d"
Sep 17 17:27:16.087131 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:16.087128    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d\" (UID: \"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d"
Sep 17 17:27:16.087296 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:16.087175    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d\" (UID: \"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d"
Sep 17 17:27:16.188669 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:16.188623    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d\" (UID: \"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d"
Sep 17 17:27:16.188864 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:16.188692    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d\" (UID: \"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d"
Sep 17 17:27:16.188864 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:16.188736    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d\" (UID: \"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d"
Sep 17 17:27:16.188990 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:16.188880    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d\" (UID: \"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d"
Sep 17 17:27:16.188990 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:16.188933    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d\" (UID: \"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d"
Sep 17 17:27:16.188990 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:16.188979    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d\" (UID: \"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d"
Sep 17 17:27:16.189146 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:16.189004    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d\" (UID: \"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d"
Sep 17 17:27:16.189146 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:16.189058    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpxjn\" (UniqueName: \"kubernetes.io/projected/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-kube-api-access-wpxjn\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d\" (UID: \"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d"
Sep 17 17:27:16.189146 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:16.189092    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d\" (UID: \"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d"
Sep 17 17:27:16.189348 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:16.189158    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d\" (UID: \"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d"
Sep 17 17:27:16.189348 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:16.189272    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d\" (UID: \"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d"
Sep 17 17:27:16.189471 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:16.189366    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d\" (UID: \"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d"
Sep 17 17:27:16.189471 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:16.189396    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d\" (UID: \"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d"
Sep 17 17:27:16.191668 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:16.191639    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d\" (UID: \"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d"
Sep 17 17:27:16.192046 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:16.192023    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d\" (UID: \"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d"
Sep 17 17:27:16.201386 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:16.201355    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpxjn\" (UniqueName: \"kubernetes.io/projected/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-kube-api-access-wpxjn\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d\" (UID: \"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d"
Sep 17 17:27:16.269974 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:16.269875    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d"
Sep 17 17:27:16.423997 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:16.423950    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn" podUID="ff260ff7-fae7-4ef1-b4df-e028b25c48cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Sep 17 17:27:16.578960 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:16.578931    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d"]
Sep 17 17:27:16.580919 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:27:16.580884    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69fb7dbd_18f1_408c_b76e_3d3d1aeb5a5c.slice/crio-1aa3c6834a83c48848d8aec805b5b8d56e750234117b6cd9780b49f2bd93a964 WatchSource:0}: Error finding container 1aa3c6834a83c48848d8aec805b5b8d56e750234117b6cd9780b49f2bd93a964: Status 404 returned error can't find the container with id 1aa3c6834a83c48848d8aec805b5b8d56e750234117b6cd9780b49f2bd93a964
Sep 17 17:27:16.822565 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:16.822528    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d" event={"ID":"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c","Type":"ContainerStarted","Data":"668f568b6c7af701a5cc7aa39564a3d24abf21ebd5361c6a236a137ad72966cf"}
Sep 17 17:27:16.822756 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:16.822583    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d" event={"ID":"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c","Type":"ContainerStarted","Data":"1aa3c6834a83c48848d8aec805b5b8d56e750234117b6cd9780b49f2bd93a964"}
Sep 17 17:27:19.843187 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:19.843153    2566 generic.go:356] "Generic (PLEG): container finished" podID="69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c" containerID="668f568b6c7af701a5cc7aa39564a3d24abf21ebd5361c6a236a137ad72966cf" exitCode=0
Sep 17 17:27:19.843626 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:19.843202    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d" event={"ID":"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c","Type":"ContainerDied","Data":"668f568b6c7af701a5cc7aa39564a3d24abf21ebd5361c6a236a137ad72966cf"}
Sep 17 17:27:20.555415 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:20.555326    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"
Sep 17 17:27:20.733388 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:20.733350    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-istio-podinfo\") pod \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\" (UID: \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\") "
Sep 17 17:27:20.733579 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:20.733399    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-istio-data\") pod \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\" (UID: \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\") "
Sep 17 17:27:20.733579 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:20.733426    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-istiod-ca-cert\") pod \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\" (UID: \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\") "
Sep 17 17:27:20.733579 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:20.733473    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-workload-certs\") pod \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\" (UID: \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\") "
Sep 17 17:27:20.733579 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:20.733514    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-workload-socket\") pod \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\" (UID: \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\") "
Sep 17 17:27:20.733579 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:20.733552    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-istio-token\") pod \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\" (UID: \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\") "
Sep 17 17:27:20.733749 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:20.733592    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-credential-socket\") pod \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\" (UID: \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\") "
Sep 17 17:27:20.733749 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:20.733630    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-istio-envoy\") pod \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\" (UID: \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\") "
Sep 17 17:27:20.733749 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:20.733679    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mkwq5\" (UniqueName: \"kubernetes.io/projected/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-kube-api-access-mkwq5\") pod \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\" (UID: \"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd\") "
Sep 17 17:27:20.734247 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:20.733914    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd" (UID: "5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:27:20.734247 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:20.734006    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd" (UID: "5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:27:20.734247 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:20.734092    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd" (UID: "5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:27:20.734247 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:20.734128    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd" (UID: "5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:27:20.734247 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:20.734245    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-istio-data" (OuterVolumeSpecName: "istio-data") pod "5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd" (UID: "5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:27:20.736950 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:20.736919    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-kube-api-access-mkwq5" (OuterVolumeSpecName: "kube-api-access-mkwq5") pod "5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd" (UID: "5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd"). InnerVolumeSpecName "kube-api-access-mkwq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:27:20.736950 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:20.736932    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd" (UID: "5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:27:20.736950 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:20.736941    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-istio-token" (OuterVolumeSpecName: "istio-token") pod "5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd" (UID: "5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:27:20.737336 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:20.737319    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd" (UID: "5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 17:27:20.834711 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:20.834607    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkwq5\" (UniqueName: \"kubernetes.io/projected/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-kube-api-access-mkwq5\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:27:20.834711 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:20.834651    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-istio-podinfo\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:27:20.834711 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:20.834668    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-istio-data\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:27:20.834711 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:20.834679    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-istiod-ca-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:27:20.834711 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:20.834691    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-workload-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:27:20.834711 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:20.834703    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-workload-socket\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:27:20.834711 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:20.834714    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-istio-token\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:27:20.835040 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:20.834726    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-credential-socket\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:27:20.835040 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:20.834737    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd-istio-envoy\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:27:20.852444 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:20.852391    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d" event={"ID":"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c","Type":"ContainerStarted","Data":"6eef73d13be107001e7c44844a718cd9b868270db1011ac589ad351ca360914d"}
Sep 17 17:27:20.854545 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:20.854516    2566 generic.go:356] "Generic (PLEG): container finished" podID="5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd" containerID="9caa62b55901497bb0733566f22d60d02b93105534809b3f5f7a8f6f7380b50b" exitCode=0
Sep 17 17:27:20.854743 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:20.854565    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct" event={"ID":"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd","Type":"ContainerDied","Data":"9caa62b55901497bb0733566f22d60d02b93105534809b3f5f7a8f6f7380b50b"}
Sep 17 17:27:20.854743 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:20.854608    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct" event={"ID":"5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd","Type":"ContainerDied","Data":"7e4caf78f6963a321dbba2e20f73af51bc0511681ccf43e7c4c5636d39a77eb1"}
Sep 17 17:27:20.854743 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:20.854632    2566 scope.go:117] "RemoveContainer" containerID="9caa62b55901497bb0733566f22d60d02b93105534809b3f5f7a8f6f7380b50b"
Sep 17 17:27:20.854743 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:20.854572    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"
Sep 17 17:27:20.865885 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:20.865862    2566 scope.go:117] "RemoveContainer" containerID="9c5aa51709e60ee2a48bf3ba96849079bef874a22b746612c5fc8b780f2092d2"
Sep 17 17:27:20.875385 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:20.875355    2566 scope.go:117] "RemoveContainer" containerID="9caa62b55901497bb0733566f22d60d02b93105534809b3f5f7a8f6f7380b50b"
Sep 17 17:27:20.875761 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:27:20.875730    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9caa62b55901497bb0733566f22d60d02b93105534809b3f5f7a8f6f7380b50b\": container with ID starting with 9caa62b55901497bb0733566f22d60d02b93105534809b3f5f7a8f6f7380b50b not found: ID does not exist" containerID="9caa62b55901497bb0733566f22d60d02b93105534809b3f5f7a8f6f7380b50b"
Sep 17 17:27:20.875846 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:20.875773    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9caa62b55901497bb0733566f22d60d02b93105534809b3f5f7a8f6f7380b50b"} err="failed to get container status \"9caa62b55901497bb0733566f22d60d02b93105534809b3f5f7a8f6f7380b50b\": rpc error: code = NotFound desc = could not find container \"9caa62b55901497bb0733566f22d60d02b93105534809b3f5f7a8f6f7380b50b\": container with ID starting with 9caa62b55901497bb0733566f22d60d02b93105534809b3f5f7a8f6f7380b50b not found: ID does not exist"
Sep 17 17:27:20.875846 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:20.875800    2566 scope.go:117] "RemoveContainer" containerID="9c5aa51709e60ee2a48bf3ba96849079bef874a22b746612c5fc8b780f2092d2"
Sep 17 17:27:20.876108 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:27:20.876084    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c5aa51709e60ee2a48bf3ba96849079bef874a22b746612c5fc8b780f2092d2\": container with ID starting with 9c5aa51709e60ee2a48bf3ba96849079bef874a22b746612c5fc8b780f2092d2 not found: ID does not exist" containerID="9c5aa51709e60ee2a48bf3ba96849079bef874a22b746612c5fc8b780f2092d2"
Sep 17 17:27:20.876247 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:20.876111    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c5aa51709e60ee2a48bf3ba96849079bef874a22b746612c5fc8b780f2092d2"} err="failed to get container status \"9c5aa51709e60ee2a48bf3ba96849079bef874a22b746612c5fc8b780f2092d2\": rpc error: code = NotFound desc = could not find container \"9c5aa51709e60ee2a48bf3ba96849079bef874a22b746612c5fc8b780f2092d2\": container with ID starting with 9c5aa51709e60ee2a48bf3ba96849079bef874a22b746612c5fc8b780f2092d2 not found: ID does not exist"
Sep 17 17:27:20.890803 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:20.890748    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d" podStartSLOduration=3.3968634 podStartE2EDuration="5.890729178s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.493865778s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:27:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:27:20.887581284 +0000 UTC m=+2726.782060424" watchObservedRunningTime="2026-09-17 17:27:20.890729178 +0000 UTC m=+2726.785208323"
Sep 17 17:27:20.912133 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:20.912091    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"]
Sep 17 17:27:20.921916 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:20.921879    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-7xcct"]
Sep 17 17:27:22.348637 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:22.348601    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd" path="/var/lib/kubelet/pods/5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd/volumes"
Sep 17 17:27:26.270633 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:26.270591    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d"
Sep 17 17:27:26.271053 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:26.270666    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d"
Sep 17 17:27:26.272309 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:26.272276    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d" podUID="69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 17 17:27:26.424206 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:26.424157    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn" podUID="ff260ff7-fae7-4ef1-b4df-e028b25c48cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Sep 17 17:27:36.271061 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:36.271002    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d" podUID="69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 17 17:27:36.424868 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:36.424818    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn" podUID="ff260ff7-fae7-4ef1-b4df-e028b25c48cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Sep 17 17:27:46.271282 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:46.271228    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d" podUID="69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 17 17:27:46.434295 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:46.434259    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn"
Sep 17 17:27:46.442450 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:46.442408    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn"
Sep 17 17:27:48.800380 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:48.800345    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn"]
Sep 17 17:27:48.911227 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:48.911188    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nj22k"]
Sep 17 17:27:56.270303 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:56.270257    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d" podUID="69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 17 17:27:58.319981 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:27:58.319939    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nj22k" podUID="43fd6571-80e0-4f9d-9d54-0ac7bd88efe2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:28:03.823630 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:03.823567    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn" podUID="ff260ff7-fae7-4ef1-b4df-e028b25c48cd" containerName="main" containerID="cri-o://8233815aab935b2f2b9de7e7fe635f8603e1f98c057f6986d63dab0e2c18faf6" gracePeriod=45
Sep 17 17:28:03.931661 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:03.931586    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nj22k" podUID="43fd6571-80e0-4f9d-9d54-0ac7bd88efe2" containerName="main" containerID="cri-o://7f5f67d9a0635ebf0c2114a47ca6171a84cb9d2127fcef7496d602f12e9c7112" gracePeriod=45
Sep 17 17:28:04.435417 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:04.435372    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn" podUID="ff260ff7-fae7-4ef1-b4df-e028b25c48cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Sep 17 17:28:04.596319 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:04.596286    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn"
Sep 17 17:28:04.718221 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:04.718187    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-dshm\") pod \"ff260ff7-fae7-4ef1-b4df-e028b25c48cd\" (UID: \"ff260ff7-fae7-4ef1-b4df-e028b25c48cd\") "
Sep 17 17:28:04.718221 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:04.718226    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-tmp-dir\") pod \"ff260ff7-fae7-4ef1-b4df-e028b25c48cd\" (UID: \"ff260ff7-fae7-4ef1-b4df-e028b25c48cd\") "
Sep 17 17:28:04.718525 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:04.718253    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-home\") pod \"ff260ff7-fae7-4ef1-b4df-e028b25c48cd\" (UID: \"ff260ff7-fae7-4ef1-b4df-e028b25c48cd\") "
Sep 17 17:28:04.718525 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:04.718310    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-cabundle-cert\") pod \"ff260ff7-fae7-4ef1-b4df-e028b25c48cd\" (UID: \"ff260ff7-fae7-4ef1-b4df-e028b25c48cd\") "
Sep 17 17:28:04.718525 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:04.718367    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-tls-certs\") pod \"ff260ff7-fae7-4ef1-b4df-e028b25c48cd\" (UID: \"ff260ff7-fae7-4ef1-b4df-e028b25c48cd\") "
Sep 17 17:28:04.718525 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:04.718415    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwbrx\" (UniqueName: \"kubernetes.io/projected/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-kube-api-access-pwbrx\") pod \"ff260ff7-fae7-4ef1-b4df-e028b25c48cd\" (UID: \"ff260ff7-fae7-4ef1-b4df-e028b25c48cd\") "
Sep 17 17:28:04.718525 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:04.718480    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-model-cache\") pod \"ff260ff7-fae7-4ef1-b4df-e028b25c48cd\" (UID: \"ff260ff7-fae7-4ef1-b4df-e028b25c48cd\") "
Sep 17 17:28:04.718525 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:04.718508    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-kserve-provision-location\") pod \"ff260ff7-fae7-4ef1-b4df-e028b25c48cd\" (UID: \"ff260ff7-fae7-4ef1-b4df-e028b25c48cd\") "
Sep 17 17:28:04.718862 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:04.718690    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ff260ff7-fae7-4ef1-b4df-e028b25c48cd" (UID: "ff260ff7-fae7-4ef1-b4df-e028b25c48cd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:28:04.718862 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:04.718787    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-cabundle-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:28:04.719001 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:04.718974    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-model-cache" (OuterVolumeSpecName: "model-cache") pod "ff260ff7-fae7-4ef1-b4df-e028b25c48cd" (UID: "ff260ff7-fae7-4ef1-b4df-e028b25c48cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:28:04.719122 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:04.719089    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-home" (OuterVolumeSpecName: "home") pod "ff260ff7-fae7-4ef1-b4df-e028b25c48cd" (UID: "ff260ff7-fae7-4ef1-b4df-e028b25c48cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:28:04.721164 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:04.721137    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-dshm" (OuterVolumeSpecName: "dshm") pod "ff260ff7-fae7-4ef1-b4df-e028b25c48cd" (UID: "ff260ff7-fae7-4ef1-b4df-e028b25c48cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:28:04.721508 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:04.721478    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-kube-api-access-pwbrx" (OuterVolumeSpecName: "kube-api-access-pwbrx") pod "ff260ff7-fae7-4ef1-b4df-e028b25c48cd" (UID: "ff260ff7-fae7-4ef1-b4df-e028b25c48cd"). InnerVolumeSpecName "kube-api-access-pwbrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:28:04.721625 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:04.721520    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ff260ff7-fae7-4ef1-b4df-e028b25c48cd" (UID: "ff260ff7-fae7-4ef1-b4df-e028b25c48cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:28:04.731355 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:04.731319    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ff260ff7-fae7-4ef1-b4df-e028b25c48cd" (UID: "ff260ff7-fae7-4ef1-b4df-e028b25c48cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:28:04.742661 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:04.742624    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ff260ff7-fae7-4ef1-b4df-e028b25c48cd" (UID: "ff260ff7-fae7-4ef1-b4df-e028b25c48cd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:28:04.819725 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:04.819672    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:28:04.819725 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:04.819708    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwbrx\" (UniqueName: \"kubernetes.io/projected/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-kube-api-access-pwbrx\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:28:04.819725 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:04.819718    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-model-cache\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:28:04.819725 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:04.819727    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-kserve-provision-location\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:28:04.819725 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:04.819735    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-dshm\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:28:04.819725 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:04.819743    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-tmp-dir\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:28:04.820231 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:04.819751    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff260ff7-fae7-4ef1-b4df-e028b25c48cd-home\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:28:05.113328 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:05.113215    2566 generic.go:356] "Generic (PLEG): container finished" podID="ff260ff7-fae7-4ef1-b4df-e028b25c48cd" containerID="8233815aab935b2f2b9de7e7fe635f8603e1f98c057f6986d63dab0e2c18faf6" exitCode=0
Sep 17 17:28:05.113328 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:05.113277    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn"
Sep 17 17:28:05.113328 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:05.113306    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn" event={"ID":"ff260ff7-fae7-4ef1-b4df-e028b25c48cd","Type":"ContainerDied","Data":"8233815aab935b2f2b9de7e7fe635f8603e1f98c057f6986d63dab0e2c18faf6"}
Sep 17 17:28:05.113906 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:05.113373    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn" event={"ID":"ff260ff7-fae7-4ef1-b4df-e028b25c48cd","Type":"ContainerDied","Data":"1f8239226cd4dbbbd3775b837c6d13a33f60af0f330ed00173d8de169bf720b9"}
Sep 17 17:28:05.113906 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:05.113404    2566 scope.go:117] "RemoveContainer" containerID="8233815aab935b2f2b9de7e7fe635f8603e1f98c057f6986d63dab0e2c18faf6"
Sep 17 17:28:05.124104 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:05.124080    2566 scope.go:117] "RemoveContainer" containerID="93a086172090e780d956fe5556315618e11d2b6758345d6c3fd3f732bf56742b"
Sep 17 17:28:05.137089 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:05.137059    2566 scope.go:117] "RemoveContainer" containerID="8233815aab935b2f2b9de7e7fe635f8603e1f98c057f6986d63dab0e2c18faf6"
Sep 17 17:28:05.137665 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:28:05.137639    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8233815aab935b2f2b9de7e7fe635f8603e1f98c057f6986d63dab0e2c18faf6\": container with ID starting with 8233815aab935b2f2b9de7e7fe635f8603e1f98c057f6986d63dab0e2c18faf6 not found: ID does not exist" containerID="8233815aab935b2f2b9de7e7fe635f8603e1f98c057f6986d63dab0e2c18faf6"
Sep 17 17:28:05.137795 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:05.137674    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8233815aab935b2f2b9de7e7fe635f8603e1f98c057f6986d63dab0e2c18faf6"} err="failed to get container status \"8233815aab935b2f2b9de7e7fe635f8603e1f98c057f6986d63dab0e2c18faf6\": rpc error: code = NotFound desc = could not find container \"8233815aab935b2f2b9de7e7fe635f8603e1f98c057f6986d63dab0e2c18faf6\": container with ID starting with 8233815aab935b2f2b9de7e7fe635f8603e1f98c057f6986d63dab0e2c18faf6 not found: ID does not exist"
Sep 17 17:28:05.137795 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:05.137693    2566 scope.go:117] "RemoveContainer" containerID="93a086172090e780d956fe5556315618e11d2b6758345d6c3fd3f732bf56742b"
Sep 17 17:28:05.137996 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:28:05.137969    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93a086172090e780d956fe5556315618e11d2b6758345d6c3fd3f732bf56742b\": container with ID starting with 93a086172090e780d956fe5556315618e11d2b6758345d6c3fd3f732bf56742b not found: ID does not exist" containerID="93a086172090e780d956fe5556315618e11d2b6758345d6c3fd3f732bf56742b"
Sep 17 17:28:05.138067 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:05.138003    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93a086172090e780d956fe5556315618e11d2b6758345d6c3fd3f732bf56742b"} err="failed to get container status \"93a086172090e780d956fe5556315618e11d2b6758345d6c3fd3f732bf56742b\": rpc error: code = NotFound desc = could not find container \"93a086172090e780d956fe5556315618e11d2b6758345d6c3fd3f732bf56742b\": container with ID starting with 93a086172090e780d956fe5556315618e11d2b6758345d6c3fd3f732bf56742b not found: ID does not exist"
Sep 17 17:28:05.143244 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:05.143211    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn"]
Sep 17 17:28:05.148412 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:05.148379    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-5hccn"]
Sep 17 17:28:06.271219 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:06.271176    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d" podUID="69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 17 17:28:06.347609 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:06.347576    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff260ff7-fae7-4ef1-b4df-e028b25c48cd" path="/var/lib/kubelet/pods/ff260ff7-fae7-4ef1-b4df-e028b25c48cd/volumes"
Sep 17 17:28:08.320027 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:08.319981    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nj22k" podUID="43fd6571-80e0-4f9d-9d54-0ac7bd88efe2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:28:16.271120 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:16.271059    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d" podUID="69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 17 17:28:18.319734 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:18.319692    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nj22k" podUID="43fd6571-80e0-4f9d-9d54-0ac7bd88efe2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:28:18.320187 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:18.319807    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nj22k"
Sep 17 17:28:26.270686 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:26.270635    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d" podUID="69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 17 17:28:28.319513 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:28.319476    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nj22k" podUID="43fd6571-80e0-4f9d-9d54-0ac7bd88efe2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:28:29.087666 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:29.087619    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh"]
Sep 17 17:28:29.088259 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:29.088235    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd" containerName="istio-proxy"
Sep 17 17:28:29.088338 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:29.088263    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd" containerName="istio-proxy"
Sep 17 17:28:29.088338 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:29.088277    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd" containerName="istio-proxy"
Sep 17 17:28:29.088338 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:29.088287    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd" containerName="istio-proxy"
Sep 17 17:28:29.088338 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:29.088320    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff260ff7-fae7-4ef1-b4df-e028b25c48cd" containerName="storage-initializer"
Sep 17 17:28:29.088338 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:29.088329    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff260ff7-fae7-4ef1-b4df-e028b25c48cd" containerName="storage-initializer"
Sep 17 17:28:29.088531 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:29.088345    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff260ff7-fae7-4ef1-b4df-e028b25c48cd" containerName="main"
Sep 17 17:28:29.088531 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:29.088353    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff260ff7-fae7-4ef1-b4df-e028b25c48cd" containerName="main"
Sep 17 17:28:29.088531 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:29.088452    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd" containerName="istio-proxy"
Sep 17 17:28:29.088531 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:29.088468    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e5e2ade-998c-4ec1-a2a9-ae7ef2dc2acd" containerName="istio-proxy"
Sep 17 17:28:29.088531 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:29.088478    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff260ff7-fae7-4ef1-b4df-e028b25c48cd" containerName="main"
Sep 17 17:28:29.091684 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:29.091649    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh"
Sep 17 17:28:29.094260 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:29.094238    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 17 17:28:29.094389 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:29.094242    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:28:29.104881 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:29.104852    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh"]
Sep 17 17:28:29.122579 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:29.122547    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-tls-certs\") pod \"stop-feature-test-kserve-6cc97bc96-655hh\" (UID: \"fa12b5a7-5b90-4b31-a5ed-db75b85aa258\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh"
Sep 17 17:28:29.122764 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:29.122619    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-cabundle-cert\") pod \"stop-feature-test-kserve-6cc97bc96-655hh\" (UID: \"fa12b5a7-5b90-4b31-a5ed-db75b85aa258\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh"
Sep 17 17:28:29.122764 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:29.122651    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-tmp-dir\") pod \"stop-feature-test-kserve-6cc97bc96-655hh\" (UID: \"fa12b5a7-5b90-4b31-a5ed-db75b85aa258\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh"
Sep 17 17:28:29.123095 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:29.123048    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-model-cache\") pod \"stop-feature-test-kserve-6cc97bc96-655hh\" (UID: \"fa12b5a7-5b90-4b31-a5ed-db75b85aa258\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh"
Sep 17 17:28:29.123209 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:29.123118    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-home\") pod \"stop-feature-test-kserve-6cc97bc96-655hh\" (UID: \"fa12b5a7-5b90-4b31-a5ed-db75b85aa258\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh"
Sep 17 17:28:29.123209 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:29.123164    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-dshm\") pod \"stop-feature-test-kserve-6cc97bc96-655hh\" (UID: \"fa12b5a7-5b90-4b31-a5ed-db75b85aa258\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh"
Sep 17 17:28:29.123209 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:29.123192    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nk2z\" (UniqueName: \"kubernetes.io/projected/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-kube-api-access-6nk2z\") pod \"stop-feature-test-kserve-6cc97bc96-655hh\" (UID: \"fa12b5a7-5b90-4b31-a5ed-db75b85aa258\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh"
Sep 17 17:28:29.123340 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:29.123247    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-kserve-provision-location\") pod \"stop-feature-test-kserve-6cc97bc96-655hh\" (UID: \"fa12b5a7-5b90-4b31-a5ed-db75b85aa258\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh"
Sep 17 17:28:29.224057 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:29.224010    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nk2z\" (UniqueName: \"kubernetes.io/projected/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-kube-api-access-6nk2z\") pod \"stop-feature-test-kserve-6cc97bc96-655hh\" (UID: \"fa12b5a7-5b90-4b31-a5ed-db75b85aa258\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh"
Sep 17 17:28:29.224269 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:29.224094    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-kserve-provision-location\") pod \"stop-feature-test-kserve-6cc97bc96-655hh\" (UID: \"fa12b5a7-5b90-4b31-a5ed-db75b85aa258\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh"
Sep 17 17:28:29.224269 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:29.224122    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-tls-certs\") pod \"stop-feature-test-kserve-6cc97bc96-655hh\" (UID: \"fa12b5a7-5b90-4b31-a5ed-db75b85aa258\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh"
Sep 17 17:28:29.224269 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:29.224161    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-cabundle-cert\") pod \"stop-feature-test-kserve-6cc97bc96-655hh\" (UID: \"fa12b5a7-5b90-4b31-a5ed-db75b85aa258\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh"
Sep 17 17:28:29.224269 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:29.224188    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-tmp-dir\") pod \"stop-feature-test-kserve-6cc97bc96-655hh\" (UID: \"fa12b5a7-5b90-4b31-a5ed-db75b85aa258\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh"
Sep 17 17:28:29.224269 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:29.224243    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-model-cache\") pod \"stop-feature-test-kserve-6cc97bc96-655hh\" (UID: \"fa12b5a7-5b90-4b31-a5ed-db75b85aa258\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh"
Sep 17 17:28:29.224616 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:29.224280    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-home\") pod \"stop-feature-test-kserve-6cc97bc96-655hh\" (UID: \"fa12b5a7-5b90-4b31-a5ed-db75b85aa258\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh"
Sep 17 17:28:29.224616 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:29.224332    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-dshm\") pod \"stop-feature-test-kserve-6cc97bc96-655hh\" (UID: \"fa12b5a7-5b90-4b31-a5ed-db75b85aa258\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh"
Sep 17 17:28:29.224727 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:29.224660    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-kserve-provision-location\") pod \"stop-feature-test-kserve-6cc97bc96-655hh\" (UID: \"fa12b5a7-5b90-4b31-a5ed-db75b85aa258\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh"
Sep 17 17:28:29.224727 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:29.224693    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-tmp-dir\") pod \"stop-feature-test-kserve-6cc97bc96-655hh\" (UID: \"fa12b5a7-5b90-4b31-a5ed-db75b85aa258\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh"
Sep 17 17:28:29.224811 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:29.224777    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-model-cache\") pod \"stop-feature-test-kserve-6cc97bc96-655hh\" (UID: \"fa12b5a7-5b90-4b31-a5ed-db75b85aa258\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh"
Sep 17 17:28:29.224848 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:29.224829    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-home\") pod \"stop-feature-test-kserve-6cc97bc96-655hh\" (UID: \"fa12b5a7-5b90-4b31-a5ed-db75b85aa258\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh"
Sep 17 17:28:29.225269 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:29.225181    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-cabundle-cert\") pod \"stop-feature-test-kserve-6cc97bc96-655hh\" (UID: \"fa12b5a7-5b90-4b31-a5ed-db75b85aa258\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh"
Sep 17 17:28:29.226855 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:29.226834    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-dshm\") pod \"stop-feature-test-kserve-6cc97bc96-655hh\" (UID: \"fa12b5a7-5b90-4b31-a5ed-db75b85aa258\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh"
Sep 17 17:28:29.227037 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:29.227021    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-tls-certs\") pod \"stop-feature-test-kserve-6cc97bc96-655hh\" (UID: \"fa12b5a7-5b90-4b31-a5ed-db75b85aa258\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh"
Sep 17 17:28:29.234271 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:29.234251    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nk2z\" (UniqueName: \"kubernetes.io/projected/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-kube-api-access-6nk2z\") pod \"stop-feature-test-kserve-6cc97bc96-655hh\" (UID: \"fa12b5a7-5b90-4b31-a5ed-db75b85aa258\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh"
Sep 17 17:28:29.403339 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:29.403240    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh"
Sep 17 17:28:29.706669 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:29.706641    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh"]
Sep 17 17:28:29.708711 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:28:29.708671    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa12b5a7_5b90_4b31_a5ed_db75b85aa258.slice/crio-366121aa1f429f29b705a1756a048182856a6a5b4118a2d7919cb0e5837114e3 WatchSource:0}: Error finding container 366121aa1f429f29b705a1756a048182856a6a5b4118a2d7919cb0e5837114e3: Status 404 returned error can't find the container with id 366121aa1f429f29b705a1756a048182856a6a5b4118a2d7919cb0e5837114e3
Sep 17 17:28:30.260522 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:30.260473    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh" event={"ID":"fa12b5a7-5b90-4b31-a5ed-db75b85aa258","Type":"ContainerStarted","Data":"70aa96f43edd3cf64d2c8892ee07438ef06e957e632f39107a5aa9e0221d36ff"}
Sep 17 17:28:30.260745 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:30.260546    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh" event={"ID":"fa12b5a7-5b90-4b31-a5ed-db75b85aa258","Type":"ContainerStarted","Data":"366121aa1f429f29b705a1756a048182856a6a5b4118a2d7919cb0e5837114e3"}
Sep 17 17:28:32.272726 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:32.272692    2566 generic.go:356] "Generic (PLEG): container finished" podID="fa12b5a7-5b90-4b31-a5ed-db75b85aa258" containerID="70aa96f43edd3cf64d2c8892ee07438ef06e957e632f39107a5aa9e0221d36ff" exitCode=0
Sep 17 17:28:32.273186 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:32.272733    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh" event={"ID":"fa12b5a7-5b90-4b31-a5ed-db75b85aa258","Type":"ContainerDied","Data":"70aa96f43edd3cf64d2c8892ee07438ef06e957e632f39107a5aa9e0221d36ff"}
Sep 17 17:28:33.279885 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:33.279841    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh" event={"ID":"fa12b5a7-5b90-4b31-a5ed-db75b85aa258","Type":"ContainerStarted","Data":"27a1a3160427549eeee3e29b6024c5d6c18519a0d1b7f3eeb71ba892195634cf"}
Sep 17 17:28:33.313450 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:33.313371    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh" podStartSLOduration=1.844910769 podStartE2EDuration="4.313347632s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.468436863s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:28:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:28:33.311473125 +0000 UTC m=+2799.205952257" watchObservedRunningTime="2026-09-17 17:28:33.313347632 +0000 UTC m=+2799.207826771"
Sep 17 17:28:33.404222 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:33.404180    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh"
Sep 17 17:28:34.192797 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:34.192712    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nj22k"
Sep 17 17:28:34.270056 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:34.270005    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cb8px\" (UniqueName: \"kubernetes.io/projected/43fd6571-80e0-4f9d-9d54-0ac7bd88efe2-kube-api-access-cb8px\") pod \"43fd6571-80e0-4f9d-9d54-0ac7bd88efe2\" (UID: \"43fd6571-80e0-4f9d-9d54-0ac7bd88efe2\") "
Sep 17 17:28:34.270056 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:34.270061    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43fd6571-80e0-4f9d-9d54-0ac7bd88efe2-tls-certs\") pod \"43fd6571-80e0-4f9d-9d54-0ac7bd88efe2\" (UID: \"43fd6571-80e0-4f9d-9d54-0ac7bd88efe2\") "
Sep 17 17:28:34.272558 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:34.272525    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43fd6571-80e0-4f9d-9d54-0ac7bd88efe2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "43fd6571-80e0-4f9d-9d54-0ac7bd88efe2" (UID: "43fd6571-80e0-4f9d-9d54-0ac7bd88efe2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:28:34.272683 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:34.272568    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43fd6571-80e0-4f9d-9d54-0ac7bd88efe2-kube-api-access-cb8px" (OuterVolumeSpecName: "kube-api-access-cb8px") pod "43fd6571-80e0-4f9d-9d54-0ac7bd88efe2" (UID: "43fd6571-80e0-4f9d-9d54-0ac7bd88efe2"). InnerVolumeSpecName "kube-api-access-cb8px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:28:34.287679 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:34.287643    2566 generic.go:356] "Generic (PLEG): container finished" podID="43fd6571-80e0-4f9d-9d54-0ac7bd88efe2" containerID="7f5f67d9a0635ebf0c2114a47ca6171a84cb9d2127fcef7496d602f12e9c7112" exitCode=0
Sep 17 17:28:34.288130 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:34.287685    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nj22k"
Sep 17 17:28:34.288130 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:34.287732    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nj22k" event={"ID":"43fd6571-80e0-4f9d-9d54-0ac7bd88efe2","Type":"ContainerDied","Data":"7f5f67d9a0635ebf0c2114a47ca6171a84cb9d2127fcef7496d602f12e9c7112"}
Sep 17 17:28:34.288130 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:34.287774    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nj22k" event={"ID":"43fd6571-80e0-4f9d-9d54-0ac7bd88efe2","Type":"ContainerDied","Data":"b928625b50e82cb3e8f6cfa690e5e88153d7764af2c6fe4a7dd5497eea0582fa"}
Sep 17 17:28:34.288130 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:34.287803    2566 scope.go:117] "RemoveContainer" containerID="7f5f67d9a0635ebf0c2114a47ca6171a84cb9d2127fcef7496d602f12e9c7112"
Sep 17 17:28:34.299031 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:34.299002    2566 scope.go:117] "RemoveContainer" containerID="7f5f67d9a0635ebf0c2114a47ca6171a84cb9d2127fcef7496d602f12e9c7112"
Sep 17 17:28:34.299392 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:28:34.299371    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f5f67d9a0635ebf0c2114a47ca6171a84cb9d2127fcef7496d602f12e9c7112\": container with ID starting with 7f5f67d9a0635ebf0c2114a47ca6171a84cb9d2127fcef7496d602f12e9c7112 not found: ID does not exist" containerID="7f5f67d9a0635ebf0c2114a47ca6171a84cb9d2127fcef7496d602f12e9c7112"
Sep 17 17:28:34.299482 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:34.299405    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f5f67d9a0635ebf0c2114a47ca6171a84cb9d2127fcef7496d602f12e9c7112"} err="failed to get container status \"7f5f67d9a0635ebf0c2114a47ca6171a84cb9d2127fcef7496d602f12e9c7112\": rpc error: code = NotFound desc = could not find container \"7f5f67d9a0635ebf0c2114a47ca6171a84cb9d2127fcef7496d602f12e9c7112\": container with ID starting with 7f5f67d9a0635ebf0c2114a47ca6171a84cb9d2127fcef7496d602f12e9c7112 not found: ID does not exist"
Sep 17 17:28:34.314197 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:34.314161    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nj22k"]
Sep 17 17:28:34.319584 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:34.319551    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nj22k"]
Sep 17 17:28:34.349573 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:34.349531    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43fd6571-80e0-4f9d-9d54-0ac7bd88efe2" path="/var/lib/kubelet/pods/43fd6571-80e0-4f9d-9d54-0ac7bd88efe2/volumes"
Sep 17 17:28:34.371675 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:34.371372    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb8px\" (UniqueName: \"kubernetes.io/projected/43fd6571-80e0-4f9d-9d54-0ac7bd88efe2-kube-api-access-cb8px\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:28:34.371675 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:34.371403    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43fd6571-80e0-4f9d-9d54-0ac7bd88efe2-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:28:36.270611 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:36.270554    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d" podUID="69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 17 17:28:39.404343 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:39.404303    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh"
Sep 17 17:28:39.406286 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:39.406252    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh" podUID="fa12b5a7-5b90-4b31-a5ed-db75b85aa258" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Sep 17 17:28:46.271113 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:46.271060    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d" podUID="69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 17 17:28:49.404095 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:49.404042    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh" podUID="fa12b5a7-5b90-4b31-a5ed-db75b85aa258" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Sep 17 17:28:56.270337 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:56.270283    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d" podUID="69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 17 17:28:59.404799 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:28:59.404743    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh" podUID="fa12b5a7-5b90-4b31-a5ed-db75b85aa258" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Sep 17 17:29:06.270568 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:06.270515    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d" podUID="69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 17 17:29:09.404268 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:09.404210    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh" podUID="fa12b5a7-5b90-4b31-a5ed-db75b85aa258" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Sep 17 17:29:16.270852 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:16.270803    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d" podUID="69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 17 17:29:19.404015 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:19.403960    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh" podUID="fa12b5a7-5b90-4b31-a5ed-db75b85aa258" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Sep 17 17:29:26.280733 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:26.280694    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d"
Sep 17 17:29:26.289056 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:26.289026    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d"
Sep 17 17:29:29.404705 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:29.404657    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh" podUID="fa12b5a7-5b90-4b31-a5ed-db75b85aa258" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Sep 17 17:29:36.119319 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:36.119234    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d"]
Sep 17 17:29:39.403969 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:39.403922    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh" podUID="fa12b5a7-5b90-4b31-a5ed-db75b85aa258" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Sep 17 17:29:49.404327 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:49.404270    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh" podUID="fa12b5a7-5b90-4b31-a5ed-db75b85aa258" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Sep 17 17:29:51.142807 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:51.142759    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d" podUID="69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c" containerName="main" containerID="cri-o://6eef73d13be107001e7c44844a718cd9b868270db1011ac589ad351ca360914d" gracePeriod=45
Sep 17 17:29:52.189056 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:52.189019    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d"
Sep 17 17:29:52.283136 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:52.283033    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpxjn\" (UniqueName: \"kubernetes.io/projected/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-kube-api-access-wpxjn\") pod \"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c\" (UID: \"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c\") "
Sep 17 17:29:52.283136 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:52.283128    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-tls-certs\") pod \"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c\" (UID: \"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c\") "
Sep 17 17:29:52.283351 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:52.283185    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-kserve-provision-location\") pod \"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c\" (UID: \"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c\") "
Sep 17 17:29:52.283351 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:52.283222    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-tmp-dir\") pod \"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c\" (UID: \"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c\") "
Sep 17 17:29:52.283351 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:52.283250    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-home\") pod \"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c\" (UID: \"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c\") "
Sep 17 17:29:52.283351 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:52.283275    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-model-cache\") pod \"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c\" (UID: \"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c\") "
Sep 17 17:29:52.283351 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:52.283300    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-dshm\") pod \"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c\" (UID: \"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c\") "
Sep 17 17:29:52.283351 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:52.283344    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-cabundle-cert\") pod \"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c\" (UID: \"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c\") "
Sep 17 17:29:52.284095 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:52.283843    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-model-cache" (OuterVolumeSpecName: "model-cache") pod "69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c" (UID: "69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:29:52.284095 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:52.283994    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c" (UID: "69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:29:52.284095 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:52.284030    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-home" (OuterVolumeSpecName: "home") pod "69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c" (UID: "69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:29:52.286254 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:52.286166    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-kube-api-access-wpxjn" (OuterVolumeSpecName: "kube-api-access-wpxjn") pod "69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c" (UID: "69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c"). InnerVolumeSpecName "kube-api-access-wpxjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:29:52.286254 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:52.286183    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-dshm" (OuterVolumeSpecName: "dshm") pod "69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c" (UID: "69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:29:52.286254 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:52.286197    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c" (UID: "69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:29:52.303667 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:52.303608    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c" (UID: "69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:29:52.306771 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:52.306737    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c" (UID: "69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:29:52.384886 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:52.384848    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-home\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:29:52.384886 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:52.384881    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-model-cache\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:29:52.384886 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:52.384892    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-dshm\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:29:52.385132 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:52.384899    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-cabundle-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:29:52.385132 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:52.384908    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpxjn\" (UniqueName: \"kubernetes.io/projected/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-kube-api-access-wpxjn\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:29:52.385132 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:52.384917    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:29:52.385132 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:52.384925    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-kserve-provision-location\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:29:52.385132 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:52.384952    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c-tmp-dir\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:29:52.766032 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:52.765992    2566 generic.go:356] "Generic (PLEG): container finished" podID="69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c" containerID="6eef73d13be107001e7c44844a718cd9b868270db1011ac589ad351ca360914d" exitCode=0
Sep 17 17:29:52.766225 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:52.766050    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d" event={"ID":"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c","Type":"ContainerDied","Data":"6eef73d13be107001e7c44844a718cd9b868270db1011ac589ad351ca360914d"}
Sep 17 17:29:52.766225 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:52.766104    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d" event={"ID":"69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c","Type":"ContainerDied","Data":"1aa3c6834a83c48848d8aec805b5b8d56e750234117b6cd9780b49f2bd93a964"}
Sep 17 17:29:52.766225 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:52.766126    2566 scope.go:117] "RemoveContainer" containerID="6eef73d13be107001e7c44844a718cd9b868270db1011ac589ad351ca360914d"
Sep 17 17:29:52.766225 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:52.766055    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d"
Sep 17 17:29:52.777412 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:52.777389    2566 scope.go:117] "RemoveContainer" containerID="668f568b6c7af701a5cc7aa39564a3d24abf21ebd5361c6a236a137ad72966cf"
Sep 17 17:29:52.786246 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:52.786219    2566 scope.go:117] "RemoveContainer" containerID="6eef73d13be107001e7c44844a718cd9b868270db1011ac589ad351ca360914d"
Sep 17 17:29:52.786667 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:29:52.786619    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eef73d13be107001e7c44844a718cd9b868270db1011ac589ad351ca360914d\": container with ID starting with 6eef73d13be107001e7c44844a718cd9b868270db1011ac589ad351ca360914d not found: ID does not exist" containerID="6eef73d13be107001e7c44844a718cd9b868270db1011ac589ad351ca360914d"
Sep 17 17:29:52.786792 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:52.786659    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eef73d13be107001e7c44844a718cd9b868270db1011ac589ad351ca360914d"} err="failed to get container status \"6eef73d13be107001e7c44844a718cd9b868270db1011ac589ad351ca360914d\": rpc error: code = NotFound desc = could not find container \"6eef73d13be107001e7c44844a718cd9b868270db1011ac589ad351ca360914d\": container with ID starting with 6eef73d13be107001e7c44844a718cd9b868270db1011ac589ad351ca360914d not found: ID does not exist"
Sep 17 17:29:52.786792 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:52.786685    2566 scope.go:117] "RemoveContainer" containerID="668f568b6c7af701a5cc7aa39564a3d24abf21ebd5361c6a236a137ad72966cf"
Sep 17 17:29:52.787160 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:29:52.787131    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"668f568b6c7af701a5cc7aa39564a3d24abf21ebd5361c6a236a137ad72966cf\": container with ID starting with 668f568b6c7af701a5cc7aa39564a3d24abf21ebd5361c6a236a137ad72966cf not found: ID does not exist" containerID="668f568b6c7af701a5cc7aa39564a3d24abf21ebd5361c6a236a137ad72966cf"
Sep 17 17:29:52.787265 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:52.787166    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"668f568b6c7af701a5cc7aa39564a3d24abf21ebd5361c6a236a137ad72966cf"} err="failed to get container status \"668f568b6c7af701a5cc7aa39564a3d24abf21ebd5361c6a236a137ad72966cf\": rpc error: code = NotFound desc = could not find container \"668f568b6c7af701a5cc7aa39564a3d24abf21ebd5361c6a236a137ad72966cf\": container with ID starting with 668f568b6c7af701a5cc7aa39564a3d24abf21ebd5361c6a236a137ad72966cf not found: ID does not exist"
Sep 17 17:29:52.791156 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:52.791127    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d"]
Sep 17 17:29:52.796850 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:52.796824    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56x295d"]
Sep 17 17:29:54.346666 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:54.346632    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c" path="/var/lib/kubelet/pods/69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c/volumes"
Sep 17 17:29:59.404649 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:29:59.404604    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh" podUID="fa12b5a7-5b90-4b31-a5ed-db75b85aa258" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Sep 17 17:30:09.404443 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:09.404383    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh" podUID="fa12b5a7-5b90-4b31-a5ed-db75b85aa258" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Sep 17 17:30:19.404182 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:19.404139    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh" podUID="fa12b5a7-5b90-4b31-a5ed-db75b85aa258" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Sep 17 17:30:29.413786 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:29.413750    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh"
Sep 17 17:30:29.421745 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:29.421724    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh"
Sep 17 17:30:36.959870 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:36.959833    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm"]
Sep 17 17:30:36.960382 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:36.960174    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43fd6571-80e0-4f9d-9d54-0ac7bd88efe2" containerName="main"
Sep 17 17:30:36.960382 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:36.960185    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="43fd6571-80e0-4f9d-9d54-0ac7bd88efe2" containerName="main"
Sep 17 17:30:36.960382 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:36.960196    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c" containerName="storage-initializer"
Sep 17 17:30:36.960382 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:36.960215    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c" containerName="storage-initializer"
Sep 17 17:30:36.960382 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:36.960226    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c" containerName="main"
Sep 17 17:30:36.960382 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:36.960231    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c" containerName="main"
Sep 17 17:30:36.960382 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:36.960290    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="69fb7dbd-18f1-408c-b76e-3d3d1aeb5a5c" containerName="main"
Sep 17 17:30:36.960382 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:36.960299    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="43fd6571-80e0-4f9d-9d54-0ac7bd88efe2" containerName="main"
Sep 17 17:30:36.964574 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:36.964546    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm"
Sep 17 17:30:36.967883 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:36.967860    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 17 17:30:36.968028 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:36.967909    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-rz8kz\""
Sep 17 17:30:36.968414 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:36.968380    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 17 17:30:36.968414 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:36.968379    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:30:36.968672 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:36.968425    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 17 17:30:36.968672 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:36.968502    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 17 17:30:37.016272 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:37.016233    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm"]
Sep 17 17:30:37.045562 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:37.045529    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54c07692-6cf6-4578-9e66-b3130ff42bd8-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm\" (UID: \"54c07692-6cf6-4578-9e66-b3130ff42bd8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm"
Sep 17 17:30:37.045562 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:37.045566    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54c07692-6cf6-4578-9e66-b3130ff42bd8-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm\" (UID: \"54c07692-6cf6-4578-9e66-b3130ff42bd8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm"
Sep 17 17:30:37.045789 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:37.045595    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54c07692-6cf6-4578-9e66-b3130ff42bd8-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm\" (UID: \"54c07692-6cf6-4578-9e66-b3130ff42bd8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm"
Sep 17 17:30:37.045789 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:37.045671    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54c07692-6cf6-4578-9e66-b3130ff42bd8-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm\" (UID: \"54c07692-6cf6-4578-9e66-b3130ff42bd8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm"
Sep 17 17:30:37.045789 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:37.045716    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54c07692-6cf6-4578-9e66-b3130ff42bd8-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm\" (UID: \"54c07692-6cf6-4578-9e66-b3130ff42bd8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm"
Sep 17 17:30:37.045789 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:37.045734    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s4qs\" (UniqueName: \"kubernetes.io/projected/54c07692-6cf6-4578-9e66-b3130ff42bd8-kube-api-access-7s4qs\") pod \"custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm\" (UID: \"54c07692-6cf6-4578-9e66-b3130ff42bd8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm"
Sep 17 17:30:37.045789 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:37.045771    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54c07692-6cf6-4578-9e66-b3130ff42bd8-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm\" (UID: \"54c07692-6cf6-4578-9e66-b3130ff42bd8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm"
Sep 17 17:30:37.045789 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:37.045790    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54c07692-6cf6-4578-9e66-b3130ff42bd8-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm\" (UID: \"54c07692-6cf6-4578-9e66-b3130ff42bd8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm"
Sep 17 17:30:37.146820 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:37.146777    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54c07692-6cf6-4578-9e66-b3130ff42bd8-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm\" (UID: \"54c07692-6cf6-4578-9e66-b3130ff42bd8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm"
Sep 17 17:30:37.146820 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:37.146822    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54c07692-6cf6-4578-9e66-b3130ff42bd8-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm\" (UID: \"54c07692-6cf6-4578-9e66-b3130ff42bd8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm"
Sep 17 17:30:37.147100 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:37.146948    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54c07692-6cf6-4578-9e66-b3130ff42bd8-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm\" (UID: \"54c07692-6cf6-4578-9e66-b3130ff42bd8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm"
Sep 17 17:30:37.147100 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:37.146983    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s4qs\" (UniqueName: \"kubernetes.io/projected/54c07692-6cf6-4578-9e66-b3130ff42bd8-kube-api-access-7s4qs\") pod \"custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm\" (UID: \"54c07692-6cf6-4578-9e66-b3130ff42bd8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm"
Sep 17 17:30:37.147100 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:37.147028    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54c07692-6cf6-4578-9e66-b3130ff42bd8-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm\" (UID: \"54c07692-6cf6-4578-9e66-b3130ff42bd8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm"
Sep 17 17:30:37.147100 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:37.147045    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54c07692-6cf6-4578-9e66-b3130ff42bd8-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm\" (UID: \"54c07692-6cf6-4578-9e66-b3130ff42bd8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm"
Sep 17 17:30:37.147329 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:37.147103    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54c07692-6cf6-4578-9e66-b3130ff42bd8-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm\" (UID: \"54c07692-6cf6-4578-9e66-b3130ff42bd8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm"
Sep 17 17:30:37.147329 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:37.147135    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54c07692-6cf6-4578-9e66-b3130ff42bd8-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm\" (UID: \"54c07692-6cf6-4578-9e66-b3130ff42bd8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm"
Sep 17 17:30:37.147329 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:37.147202    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54c07692-6cf6-4578-9e66-b3130ff42bd8-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm\" (UID: \"54c07692-6cf6-4578-9e66-b3130ff42bd8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm"
Sep 17 17:30:37.147329 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:37.147250    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54c07692-6cf6-4578-9e66-b3130ff42bd8-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm\" (UID: \"54c07692-6cf6-4578-9e66-b3130ff42bd8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm"
Sep 17 17:30:37.147576 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:37.147511    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54c07692-6cf6-4578-9e66-b3130ff42bd8-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm\" (UID: \"54c07692-6cf6-4578-9e66-b3130ff42bd8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm"
Sep 17 17:30:37.147633 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:37.147574    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54c07692-6cf6-4578-9e66-b3130ff42bd8-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm\" (UID: \"54c07692-6cf6-4578-9e66-b3130ff42bd8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm"
Sep 17 17:30:37.147888 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:37.147828    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54c07692-6cf6-4578-9e66-b3130ff42bd8-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm\" (UID: \"54c07692-6cf6-4578-9e66-b3130ff42bd8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm"
Sep 17 17:30:37.149553 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:37.149529    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54c07692-6cf6-4578-9e66-b3130ff42bd8-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm\" (UID: \"54c07692-6cf6-4578-9e66-b3130ff42bd8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm"
Sep 17 17:30:37.149847 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:37.149829    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54c07692-6cf6-4578-9e66-b3130ff42bd8-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm\" (UID: \"54c07692-6cf6-4578-9e66-b3130ff42bd8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm"
Sep 17 17:30:37.160918 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:37.160892    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s4qs\" (UniqueName: \"kubernetes.io/projected/54c07692-6cf6-4578-9e66-b3130ff42bd8-kube-api-access-7s4qs\") pod \"custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm\" (UID: \"54c07692-6cf6-4578-9e66-b3130ff42bd8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm"
Sep 17 17:30:37.275795 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:37.275693    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm"
Sep 17 17:30:37.570511 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:37.570309    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm"]
Sep 17 17:30:37.573206 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:30:37.573181    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54c07692_6cf6_4578_9e66_b3130ff42bd8.slice/crio-cb54cd2e61a2f21f9e9f28befdfa74c42fbf191470a40dbb0b0b033330e9a919 WatchSource:0}: Error finding container cb54cd2e61a2f21f9e9f28befdfa74c42fbf191470a40dbb0b0b033330e9a919: Status 404 returned error can't find the container with id cb54cd2e61a2f21f9e9f28befdfa74c42fbf191470a40dbb0b0b033330e9a919
Sep 17 17:30:38.016293 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:38.016184    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm" event={"ID":"54c07692-6cf6-4578-9e66-b3130ff42bd8","Type":"ContainerStarted","Data":"4725ef2221d5bc43feaec294794e1c5d43311debde0ab2810e68878ba1c81418"}
Sep 17 17:30:38.016293 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:38.016252    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm" event={"ID":"54c07692-6cf6-4578-9e66-b3130ff42bd8","Type":"ContainerStarted","Data":"cb54cd2e61a2f21f9e9f28befdfa74c42fbf191470a40dbb0b0b033330e9a919"}
Sep 17 17:30:41.034986 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:41.034948    2566 generic.go:356] "Generic (PLEG): container finished" podID="54c07692-6cf6-4578-9e66-b3130ff42bd8" containerID="4725ef2221d5bc43feaec294794e1c5d43311debde0ab2810e68878ba1c81418" exitCode=0
Sep 17 17:30:41.035490 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:41.034996    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm" event={"ID":"54c07692-6cf6-4578-9e66-b3130ff42bd8","Type":"ContainerDied","Data":"4725ef2221d5bc43feaec294794e1c5d43311debde0ab2810e68878ba1c81418"}
Sep 17 17:30:42.043342 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:42.043309    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm" event={"ID":"54c07692-6cf6-4578-9e66-b3130ff42bd8","Type":"ContainerStarted","Data":"d4ce9f7c969f718b81a950f5b716994da22665afec1a4a13eaca866a709a6362"}
Sep 17 17:30:42.074052 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:42.073996    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm" podStartSLOduration=3.588092626 podStartE2EDuration="6.07397614s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.485883514s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:30:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:30:42.07165714 +0000 UTC m=+2927.966136279" watchObservedRunningTime="2026-09-17 17:30:42.07397614 +0000 UTC m=+2927.968455280"
Sep 17 17:30:47.276854 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:47.276808    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm"
Sep 17 17:30:47.277285 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:47.276895    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm"
Sep 17 17:30:47.278500 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:47.278468    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm" podUID="54c07692-6cf6-4578-9e66-b3130ff42bd8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Sep 17 17:30:49.880133 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:49.880082    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh"]
Sep 17 17:30:57.277131 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:30:57.277081    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm" podUID="54c07692-6cf6-4578-9e66-b3130ff42bd8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Sep 17 17:31:04.905181 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:04.905086    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh" podUID="fa12b5a7-5b90-4b31-a5ed-db75b85aa258" containerName="main" containerID="cri-o://27a1a3160427549eeee3e29b6024c5d6c18519a0d1b7f3eeb71ba892195634cf" gracePeriod=45
Sep 17 17:31:05.415137 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:05.415089    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh" podUID="fa12b5a7-5b90-4b31-a5ed-db75b85aa258" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Sep 17 17:31:05.664376 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:05.664347    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh"
Sep 17 17:31:05.809095 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:05.809065    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-tmp-dir\") pod \"fa12b5a7-5b90-4b31-a5ed-db75b85aa258\" (UID: \"fa12b5a7-5b90-4b31-a5ed-db75b85aa258\") "
Sep 17 17:31:05.809277 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:05.809109    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-dshm\") pod \"fa12b5a7-5b90-4b31-a5ed-db75b85aa258\" (UID: \"fa12b5a7-5b90-4b31-a5ed-db75b85aa258\") "
Sep 17 17:31:05.809277 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:05.809127    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nk2z\" (UniqueName: \"kubernetes.io/projected/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-kube-api-access-6nk2z\") pod \"fa12b5a7-5b90-4b31-a5ed-db75b85aa258\" (UID: \"fa12b5a7-5b90-4b31-a5ed-db75b85aa258\") "
Sep 17 17:31:05.809277 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:05.809160    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-cabundle-cert\") pod \"fa12b5a7-5b90-4b31-a5ed-db75b85aa258\" (UID: \"fa12b5a7-5b90-4b31-a5ed-db75b85aa258\") "
Sep 17 17:31:05.809277 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:05.809205    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-kserve-provision-location\") pod \"fa12b5a7-5b90-4b31-a5ed-db75b85aa258\" (UID: \"fa12b5a7-5b90-4b31-a5ed-db75b85aa258\") "
Sep 17 17:31:05.809277 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:05.809234    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-model-cache\") pod \"fa12b5a7-5b90-4b31-a5ed-db75b85aa258\" (UID: \"fa12b5a7-5b90-4b31-a5ed-db75b85aa258\") "
Sep 17 17:31:05.809277 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:05.809258    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-home\") pod \"fa12b5a7-5b90-4b31-a5ed-db75b85aa258\" (UID: \"fa12b5a7-5b90-4b31-a5ed-db75b85aa258\") "
Sep 17 17:31:05.809656 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:05.809293    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-tls-certs\") pod \"fa12b5a7-5b90-4b31-a5ed-db75b85aa258\" (UID: \"fa12b5a7-5b90-4b31-a5ed-db75b85aa258\") "
Sep 17 17:31:05.809656 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:05.809584    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-model-cache" (OuterVolumeSpecName: "model-cache") pod "fa12b5a7-5b90-4b31-a5ed-db75b85aa258" (UID: "fa12b5a7-5b90-4b31-a5ed-db75b85aa258"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:31:05.810307 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:05.810260    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-home" (OuterVolumeSpecName: "home") pod "fa12b5a7-5b90-4b31-a5ed-db75b85aa258" (UID: "fa12b5a7-5b90-4b31-a5ed-db75b85aa258"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:31:05.811143 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:05.811098    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fa12b5a7-5b90-4b31-a5ed-db75b85aa258" (UID: "fa12b5a7-5b90-4b31-a5ed-db75b85aa258"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:31:05.811902 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:05.811871    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-kube-api-access-6nk2z" (OuterVolumeSpecName: "kube-api-access-6nk2z") pod "fa12b5a7-5b90-4b31-a5ed-db75b85aa258" (UID: "fa12b5a7-5b90-4b31-a5ed-db75b85aa258"). InnerVolumeSpecName "kube-api-access-6nk2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:31:05.811980 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:05.811896    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-dshm" (OuterVolumeSpecName: "dshm") pod "fa12b5a7-5b90-4b31-a5ed-db75b85aa258" (UID: "fa12b5a7-5b90-4b31-a5ed-db75b85aa258"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:31:05.812342 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:05.812315    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fa12b5a7-5b90-4b31-a5ed-db75b85aa258" (UID: "fa12b5a7-5b90-4b31-a5ed-db75b85aa258"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:31:05.824248 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:05.824198    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fa12b5a7-5b90-4b31-a5ed-db75b85aa258" (UID: "fa12b5a7-5b90-4b31-a5ed-db75b85aa258"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:31:05.844886 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:05.844828    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fa12b5a7-5b90-4b31-a5ed-db75b85aa258" (UID: "fa12b5a7-5b90-4b31-a5ed-db75b85aa258"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:31:05.910790 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:05.910749    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nk2z\" (UniqueName: \"kubernetes.io/projected/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-kube-api-access-6nk2z\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:31:05.910790 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:05.910784    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-cabundle-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:31:05.911240 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:05.910801    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-kserve-provision-location\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:31:05.911240 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:05.910813    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-model-cache\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:31:05.911240 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:05.910825    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-home\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:31:05.911240 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:05.910835    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:31:05.911240 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:05.910872    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-tmp-dir\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:31:05.911240 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:05.910885    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa12b5a7-5b90-4b31-a5ed-db75b85aa258-dshm\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:31:06.181891 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:06.181793    2566 generic.go:356] "Generic (PLEG): container finished" podID="fa12b5a7-5b90-4b31-a5ed-db75b85aa258" containerID="27a1a3160427549eeee3e29b6024c5d6c18519a0d1b7f3eeb71ba892195634cf" exitCode=0
Sep 17 17:31:06.181891 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:06.181852    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh"
Sep 17 17:31:06.182096 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:06.181879    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh" event={"ID":"fa12b5a7-5b90-4b31-a5ed-db75b85aa258","Type":"ContainerDied","Data":"27a1a3160427549eeee3e29b6024c5d6c18519a0d1b7f3eeb71ba892195634cf"}
Sep 17 17:31:06.182096 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:06.181940    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh" event={"ID":"fa12b5a7-5b90-4b31-a5ed-db75b85aa258","Type":"ContainerDied","Data":"366121aa1f429f29b705a1756a048182856a6a5b4118a2d7919cb0e5837114e3"}
Sep 17 17:31:06.182096 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:06.181963    2566 scope.go:117] "RemoveContainer" containerID="27a1a3160427549eeee3e29b6024c5d6c18519a0d1b7f3eeb71ba892195634cf"
Sep 17 17:31:06.194883 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:06.194859    2566 scope.go:117] "RemoveContainer" containerID="70aa96f43edd3cf64d2c8892ee07438ef06e957e632f39107a5aa9e0221d36ff"
Sep 17 17:31:06.203963 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:06.203939    2566 scope.go:117] "RemoveContainer" containerID="27a1a3160427549eeee3e29b6024c5d6c18519a0d1b7f3eeb71ba892195634cf"
Sep 17 17:31:06.204324 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:31:06.204284    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27a1a3160427549eeee3e29b6024c5d6c18519a0d1b7f3eeb71ba892195634cf\": container with ID starting with 27a1a3160427549eeee3e29b6024c5d6c18519a0d1b7f3eeb71ba892195634cf not found: ID does not exist" containerID="27a1a3160427549eeee3e29b6024c5d6c18519a0d1b7f3eeb71ba892195634cf"
Sep 17 17:31:06.204422 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:06.204329    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27a1a3160427549eeee3e29b6024c5d6c18519a0d1b7f3eeb71ba892195634cf"} err="failed to get container status \"27a1a3160427549eeee3e29b6024c5d6c18519a0d1b7f3eeb71ba892195634cf\": rpc error: code = NotFound desc = could not find container \"27a1a3160427549eeee3e29b6024c5d6c18519a0d1b7f3eeb71ba892195634cf\": container with ID starting with 27a1a3160427549eeee3e29b6024c5d6c18519a0d1b7f3eeb71ba892195634cf not found: ID does not exist"
Sep 17 17:31:06.204422 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:06.204350    2566 scope.go:117] "RemoveContainer" containerID="70aa96f43edd3cf64d2c8892ee07438ef06e957e632f39107a5aa9e0221d36ff"
Sep 17 17:31:06.204673 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:31:06.204645    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70aa96f43edd3cf64d2c8892ee07438ef06e957e632f39107a5aa9e0221d36ff\": container with ID starting with 70aa96f43edd3cf64d2c8892ee07438ef06e957e632f39107a5aa9e0221d36ff not found: ID does not exist" containerID="70aa96f43edd3cf64d2c8892ee07438ef06e957e632f39107a5aa9e0221d36ff"
Sep 17 17:31:06.204711 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:06.204688    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70aa96f43edd3cf64d2c8892ee07438ef06e957e632f39107a5aa9e0221d36ff"} err="failed to get container status \"70aa96f43edd3cf64d2c8892ee07438ef06e957e632f39107a5aa9e0221d36ff\": rpc error: code = NotFound desc = could not find container \"70aa96f43edd3cf64d2c8892ee07438ef06e957e632f39107a5aa9e0221d36ff\": container with ID starting with 70aa96f43edd3cf64d2c8892ee07438ef06e957e632f39107a5aa9e0221d36ff not found: ID does not exist"
Sep 17 17:31:06.222791 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:06.222752    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh"]
Sep 17 17:31:06.227500 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:06.227464    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc97bc96-655hh"]
Sep 17 17:31:06.347504 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:06.347469    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa12b5a7-5b90-4b31-a5ed-db75b85aa258" path="/var/lib/kubelet/pods/fa12b5a7-5b90-4b31-a5ed-db75b85aa258/volumes"
Sep 17 17:31:07.277066 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:07.277018    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm" podUID="54c07692-6cf6-4578-9e66-b3130ff42bd8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Sep 17 17:31:17.276802 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:17.276763    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm" podUID="54c07692-6cf6-4578-9e66-b3130ff42bd8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Sep 17 17:31:27.277002 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:27.276956    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm" podUID="54c07692-6cf6-4578-9e66-b3130ff42bd8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Sep 17 17:31:37.276555 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:37.276501    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm" podUID="54c07692-6cf6-4578-9e66-b3130ff42bd8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Sep 17 17:31:47.276862 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:47.276811    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm" podUID="54c07692-6cf6-4578-9e66-b3130ff42bd8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Sep 17 17:31:53.021338 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:53.021303    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9"]
Sep 17 17:31:53.021901 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:53.021724    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa12b5a7-5b90-4b31-a5ed-db75b85aa258" containerName="storage-initializer"
Sep 17 17:31:53.021901 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:53.021738    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa12b5a7-5b90-4b31-a5ed-db75b85aa258" containerName="storage-initializer"
Sep 17 17:31:53.021901 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:53.021748    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa12b5a7-5b90-4b31-a5ed-db75b85aa258" containerName="main"
Sep 17 17:31:53.021901 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:53.021753    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa12b5a7-5b90-4b31-a5ed-db75b85aa258" containerName="main"
Sep 17 17:31:53.021901 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:53.021813    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa12b5a7-5b90-4b31-a5ed-db75b85aa258" containerName="main"
Sep 17 17:31:53.023942 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:53.023924    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9"
Sep 17 17:31:53.027079 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:53.027048    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Sep 17 17:31:53.027247 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:53.027166    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Sep 17 17:31:53.027247 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:53.027208    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:31:53.027383 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:53.027256    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbe7478b-b6ed-429c-a89a-f12dc45edd43-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-85698b65fd-8sxr9\" (UID: \"cbe7478b-b6ed-429c-a89a-f12dc45edd43\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9"
Sep 17 17:31:53.027383 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:53.027299    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4bzd\" (UniqueName: \"kubernetes.io/projected/cbe7478b-b6ed-429c-a89a-f12dc45edd43-kube-api-access-k4bzd\") pod \"lora-single-adapter-test-kserve-85698b65fd-8sxr9\" (UID: \"cbe7478b-b6ed-429c-a89a-f12dc45edd43\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9"
Sep 17 17:31:53.027383 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:53.027306    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Sep 17 17:31:53.027383 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:53.027339    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbe7478b-b6ed-429c-a89a-f12dc45edd43-tmp-dir\") pod \"lora-single-adapter-test-kserve-85698b65fd-8sxr9\" (UID: \"cbe7478b-b6ed-429c-a89a-f12dc45edd43\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9"
Sep 17 17:31:53.027383 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:53.027381    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbe7478b-b6ed-429c-a89a-f12dc45edd43-home\") pod \"lora-single-adapter-test-kserve-85698b65fd-8sxr9\" (UID: \"cbe7478b-b6ed-429c-a89a-f12dc45edd43\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9"
Sep 17 17:31:53.027682 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:53.027411    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbe7478b-b6ed-429c-a89a-f12dc45edd43-model-cache\") pod \"lora-single-adapter-test-kserve-85698b65fd-8sxr9\" (UID: \"cbe7478b-b6ed-429c-a89a-f12dc45edd43\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9"
Sep 17 17:31:53.027682 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:53.027467    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbe7478b-b6ed-429c-a89a-f12dc45edd43-tls-certs\") pod \"lora-single-adapter-test-kserve-85698b65fd-8sxr9\" (UID: \"cbe7478b-b6ed-429c-a89a-f12dc45edd43\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9"
Sep 17 17:31:53.027682 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:53.027521    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbe7478b-b6ed-429c-a89a-f12dc45edd43-cabundle-cert\") pod \"lora-single-adapter-test-kserve-85698b65fd-8sxr9\" (UID: \"cbe7478b-b6ed-429c-a89a-f12dc45edd43\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9"
Sep 17 17:31:53.027682 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:53.027575    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbe7478b-b6ed-429c-a89a-f12dc45edd43-dshm\") pod \"lora-single-adapter-test-kserve-85698b65fd-8sxr9\" (UID: \"cbe7478b-b6ed-429c-a89a-f12dc45edd43\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9"
Sep 17 17:31:53.027682 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:53.027650    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-gzml7\""
Sep 17 17:31:53.027682 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:53.027659    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Sep 17 17:31:53.040645 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:53.040611    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9"]
Sep 17 17:31:53.128311 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:53.128268    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbe7478b-b6ed-429c-a89a-f12dc45edd43-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-85698b65fd-8sxr9\" (UID: \"cbe7478b-b6ed-429c-a89a-f12dc45edd43\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9"
Sep 17 17:31:53.128311 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:53.128310    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4bzd\" (UniqueName: \"kubernetes.io/projected/cbe7478b-b6ed-429c-a89a-f12dc45edd43-kube-api-access-k4bzd\") pod \"lora-single-adapter-test-kserve-85698b65fd-8sxr9\" (UID: \"cbe7478b-b6ed-429c-a89a-f12dc45edd43\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9"
Sep 17 17:31:53.128564 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:53.128478    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbe7478b-b6ed-429c-a89a-f12dc45edd43-tmp-dir\") pod \"lora-single-adapter-test-kserve-85698b65fd-8sxr9\" (UID: \"cbe7478b-b6ed-429c-a89a-f12dc45edd43\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9"
Sep 17 17:31:53.128564 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:53.128545    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbe7478b-b6ed-429c-a89a-f12dc45edd43-home\") pod \"lora-single-adapter-test-kserve-85698b65fd-8sxr9\" (UID: \"cbe7478b-b6ed-429c-a89a-f12dc45edd43\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9"
Sep 17 17:31:53.128678 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:53.128576    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbe7478b-b6ed-429c-a89a-f12dc45edd43-model-cache\") pod \"lora-single-adapter-test-kserve-85698b65fd-8sxr9\" (UID: \"cbe7478b-b6ed-429c-a89a-f12dc45edd43\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9"
Sep 17 17:31:53.128678 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:53.128617    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbe7478b-b6ed-429c-a89a-f12dc45edd43-tls-certs\") pod \"lora-single-adapter-test-kserve-85698b65fd-8sxr9\" (UID: \"cbe7478b-b6ed-429c-a89a-f12dc45edd43\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9"
Sep 17 17:31:53.128678 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:53.128648    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbe7478b-b6ed-429c-a89a-f12dc45edd43-cabundle-cert\") pod \"lora-single-adapter-test-kserve-85698b65fd-8sxr9\" (UID: \"cbe7478b-b6ed-429c-a89a-f12dc45edd43\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9"
Sep 17 17:31:53.128832 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:53.128679    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbe7478b-b6ed-429c-a89a-f12dc45edd43-dshm\") pod \"lora-single-adapter-test-kserve-85698b65fd-8sxr9\" (UID: \"cbe7478b-b6ed-429c-a89a-f12dc45edd43\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9"
Sep 17 17:31:53.128832 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:53.128778    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbe7478b-b6ed-429c-a89a-f12dc45edd43-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-85698b65fd-8sxr9\" (UID: \"cbe7478b-b6ed-429c-a89a-f12dc45edd43\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9"
Sep 17 17:31:53.128934 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:53.128838    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbe7478b-b6ed-429c-a89a-f12dc45edd43-tmp-dir\") pod \"lora-single-adapter-test-kserve-85698b65fd-8sxr9\" (UID: \"cbe7478b-b6ed-429c-a89a-f12dc45edd43\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9"
Sep 17 17:31:53.128990 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:53.128928    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbe7478b-b6ed-429c-a89a-f12dc45edd43-model-cache\") pod \"lora-single-adapter-test-kserve-85698b65fd-8sxr9\" (UID: \"cbe7478b-b6ed-429c-a89a-f12dc45edd43\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9"
Sep 17 17:31:53.128990 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:53.128957    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbe7478b-b6ed-429c-a89a-f12dc45edd43-home\") pod \"lora-single-adapter-test-kserve-85698b65fd-8sxr9\" (UID: \"cbe7478b-b6ed-429c-a89a-f12dc45edd43\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9"
Sep 17 17:31:53.129409 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:53.129386    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbe7478b-b6ed-429c-a89a-f12dc45edd43-cabundle-cert\") pod \"lora-single-adapter-test-kserve-85698b65fd-8sxr9\" (UID: \"cbe7478b-b6ed-429c-a89a-f12dc45edd43\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9"
Sep 17 17:31:53.131261 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:53.131228    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbe7478b-b6ed-429c-a89a-f12dc45edd43-dshm\") pod \"lora-single-adapter-test-kserve-85698b65fd-8sxr9\" (UID: \"cbe7478b-b6ed-429c-a89a-f12dc45edd43\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9"
Sep 17 17:31:53.131367 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:53.131291    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbe7478b-b6ed-429c-a89a-f12dc45edd43-tls-certs\") pod \"lora-single-adapter-test-kserve-85698b65fd-8sxr9\" (UID: \"cbe7478b-b6ed-429c-a89a-f12dc45edd43\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9"
Sep 17 17:31:53.141020 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:53.140993    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4bzd\" (UniqueName: \"kubernetes.io/projected/cbe7478b-b6ed-429c-a89a-f12dc45edd43-kube-api-access-k4bzd\") pod \"lora-single-adapter-test-kserve-85698b65fd-8sxr9\" (UID: \"cbe7478b-b6ed-429c-a89a-f12dc45edd43\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9"
Sep 17 17:31:53.335364 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:53.335321    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9"
Sep 17 17:31:53.627415 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:31:53.627383    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbe7478b_b6ed_429c_a89a_f12dc45edd43.slice/crio-efe244e2522dfabce8b3052c1fb32824cf05ef758bacc4e35b36e5a306651ea8 WatchSource:0}: Error finding container efe244e2522dfabce8b3052c1fb32824cf05ef758bacc4e35b36e5a306651ea8: Status 404 returned error can't find the container with id efe244e2522dfabce8b3052c1fb32824cf05ef758bacc4e35b36e5a306651ea8
Sep 17 17:31:53.627713 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:53.627688    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9"]
Sep 17 17:31:54.454099 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:54.454049    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9" event={"ID":"cbe7478b-b6ed-429c-a89a-f12dc45edd43","Type":"ContainerStarted","Data":"d16745a2658e76289d0ca7650d5840abe8b9ff3579bf55273478994a35984756"}
Sep 17 17:31:54.454550 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:54.454133    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9" event={"ID":"cbe7478b-b6ed-429c-a89a-f12dc45edd43","Type":"ContainerStarted","Data":"efe244e2522dfabce8b3052c1fb32824cf05ef758bacc4e35b36e5a306651ea8"}
Sep 17 17:31:56.564276 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:56.564243    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gblhr_91477834-6648-4e75-8530-e01bdee5e57e/ovn-acl-logging/0.log"
Sep 17 17:31:56.584512 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:56.584476    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gblhr_91477834-6648-4e75-8530-e01bdee5e57e/ovn-acl-logging/0.log"
Sep 17 17:31:57.276947 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:57.276890    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm" podUID="54c07692-6cf6-4578-9e66-b3130ff42bd8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Sep 17 17:31:57.472277 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:57.472241    2566 generic.go:356] "Generic (PLEG): container finished" podID="cbe7478b-b6ed-429c-a89a-f12dc45edd43" containerID="d16745a2658e76289d0ca7650d5840abe8b9ff3579bf55273478994a35984756" exitCode=0
Sep 17 17:31:57.472481 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:57.472282    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9" event={"ID":"cbe7478b-b6ed-429c-a89a-f12dc45edd43","Type":"ContainerDied","Data":"d16745a2658e76289d0ca7650d5840abe8b9ff3579bf55273478994a35984756"}
Sep 17 17:31:58.480474 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:58.480388    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9" event={"ID":"cbe7478b-b6ed-429c-a89a-f12dc45edd43","Type":"ContainerStarted","Data":"3e990809f1edb88f1ad142d495df8b449a6e4c07a9ca5dde0048e867f446bc20"}
Sep 17 17:31:58.512474 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:58.512386    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9" podStartSLOduration=3.455852129 podStartE2EDuration="6.512362627s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.056510498s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:31:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:31:58.509981101 +0000 UTC m=+3004.404460240" watchObservedRunningTime="2026-09-17 17:31:58.512362627 +0000 UTC m=+3004.406841770"
Sep 17 17:31:59.335508 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:31:59.335458    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9"
Sep 17 17:32:03.335724 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:32:03.335666    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9"
Sep 17 17:32:03.337362 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:32:03.337328    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9" podUID="cbe7478b-b6ed-429c-a89a-f12dc45edd43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 17 17:32:07.277191 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:32:07.277136    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm" podUID="54c07692-6cf6-4578-9e66-b3130ff42bd8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Sep 17 17:32:13.336208 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:32:13.336150    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9" podUID="cbe7478b-b6ed-429c-a89a-f12dc45edd43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 17 17:32:17.276561 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:32:17.276512    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm" podUID="54c07692-6cf6-4578-9e66-b3130ff42bd8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Sep 17 17:32:23.336842 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:32:23.336786    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9" podUID="cbe7478b-b6ed-429c-a89a-f12dc45edd43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 17 17:32:27.276904 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:32:27.276843    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm" podUID="54c07692-6cf6-4578-9e66-b3130ff42bd8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Sep 17 17:32:33.336673 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:32:33.336612    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9" podUID="cbe7478b-b6ed-429c-a89a-f12dc45edd43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 17 17:32:37.276612 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:32:37.276507    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm" podUID="54c07692-6cf6-4578-9e66-b3130ff42bd8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Sep 17 17:32:43.336430 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:32:43.336385    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9" podUID="cbe7478b-b6ed-429c-a89a-f12dc45edd43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 17 17:32:47.286505 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:32:47.286469    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm"
Sep 17 17:32:47.294410 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:32:47.294382    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm"
Sep 17 17:32:53.336368 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:32:53.336324    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9" podUID="cbe7478b-b6ed-429c-a89a-f12dc45edd43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 17 17:32:57.206101 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:32:57.206065    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm"]
Sep 17 17:33:03.336159 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:03.336116    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9" podUID="cbe7478b-b6ed-429c-a89a-f12dc45edd43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 17 17:33:12.229680 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:12.229614    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm" podUID="54c07692-6cf6-4578-9e66-b3130ff42bd8" containerName="main" containerID="cri-o://d4ce9f7c969f718b81a950f5b716994da22665afec1a4a13eaca866a709a6362" gracePeriod=45
Sep 17 17:33:13.310769 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:13.310613    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm"
Sep 17 17:33:13.327491 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:13.326680    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54c07692-6cf6-4578-9e66-b3130ff42bd8-model-cache\") pod \"54c07692-6cf6-4578-9e66-b3130ff42bd8\" (UID: \"54c07692-6cf6-4578-9e66-b3130ff42bd8\") "
Sep 17 17:33:13.327491 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:13.326738    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54c07692-6cf6-4578-9e66-b3130ff42bd8-dshm\") pod \"54c07692-6cf6-4578-9e66-b3130ff42bd8\" (UID: \"54c07692-6cf6-4578-9e66-b3130ff42bd8\") "
Sep 17 17:33:13.327491 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:13.326799    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54c07692-6cf6-4578-9e66-b3130ff42bd8-home\") pod \"54c07692-6cf6-4578-9e66-b3130ff42bd8\" (UID: \"54c07692-6cf6-4578-9e66-b3130ff42bd8\") "
Sep 17 17:33:13.327491 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:13.326825    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54c07692-6cf6-4578-9e66-b3130ff42bd8-tmp-dir\") pod \"54c07692-6cf6-4578-9e66-b3130ff42bd8\" (UID: \"54c07692-6cf6-4578-9e66-b3130ff42bd8\") "
Sep 17 17:33:13.327491 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:13.326870    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54c07692-6cf6-4578-9e66-b3130ff42bd8-tls-certs\") pod \"54c07692-6cf6-4578-9e66-b3130ff42bd8\" (UID: \"54c07692-6cf6-4578-9e66-b3130ff42bd8\") "
Sep 17 17:33:13.327491 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:13.326913    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54c07692-6cf6-4578-9e66-b3130ff42bd8-kserve-provision-location\") pod \"54c07692-6cf6-4578-9e66-b3130ff42bd8\" (UID: \"54c07692-6cf6-4578-9e66-b3130ff42bd8\") "
Sep 17 17:33:13.327491 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:13.326955    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54c07692-6cf6-4578-9e66-b3130ff42bd8-cabundle-cert\") pod \"54c07692-6cf6-4578-9e66-b3130ff42bd8\" (UID: \"54c07692-6cf6-4578-9e66-b3130ff42bd8\") "
Sep 17 17:33:13.327491 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:13.326965    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c07692-6cf6-4578-9e66-b3130ff42bd8-model-cache" (OuterVolumeSpecName: "model-cache") pod "54c07692-6cf6-4578-9e66-b3130ff42bd8" (UID: "54c07692-6cf6-4578-9e66-b3130ff42bd8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:33:13.327491 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:13.327006    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7s4qs\" (UniqueName: \"kubernetes.io/projected/54c07692-6cf6-4578-9e66-b3130ff42bd8-kube-api-access-7s4qs\") pod \"54c07692-6cf6-4578-9e66-b3130ff42bd8\" (UID: \"54c07692-6cf6-4578-9e66-b3130ff42bd8\") "
Sep 17 17:33:13.327491 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:13.327211    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54c07692-6cf6-4578-9e66-b3130ff42bd8-model-cache\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:33:13.328474 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:13.328402    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54c07692-6cf6-4578-9e66-b3130ff42bd8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "54c07692-6cf6-4578-9e66-b3130ff42bd8" (UID: "54c07692-6cf6-4578-9e66-b3130ff42bd8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:33:13.328474 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:13.328459    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c07692-6cf6-4578-9e66-b3130ff42bd8-home" (OuterVolumeSpecName: "home") pod "54c07692-6cf6-4578-9e66-b3130ff42bd8" (UID: "54c07692-6cf6-4578-9e66-b3130ff42bd8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:33:13.330211 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:13.330172    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c07692-6cf6-4578-9e66-b3130ff42bd8-dshm" (OuterVolumeSpecName: "dshm") pod "54c07692-6cf6-4578-9e66-b3130ff42bd8" (UID: "54c07692-6cf6-4578-9e66-b3130ff42bd8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:33:13.331234 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:13.331210    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54c07692-6cf6-4578-9e66-b3130ff42bd8-kube-api-access-7s4qs" (OuterVolumeSpecName: "kube-api-access-7s4qs") pod "54c07692-6cf6-4578-9e66-b3130ff42bd8" (UID: "54c07692-6cf6-4578-9e66-b3130ff42bd8"). InnerVolumeSpecName "kube-api-access-7s4qs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:33:13.331485 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:13.331461    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54c07692-6cf6-4578-9e66-b3130ff42bd8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "54c07692-6cf6-4578-9e66-b3130ff42bd8" (UID: "54c07692-6cf6-4578-9e66-b3130ff42bd8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:33:13.336462 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:13.336024    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9" podUID="cbe7478b-b6ed-429c-a89a-f12dc45edd43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 17 17:33:13.349029 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:13.348990    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c07692-6cf6-4578-9e66-b3130ff42bd8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "54c07692-6cf6-4578-9e66-b3130ff42bd8" (UID: "54c07692-6cf6-4578-9e66-b3130ff42bd8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:33:13.363491 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:13.363411    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c07692-6cf6-4578-9e66-b3130ff42bd8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "54c07692-6cf6-4578-9e66-b3130ff42bd8" (UID: "54c07692-6cf6-4578-9e66-b3130ff42bd8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:33:13.428706 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:13.428662    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54c07692-6cf6-4578-9e66-b3130ff42bd8-home\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:33:13.428963 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:13.428946    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54c07692-6cf6-4578-9e66-b3130ff42bd8-tmp-dir\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:33:13.429081 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:13.429056    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54c07692-6cf6-4578-9e66-b3130ff42bd8-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:33:13.429081 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:13.429082    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54c07692-6cf6-4578-9e66-b3130ff42bd8-kserve-provision-location\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:33:13.429272 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:13.429097    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54c07692-6cf6-4578-9e66-b3130ff42bd8-cabundle-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:33:13.429272 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:13.429114    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s4qs\" (UniqueName: \"kubernetes.io/projected/54c07692-6cf6-4578-9e66-b3130ff42bd8-kube-api-access-7s4qs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:33:13.429272 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:13.429128    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54c07692-6cf6-4578-9e66-b3130ff42bd8-dshm\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:33:13.921116 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:13.921075    2566 generic.go:356] "Generic (PLEG): container finished" podID="54c07692-6cf6-4578-9e66-b3130ff42bd8" containerID="d4ce9f7c969f718b81a950f5b716994da22665afec1a4a13eaca866a709a6362" exitCode=0
Sep 17 17:33:13.921312 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:13.921118    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm" event={"ID":"54c07692-6cf6-4578-9e66-b3130ff42bd8","Type":"ContainerDied","Data":"d4ce9f7c969f718b81a950f5b716994da22665afec1a4a13eaca866a709a6362"}
Sep 17 17:33:13.921312 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:13.921149    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm"
Sep 17 17:33:13.921312 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:13.921177    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm" event={"ID":"54c07692-6cf6-4578-9e66-b3130ff42bd8","Type":"ContainerDied","Data":"cb54cd2e61a2f21f9e9f28befdfa74c42fbf191470a40dbb0b0b033330e9a919"}
Sep 17 17:33:13.921312 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:13.921202    2566 scope.go:117] "RemoveContainer" containerID="d4ce9f7c969f718b81a950f5b716994da22665afec1a4a13eaca866a709a6362"
Sep 17 17:33:13.930709 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:13.930685    2566 scope.go:117] "RemoveContainer" containerID="4725ef2221d5bc43feaec294794e1c5d43311debde0ab2810e68878ba1c81418"
Sep 17 17:33:13.940047 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:13.940015    2566 scope.go:117] "RemoveContainer" containerID="d4ce9f7c969f718b81a950f5b716994da22665afec1a4a13eaca866a709a6362"
Sep 17 17:33:13.940360 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:33:13.940339    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4ce9f7c969f718b81a950f5b716994da22665afec1a4a13eaca866a709a6362\": container with ID starting with d4ce9f7c969f718b81a950f5b716994da22665afec1a4a13eaca866a709a6362 not found: ID does not exist" containerID="d4ce9f7c969f718b81a950f5b716994da22665afec1a4a13eaca866a709a6362"
Sep 17 17:33:13.940407 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:13.940371    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4ce9f7c969f718b81a950f5b716994da22665afec1a4a13eaca866a709a6362"} err="failed to get container status \"d4ce9f7c969f718b81a950f5b716994da22665afec1a4a13eaca866a709a6362\": rpc error: code = NotFound desc = could not find container \"d4ce9f7c969f718b81a950f5b716994da22665afec1a4a13eaca866a709a6362\": container with ID starting with d4ce9f7c969f718b81a950f5b716994da22665afec1a4a13eaca866a709a6362 not found: ID does not exist"
Sep 17 17:33:13.940407 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:13.940387    2566 scope.go:117] "RemoveContainer" containerID="4725ef2221d5bc43feaec294794e1c5d43311debde0ab2810e68878ba1c81418"
Sep 17 17:33:13.940676 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:33:13.940648    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4725ef2221d5bc43feaec294794e1c5d43311debde0ab2810e68878ba1c81418\": container with ID starting with 4725ef2221d5bc43feaec294794e1c5d43311debde0ab2810e68878ba1c81418 not found: ID does not exist" containerID="4725ef2221d5bc43feaec294794e1c5d43311debde0ab2810e68878ba1c81418"
Sep 17 17:33:13.940717 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:13.940687    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4725ef2221d5bc43feaec294794e1c5d43311debde0ab2810e68878ba1c81418"} err="failed to get container status \"4725ef2221d5bc43feaec294794e1c5d43311debde0ab2810e68878ba1c81418\": rpc error: code = NotFound desc = could not find container \"4725ef2221d5bc43feaec294794e1c5d43311debde0ab2810e68878ba1c81418\": container with ID starting with 4725ef2221d5bc43feaec294794e1c5d43311debde0ab2810e68878ba1c81418 not found: ID does not exist"
Sep 17 17:33:13.953659 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:13.953627    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm"]
Sep 17 17:33:13.955725 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:13.955702    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-56d5b64f7-psdvm"]
Sep 17 17:33:14.348145 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:14.348107    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54c07692-6cf6-4578-9e66-b3130ff42bd8" path="/var/lib/kubelet/pods/54c07692-6cf6-4578-9e66-b3130ff42bd8/volumes"
Sep 17 17:33:23.345883 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:23.345852    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9"
Sep 17 17:33:23.353812 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:23.353779    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9"
Sep 17 17:33:47.635970 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:47.635901    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"]
Sep 17 17:33:47.636682 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:47.636579    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54c07692-6cf6-4578-9e66-b3130ff42bd8" containerName="main"
Sep 17 17:33:47.636682 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:47.636607    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="54c07692-6cf6-4578-9e66-b3130ff42bd8" containerName="main"
Sep 17 17:33:47.636682 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:47.636625    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54c07692-6cf6-4578-9e66-b3130ff42bd8" containerName="storage-initializer"
Sep 17 17:33:47.636682 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:47.636637    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="54c07692-6cf6-4578-9e66-b3130ff42bd8" containerName="storage-initializer"
Sep 17 17:33:47.636876 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:47.636778    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="54c07692-6cf6-4578-9e66-b3130ff42bd8" containerName="main"
Sep 17 17:33:47.643181 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:47.643148    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"
Sep 17 17:33:47.647116 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:47.647090    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 17 17:33:47.647245 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:47.647090    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-gateway-2-openshift-default-dockercfg-pgs7t\""
Sep 17 17:33:47.647389 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:47.647371    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 17 17:33:47.647456 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:47.647091    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-gw-ca-root-cert\""
Sep 17 17:33:47.653562 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:47.653537    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"]
Sep 17 17:33:47.828975 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:47.828912    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c88771f6-48ea-4b56-a47c-fd7b29bfc918-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-78888675c6-5s9b9\" (UID: \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"
Sep 17 17:33:47.829201 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:47.828983    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c88771f6-48ea-4b56-a47c-fd7b29bfc918-istio-data\") pod \"router-gateway-2-openshift-default-78888675c6-5s9b9\" (UID: \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"
Sep 17 17:33:47.829201 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:47.829013    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c88771f6-48ea-4b56-a47c-fd7b29bfc918-istio-podinfo\") pod \"router-gateway-2-openshift-default-78888675c6-5s9b9\" (UID: \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"
Sep 17 17:33:47.829201 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:47.829106    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c88771f6-48ea-4b56-a47c-fd7b29bfc918-istio-envoy\") pod \"router-gateway-2-openshift-default-78888675c6-5s9b9\" (UID: \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"
Sep 17 17:33:47.829201 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:47.829175    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c88771f6-48ea-4b56-a47c-fd7b29bfc918-workload-certs\") pod \"router-gateway-2-openshift-default-78888675c6-5s9b9\" (UID: \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"
Sep 17 17:33:47.829425 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:47.829274    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88771f6-48ea-4b56-a47c-fd7b29bfc918-workload-socket\") pod \"router-gateway-2-openshift-default-78888675c6-5s9b9\" (UID: \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"
Sep 17 17:33:47.829425 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:47.829352    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88771f6-48ea-4b56-a47c-fd7b29bfc918-credential-socket\") pod \"router-gateway-2-openshift-default-78888675c6-5s9b9\" (UID: \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"
Sep 17 17:33:47.829425 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:47.829387    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c88771f6-48ea-4b56-a47c-fd7b29bfc918-istio-token\") pod \"router-gateway-2-openshift-default-78888675c6-5s9b9\" (UID: \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"
Sep 17 17:33:47.829425 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:47.829419    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7frl7\" (UniqueName: \"kubernetes.io/projected/c88771f6-48ea-4b56-a47c-fd7b29bfc918-kube-api-access-7frl7\") pod \"router-gateway-2-openshift-default-78888675c6-5s9b9\" (UID: \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"
Sep 17 17:33:47.931155 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:47.931007    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c88771f6-48ea-4b56-a47c-fd7b29bfc918-istio-data\") pod \"router-gateway-2-openshift-default-78888675c6-5s9b9\" (UID: \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"
Sep 17 17:33:47.931155 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:47.931107    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c88771f6-48ea-4b56-a47c-fd7b29bfc918-istio-podinfo\") pod \"router-gateway-2-openshift-default-78888675c6-5s9b9\" (UID: \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"
Sep 17 17:33:47.931155 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:47.931158    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c88771f6-48ea-4b56-a47c-fd7b29bfc918-istio-envoy\") pod \"router-gateway-2-openshift-default-78888675c6-5s9b9\" (UID: \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"
Sep 17 17:33:47.931513 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:47.931204    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c88771f6-48ea-4b56-a47c-fd7b29bfc918-workload-certs\") pod \"router-gateway-2-openshift-default-78888675c6-5s9b9\" (UID: \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"
Sep 17 17:33:47.931513 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:47.931270    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88771f6-48ea-4b56-a47c-fd7b29bfc918-workload-socket\") pod \"router-gateway-2-openshift-default-78888675c6-5s9b9\" (UID: \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"
Sep 17 17:33:47.931513 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:47.931337    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88771f6-48ea-4b56-a47c-fd7b29bfc918-credential-socket\") pod \"router-gateway-2-openshift-default-78888675c6-5s9b9\" (UID: \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"
Sep 17 17:33:47.931513 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:47.931368    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c88771f6-48ea-4b56-a47c-fd7b29bfc918-istio-token\") pod \"router-gateway-2-openshift-default-78888675c6-5s9b9\" (UID: \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"
Sep 17 17:33:47.931513 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:47.931411    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7frl7\" (UniqueName: \"kubernetes.io/projected/c88771f6-48ea-4b56-a47c-fd7b29bfc918-kube-api-access-7frl7\") pod \"router-gateway-2-openshift-default-78888675c6-5s9b9\" (UID: \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"
Sep 17 17:33:47.931513 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:47.931487    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c88771f6-48ea-4b56-a47c-fd7b29bfc918-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-78888675c6-5s9b9\" (UID: \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"
Sep 17 17:33:47.931799 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:47.931610    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c88771f6-48ea-4b56-a47c-fd7b29bfc918-istio-data\") pod \"router-gateway-2-openshift-default-78888675c6-5s9b9\" (UID: \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"
Sep 17 17:33:47.931862 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:47.931837    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88771f6-48ea-4b56-a47c-fd7b29bfc918-workload-socket\") pod \"router-gateway-2-openshift-default-78888675c6-5s9b9\" (UID: \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"
Sep 17 17:33:47.932324 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:47.932294    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c88771f6-48ea-4b56-a47c-fd7b29bfc918-workload-certs\") pod \"router-gateway-2-openshift-default-78888675c6-5s9b9\" (UID: \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"
Sep 17 17:33:47.932462 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:47.932403    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88771f6-48ea-4b56-a47c-fd7b29bfc918-credential-socket\") pod \"router-gateway-2-openshift-default-78888675c6-5s9b9\" (UID: \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"
Sep 17 17:33:47.932546 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:47.932479    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c88771f6-48ea-4b56-a47c-fd7b29bfc918-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-78888675c6-5s9b9\" (UID: \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"
Sep 17 17:33:47.935740 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:47.935562    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c88771f6-48ea-4b56-a47c-fd7b29bfc918-istio-podinfo\") pod \"router-gateway-2-openshift-default-78888675c6-5s9b9\" (UID: \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"
Sep 17 17:33:47.935740 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:47.935672    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c88771f6-48ea-4b56-a47c-fd7b29bfc918-istio-envoy\") pod \"router-gateway-2-openshift-default-78888675c6-5s9b9\" (UID: \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"
Sep 17 17:33:47.942404 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:47.942362    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c88771f6-48ea-4b56-a47c-fd7b29bfc918-istio-token\") pod \"router-gateway-2-openshift-default-78888675c6-5s9b9\" (UID: \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"
Sep 17 17:33:47.943007 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:47.942978    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7frl7\" (UniqueName: \"kubernetes.io/projected/c88771f6-48ea-4b56-a47c-fd7b29bfc918-kube-api-access-7frl7\") pod \"router-gateway-2-openshift-default-78888675c6-5s9b9\" (UID: \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"
Sep 17 17:33:47.958413 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:47.958386    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"
Sep 17 17:33:48.276302 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:48.276272    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"]
Sep 17 17:33:48.277962 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:33:48.277930    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc88771f6_48ea_4b56_a47c_fd7b29bfc918.slice/crio-e06f96b6dd184498d9c86d6cd7096002caddc9edd4fd8a48f7ffb490887738b3 WatchSource:0}: Error finding container e06f96b6dd184498d9c86d6cd7096002caddc9edd4fd8a48f7ffb490887738b3: Status 404 returned error can't find the container with id e06f96b6dd184498d9c86d6cd7096002caddc9edd4fd8a48f7ffb490887738b3
Sep 17 17:33:48.280112 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:48.280079    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 17:33:48.280229 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:48.280154    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 17:33:48.280229 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:48.280181    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 17:33:49.114907 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:49.114866    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9" event={"ID":"c88771f6-48ea-4b56-a47c-fd7b29bfc918","Type":"ContainerStarted","Data":"51731d91c2660c23bc6b7385ff84681d13d9851a55d96858dac739147f16c272"}
Sep 17 17:33:49.115346 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:49.114923    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9" event={"ID":"c88771f6-48ea-4b56-a47c-fd7b29bfc918","Type":"ContainerStarted","Data":"e06f96b6dd184498d9c86d6cd7096002caddc9edd4fd8a48f7ffb490887738b3"}
Sep 17 17:33:49.246163 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:49.246133    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-85698b65fd-8sxr9_cbe7478b-b6ed-429c-a89a-f12dc45edd43/storage-initializer/0.log"
Sep 17 17:33:49.265979 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:49.265945    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-85698b65fd-8sxr9_cbe7478b-b6ed-429c-a89a-f12dc45edd43/main/0.log"
Sep 17 17:33:49.958641 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:49.958599    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"
Sep 17 17:33:49.963423 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:49.963391    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"
Sep 17 17:33:49.989211 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:49.989157    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9" podStartSLOduration=2.989140308 podStartE2EDuration="2.989140308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:33:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:33:49.147187868 +0000 UTC m=+3115.041667010" watchObservedRunningTime="2026-09-17 17:33:49.989140308 +0000 UTC m=+3115.883619447"
Sep 17 17:33:50.120473 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:50.120430    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"
Sep 17 17:33:50.121422 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:33:50.121401    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"
Sep 17 17:34:00.182620 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:00.182580    2566 generic.go:356] "Generic (PLEG): container finished" podID="c88771f6-48ea-4b56-a47c-fd7b29bfc918" containerID="51731d91c2660c23bc6b7385ff84681d13d9851a55d96858dac739147f16c272" exitCode=0
Sep 17 17:34:00.183077 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:00.182653    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9" event={"ID":"c88771f6-48ea-4b56-a47c-fd7b29bfc918","Type":"ContainerDied","Data":"51731d91c2660c23bc6b7385ff84681d13d9851a55d96858dac739147f16c272"}
Sep 17 17:34:00.183077 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:00.183068    2566 scope.go:117] "RemoveContainer" containerID="51731d91c2660c23bc6b7385ff84681d13d9851a55d96858dac739147f16c272"
Sep 17 17:34:00.184005 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:00.183980    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 17:34:00.184066 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:00.184053    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 17:34:00.184104 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:00.184088    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 17:34:01.190296 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:01.190259    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9" event={"ID":"c88771f6-48ea-4b56-a47c-fd7b29bfc918","Type":"ContainerStarted","Data":"02d3b9395251b4565f3849f61a75b38ba0dcc4ae423263a81dd57d8d18974b3a"}
Sep 17 17:34:01.959281 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:01.959240    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"
Sep 17 17:34:01.960398 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:01.960366    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9" podUID="c88771f6-48ea-4b56-a47c-fd7b29bfc918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 17 17:34:02.378882 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:02.378835    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6"]
Sep 17 17:34:02.382640 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:02.382613    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6"
Sep 17 17:34:02.394661 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:02.394640    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 17 17:34:02.394903 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:02.394889    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:34:02.395135 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:02.395117    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-svcd9\""
Sep 17 17:34:02.395529 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:02.395513    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 17 17:34:02.414755 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:02.414659    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6"]
Sep 17 17:34:02.460496 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:02.460454    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6\" (UID: \"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6"
Sep 17 17:34:02.460496 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:02.460500    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6\" (UID: \"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6"
Sep 17 17:34:02.460727 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:02.460530    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6\" (UID: \"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6"
Sep 17 17:34:02.460727 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:02.460558    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6\" (UID: \"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6"
Sep 17 17:34:02.460727 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:02.460573    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6\" (UID: \"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6"
Sep 17 17:34:02.460852 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:02.460717    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gwqp\" (UniqueName: \"kubernetes.io/projected/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-kube-api-access-8gwqp\") pod \"router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6\" (UID: \"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6"
Sep 17 17:34:02.460852 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:02.460764    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-home\") pod \"router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6\" (UID: \"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6"
Sep 17 17:34:02.460852 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:02.460794    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6\" (UID: \"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6"
Sep 17 17:34:02.561837 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:02.561794    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6\" (UID: \"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6"
Sep 17 17:34:02.561837 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:02.561833    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6\" (UID: \"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6"
Sep 17 17:34:02.562119 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:02.561860    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6\" (UID: \"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6"
Sep 17 17:34:02.562119 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:02.561991    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6\" (UID: \"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6"
Sep 17 17:34:02.562119 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:02.562029    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6\" (UID: \"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6"
Sep 17 17:34:02.562119 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:02.562100    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gwqp\" (UniqueName: \"kubernetes.io/projected/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-kube-api-access-8gwqp\") pod \"router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6\" (UID: \"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6"
Sep 17 17:34:02.562341 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:02.562166    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-home\") pod \"router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6\" (UID: \"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6"
Sep 17 17:34:02.562341 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:02.562217    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6\" (UID: \"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6"
Sep 17 17:34:02.562341 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:02.562286    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6\" (UID: \"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6"
Sep 17 17:34:02.562341 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:02.562324    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6\" (UID: \"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6"
Sep 17 17:34:02.562623 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:02.562595    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6\" (UID: \"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6"
Sep 17 17:34:02.562761 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:02.562656    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-home\") pod \"router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6\" (UID: \"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6"
Sep 17 17:34:02.562955 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:02.562933    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6\" (UID: \"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6"
Sep 17 17:34:02.564611 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:02.564576    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6\" (UID: \"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6"
Sep 17 17:34:02.564816 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:02.564797    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6\" (UID: \"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6"
Sep 17 17:34:02.580103 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:02.580079    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gwqp\" (UniqueName: \"kubernetes.io/projected/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-kube-api-access-8gwqp\") pod \"router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6\" (UID: \"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6"
Sep 17 17:34:02.692726 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:02.692619    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6"
Sep 17 17:34:02.959942 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:02.959829    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9" podUID="c88771f6-48ea-4b56-a47c-fd7b29bfc918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 17 17:34:03.018849 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:03.016785    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6"]
Sep 17 17:34:03.019570 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:34:03.019530    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eadd7e2_bbf7_462e_b2aa_27bec9c97afd.slice/crio-33de892c66ead42c3a6500c6cd07ccbc2507fec9bc03443005166ad40918f0d0 WatchSource:0}: Error finding container 33de892c66ead42c3a6500c6cd07ccbc2507fec9bc03443005166ad40918f0d0: Status 404 returned error can't find the container with id 33de892c66ead42c3a6500c6cd07ccbc2507fec9bc03443005166ad40918f0d0
Sep 17 17:34:03.207919 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:03.207874    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6" event={"ID":"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd","Type":"ContainerStarted","Data":"97b241a4b7fa653c98764848cd716b8d3977ef2fcd2f860a04d43ba330ace0c8"}
Sep 17 17:34:03.208237 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:03.208198    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6" event={"ID":"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd","Type":"ContainerStarted","Data":"33de892c66ead42c3a6500c6cd07ccbc2507fec9bc03443005166ad40918f0d0"}
Sep 17 17:34:03.959741 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:03.959687    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9" podUID="c88771f6-48ea-4b56-a47c-fd7b29bfc918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 17 17:34:04.959076 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:04.958987    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9" podUID="c88771f6-48ea-4b56-a47c-fd7b29bfc918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 17 17:34:05.121088 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:05.121041    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"
Sep 17 17:34:05.959532 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:05.959487    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9" podUID="c88771f6-48ea-4b56-a47c-fd7b29bfc918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 17 17:34:06.231206 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:06.231113    2566 generic.go:356] "Generic (PLEG): container finished" podID="8eadd7e2-bbf7-462e-b2aa-27bec9c97afd" containerID="97b241a4b7fa653c98764848cd716b8d3977ef2fcd2f860a04d43ba330ace0c8" exitCode=0
Sep 17 17:34:06.231206 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:06.231185    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6" event={"ID":"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd","Type":"ContainerDied","Data":"97b241a4b7fa653c98764848cd716b8d3977ef2fcd2f860a04d43ba330ace0c8"}
Sep 17 17:34:06.963286 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:06.963253    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"
Sep 17 17:34:06.964335 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:06.964316    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"
Sep 17 17:34:07.239601 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:07.239489    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6" event={"ID":"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd","Type":"ContainerStarted","Data":"65fd760d8aa9c63ae8342c22a375da229acfe07e73f172e6bbe45cbd01aab40c"}
Sep 17 17:34:07.268098 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:07.268033    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6" podStartSLOduration=2.466040932 podStartE2EDuration="5.268016198s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.801975266s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:07.264494672 +0000 UTC m=+3133.158973811" watchObservedRunningTime="2026-09-17 17:34:07.268016198 +0000 UTC m=+3133.162495359"
Sep 17 17:34:12.693017 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:12.692971    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6"
Sep 17 17:34:12.693536 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:12.693123    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6"
Sep 17 17:34:12.694927 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:12.694896    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6" podUID="8eadd7e2-bbf7-462e-b2aa-27bec9c97afd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 17 17:34:22.693604 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:22.693551    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6" podUID="8eadd7e2-bbf7-462e-b2aa-27bec9c97afd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 17 17:34:24.716663 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:24.716628    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9"]
Sep 17 17:34:32.693571 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:32.693522    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6" podUID="8eadd7e2-bbf7-462e-b2aa-27bec9c97afd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 17 17:34:39.744390 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:39.744322    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9" podUID="cbe7478b-b6ed-429c-a89a-f12dc45edd43" containerName="main" containerID="cri-o://3e990809f1edb88f1ad142d495df8b449a6e4c07a9ca5dde0048e867f446bc20" gracePeriod=45
Sep 17 17:34:40.402402 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:40.402371    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9"
Sep 17 17:34:40.424307 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:40.424272    2566 generic.go:356] "Generic (PLEG): container finished" podID="cbe7478b-b6ed-429c-a89a-f12dc45edd43" containerID="3e990809f1edb88f1ad142d495df8b449a6e4c07a9ca5dde0048e867f446bc20" exitCode=0
Sep 17 17:34:40.424501 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:40.424336    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9"
Sep 17 17:34:40.424501 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:40.424360    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9" event={"ID":"cbe7478b-b6ed-429c-a89a-f12dc45edd43","Type":"ContainerDied","Data":"3e990809f1edb88f1ad142d495df8b449a6e4c07a9ca5dde0048e867f446bc20"}
Sep 17 17:34:40.424501 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:40.424426    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9" event={"ID":"cbe7478b-b6ed-429c-a89a-f12dc45edd43","Type":"ContainerDied","Data":"efe244e2522dfabce8b3052c1fb32824cf05ef758bacc4e35b36e5a306651ea8"}
Sep 17 17:34:40.424501 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:40.424461    2566 scope.go:117] "RemoveContainer" containerID="3e990809f1edb88f1ad142d495df8b449a6e4c07a9ca5dde0048e867f446bc20"
Sep 17 17:34:40.435063 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:40.435040    2566 scope.go:117] "RemoveContainer" containerID="d16745a2658e76289d0ca7650d5840abe8b9ff3579bf55273478994a35984756"
Sep 17 17:34:40.444342 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:40.444313    2566 scope.go:117] "RemoveContainer" containerID="3e990809f1edb88f1ad142d495df8b449a6e4c07a9ca5dde0048e867f446bc20"
Sep 17 17:34:40.444706 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:34:40.444686    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e990809f1edb88f1ad142d495df8b449a6e4c07a9ca5dde0048e867f446bc20\": container with ID starting with 3e990809f1edb88f1ad142d495df8b449a6e4c07a9ca5dde0048e867f446bc20 not found: ID does not exist" containerID="3e990809f1edb88f1ad142d495df8b449a6e4c07a9ca5dde0048e867f446bc20"
Sep 17 17:34:40.444780 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:40.444719    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e990809f1edb88f1ad142d495df8b449a6e4c07a9ca5dde0048e867f446bc20"} err="failed to get container status \"3e990809f1edb88f1ad142d495df8b449a6e4c07a9ca5dde0048e867f446bc20\": rpc error: code = NotFound desc = could not find container \"3e990809f1edb88f1ad142d495df8b449a6e4c07a9ca5dde0048e867f446bc20\": container with ID starting with 3e990809f1edb88f1ad142d495df8b449a6e4c07a9ca5dde0048e867f446bc20 not found: ID does not exist"
Sep 17 17:34:40.444780 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:40.444738    2566 scope.go:117] "RemoveContainer" containerID="d16745a2658e76289d0ca7650d5840abe8b9ff3579bf55273478994a35984756"
Sep 17 17:34:40.445043 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:34:40.445016    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d16745a2658e76289d0ca7650d5840abe8b9ff3579bf55273478994a35984756\": container with ID starting with d16745a2658e76289d0ca7650d5840abe8b9ff3579bf55273478994a35984756 not found: ID does not exist" containerID="d16745a2658e76289d0ca7650d5840abe8b9ff3579bf55273478994a35984756"
Sep 17 17:34:40.445102 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:40.445056    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d16745a2658e76289d0ca7650d5840abe8b9ff3579bf55273478994a35984756"} err="failed to get container status \"d16745a2658e76289d0ca7650d5840abe8b9ff3579bf55273478994a35984756\": rpc error: code = NotFound desc = could not find container \"d16745a2658e76289d0ca7650d5840abe8b9ff3579bf55273478994a35984756\": container with ID starting with d16745a2658e76289d0ca7650d5840abe8b9ff3579bf55273478994a35984756 not found: ID does not exist"
Sep 17 17:34:40.509467 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:40.509344    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbe7478b-b6ed-429c-a89a-f12dc45edd43-dshm\") pod \"cbe7478b-b6ed-429c-a89a-f12dc45edd43\" (UID: \"cbe7478b-b6ed-429c-a89a-f12dc45edd43\") "
Sep 17 17:34:40.509467 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:40.509424    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbe7478b-b6ed-429c-a89a-f12dc45edd43-tmp-dir\") pod \"cbe7478b-b6ed-429c-a89a-f12dc45edd43\" (UID: \"cbe7478b-b6ed-429c-a89a-f12dc45edd43\") "
Sep 17 17:34:40.509724 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:40.509492    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbe7478b-b6ed-429c-a89a-f12dc45edd43-kserve-provision-location\") pod \"cbe7478b-b6ed-429c-a89a-f12dc45edd43\" (UID: \"cbe7478b-b6ed-429c-a89a-f12dc45edd43\") "
Sep 17 17:34:40.509724 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:40.509524    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbe7478b-b6ed-429c-a89a-f12dc45edd43-home\") pod \"cbe7478b-b6ed-429c-a89a-f12dc45edd43\" (UID: \"cbe7478b-b6ed-429c-a89a-f12dc45edd43\") "
Sep 17 17:34:40.509724 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:40.509568    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4bzd\" (UniqueName: \"kubernetes.io/projected/cbe7478b-b6ed-429c-a89a-f12dc45edd43-kube-api-access-k4bzd\") pod \"cbe7478b-b6ed-429c-a89a-f12dc45edd43\" (UID: \"cbe7478b-b6ed-429c-a89a-f12dc45edd43\") "
Sep 17 17:34:40.509724 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:40.509596    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbe7478b-b6ed-429c-a89a-f12dc45edd43-model-cache\") pod \"cbe7478b-b6ed-429c-a89a-f12dc45edd43\" (UID: \"cbe7478b-b6ed-429c-a89a-f12dc45edd43\") "
Sep 17 17:34:40.509724 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:40.509642    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbe7478b-b6ed-429c-a89a-f12dc45edd43-cabundle-cert\") pod \"cbe7478b-b6ed-429c-a89a-f12dc45edd43\" (UID: \"cbe7478b-b6ed-429c-a89a-f12dc45edd43\") "
Sep 17 17:34:40.509724 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:40.509693    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbe7478b-b6ed-429c-a89a-f12dc45edd43-tls-certs\") pod \"cbe7478b-b6ed-429c-a89a-f12dc45edd43\" (UID: \"cbe7478b-b6ed-429c-a89a-f12dc45edd43\") "
Sep 17 17:34:40.510393 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:40.510359    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbe7478b-b6ed-429c-a89a-f12dc45edd43-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cbe7478b-b6ed-429c-a89a-f12dc45edd43" (UID: "cbe7478b-b6ed-429c-a89a-f12dc45edd43"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:34:40.510706 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:40.510610    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe7478b-b6ed-429c-a89a-f12dc45edd43-model-cache" (OuterVolumeSpecName: "model-cache") pod "cbe7478b-b6ed-429c-a89a-f12dc45edd43" (UID: "cbe7478b-b6ed-429c-a89a-f12dc45edd43"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:34:40.510895 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:40.510874    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe7478b-b6ed-429c-a89a-f12dc45edd43-home" (OuterVolumeSpecName: "home") pod "cbe7478b-b6ed-429c-a89a-f12dc45edd43" (UID: "cbe7478b-b6ed-429c-a89a-f12dc45edd43"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:34:40.512628 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:40.512591    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbe7478b-b6ed-429c-a89a-f12dc45edd43-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cbe7478b-b6ed-429c-a89a-f12dc45edd43" (UID: "cbe7478b-b6ed-429c-a89a-f12dc45edd43"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:34:40.512740 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:40.512619    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe7478b-b6ed-429c-a89a-f12dc45edd43-dshm" (OuterVolumeSpecName: "dshm") pod "cbe7478b-b6ed-429c-a89a-f12dc45edd43" (UID: "cbe7478b-b6ed-429c-a89a-f12dc45edd43"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:34:40.512952 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:40.512927    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbe7478b-b6ed-429c-a89a-f12dc45edd43-kube-api-access-k4bzd" (OuterVolumeSpecName: "kube-api-access-k4bzd") pod "cbe7478b-b6ed-429c-a89a-f12dc45edd43" (UID: "cbe7478b-b6ed-429c-a89a-f12dc45edd43"). InnerVolumeSpecName "kube-api-access-k4bzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:34:40.523854 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:40.523825    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe7478b-b6ed-429c-a89a-f12dc45edd43-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cbe7478b-b6ed-429c-a89a-f12dc45edd43" (UID: "cbe7478b-b6ed-429c-a89a-f12dc45edd43"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:34:40.538105 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:40.538065    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe7478b-b6ed-429c-a89a-f12dc45edd43-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cbe7478b-b6ed-429c-a89a-f12dc45edd43" (UID: "cbe7478b-b6ed-429c-a89a-f12dc45edd43"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:34:40.610684 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:40.610628    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbe7478b-b6ed-429c-a89a-f12dc45edd43-home\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:34:40.610684 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:40.610675    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4bzd\" (UniqueName: \"kubernetes.io/projected/cbe7478b-b6ed-429c-a89a-f12dc45edd43-kube-api-access-k4bzd\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:34:40.610684 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:40.610686    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbe7478b-b6ed-429c-a89a-f12dc45edd43-model-cache\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:34:40.610684 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:40.610695    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbe7478b-b6ed-429c-a89a-f12dc45edd43-cabundle-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:34:40.610957 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:40.610702    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbe7478b-b6ed-429c-a89a-f12dc45edd43-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:34:40.610957 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:40.610709    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbe7478b-b6ed-429c-a89a-f12dc45edd43-dshm\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:34:40.610957 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:40.610716    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbe7478b-b6ed-429c-a89a-f12dc45edd43-tmp-dir\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:34:40.610957 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:40.610726    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbe7478b-b6ed-429c-a89a-f12dc45edd43-kserve-provision-location\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:34:40.751418 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:40.751378    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9"]
Sep 17 17:34:40.756880 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:40.756841    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9"]
Sep 17 17:34:41.347634 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:41.347573    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85698b65fd-8sxr9" podUID="cbe7478b-b6ed-429c-a89a-f12dc45edd43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": context deadline exceeded"
Sep 17 17:34:42.348076 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:42.348040    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbe7478b-b6ed-429c-a89a-f12dc45edd43" path="/var/lib/kubelet/pods/cbe7478b-b6ed-429c-a89a-f12dc45edd43/volumes"
Sep 17 17:34:42.693703 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:42.693605    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6" podUID="8eadd7e2-bbf7-462e-b2aa-27bec9c97afd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 17 17:34:50.389662 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:50.389620    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq"]
Sep 17 17:34:50.390279 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:50.389995    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbe7478b-b6ed-429c-a89a-f12dc45edd43" containerName="storage-initializer"
Sep 17 17:34:50.390279 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:50.390009    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbe7478b-b6ed-429c-a89a-f12dc45edd43" containerName="storage-initializer"
Sep 17 17:34:50.390279 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:50.390021    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbe7478b-b6ed-429c-a89a-f12dc45edd43" containerName="main"
Sep 17 17:34:50.390279 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:50.390026    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbe7478b-b6ed-429c-a89a-f12dc45edd43" containerName="main"
Sep 17 17:34:50.390279 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:50.390090    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbe7478b-b6ed-429c-a89a-f12dc45edd43" containerName="main"
Sep 17 17:34:50.394792 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:50.394765    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq"
Sep 17 17:34:50.398156 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:50.398128    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-kb662\""
Sep 17 17:34:50.398331 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:50.398279    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Sep 17 17:34:50.398331 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:50.398309    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Sep 17 17:34:50.398671 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:50.398651    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:34:50.398834 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:50.398807    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Sep 17 17:34:50.399169 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:50.399151    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Sep 17 17:34:50.430077 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:50.430037    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq"]
Sep 17 17:34:50.489950 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:50.489918    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-dshm\") pod \"lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq\" (UID: \"4ad0cf79-454b-4234-8c0e-a529ee43eaaf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq"
Sep 17 17:34:50.489950 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:50.489955    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq\" (UID: \"4ad0cf79-454b-4234-8c0e-a529ee43eaaf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq"
Sep 17 17:34:50.490246 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:50.489991    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq\" (UID: \"4ad0cf79-454b-4234-8c0e-a529ee43eaaf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq"
Sep 17 17:34:50.490246 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:50.490090    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq\" (UID: \"4ad0cf79-454b-4234-8c0e-a529ee43eaaf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq"
Sep 17 17:34:50.490246 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:50.490118    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-home\") pod \"lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq\" (UID: \"4ad0cf79-454b-4234-8c0e-a529ee43eaaf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq"
Sep 17 17:34:50.490246 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:50.490135    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-tls-certs\") pod \"lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq\" (UID: \"4ad0cf79-454b-4234-8c0e-a529ee43eaaf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq"
Sep 17 17:34:50.490246 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:50.490198    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5r8g\" (UniqueName: \"kubernetes.io/projected/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-kube-api-access-m5r8g\") pod \"lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq\" (UID: \"4ad0cf79-454b-4234-8c0e-a529ee43eaaf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq"
Sep 17 17:34:50.490246 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:50.490227    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-model-cache\") pod \"lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq\" (UID: \"4ad0cf79-454b-4234-8c0e-a529ee43eaaf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq"
Sep 17 17:34:50.590780 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:50.590727    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5r8g\" (UniqueName: \"kubernetes.io/projected/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-kube-api-access-m5r8g\") pod \"lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq\" (UID: \"4ad0cf79-454b-4234-8c0e-a529ee43eaaf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq"
Sep 17 17:34:50.590780 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:50.590767    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-model-cache\") pod \"lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq\" (UID: \"4ad0cf79-454b-4234-8c0e-a529ee43eaaf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq"
Sep 17 17:34:50.591067 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:50.590813    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-dshm\") pod \"lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq\" (UID: \"4ad0cf79-454b-4234-8c0e-a529ee43eaaf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq"
Sep 17 17:34:50.591067 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:50.590843    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq\" (UID: \"4ad0cf79-454b-4234-8c0e-a529ee43eaaf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq"
Sep 17 17:34:50.591067 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:50.590894    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq\" (UID: \"4ad0cf79-454b-4234-8c0e-a529ee43eaaf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq"
Sep 17 17:34:50.591067 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:50.590937    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq\" (UID: \"4ad0cf79-454b-4234-8c0e-a529ee43eaaf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq"
Sep 17 17:34:50.591067 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:50.591042    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-home\") pod \"lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq\" (UID: \"4ad0cf79-454b-4234-8c0e-a529ee43eaaf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq"
Sep 17 17:34:50.591326 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:50.591072    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-tls-certs\") pod \"lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq\" (UID: \"4ad0cf79-454b-4234-8c0e-a529ee43eaaf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq"
Sep 17 17:34:50.591326 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:50.591263    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-model-cache\") pod \"lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq\" (UID: \"4ad0cf79-454b-4234-8c0e-a529ee43eaaf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq"
Sep 17 17:34:50.591453 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:50.591345    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq\" (UID: \"4ad0cf79-454b-4234-8c0e-a529ee43eaaf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq"
Sep 17 17:34:50.591453 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:50.591388    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq\" (UID: \"4ad0cf79-454b-4234-8c0e-a529ee43eaaf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq"
Sep 17 17:34:50.591592 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:50.591558    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-home\") pod \"lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq\" (UID: \"4ad0cf79-454b-4234-8c0e-a529ee43eaaf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq"
Sep 17 17:34:50.591650 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:50.591632    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq\" (UID: \"4ad0cf79-454b-4234-8c0e-a529ee43eaaf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq"
Sep 17 17:34:50.593529 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:50.593499    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-dshm\") pod \"lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq\" (UID: \"4ad0cf79-454b-4234-8c0e-a529ee43eaaf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq"
Sep 17 17:34:50.593724 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:50.593707    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-tls-certs\") pod \"lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq\" (UID: \"4ad0cf79-454b-4234-8c0e-a529ee43eaaf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq"
Sep 17 17:34:50.603311 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:50.603271    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5r8g\" (UniqueName: \"kubernetes.io/projected/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-kube-api-access-m5r8g\") pod \"lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq\" (UID: \"4ad0cf79-454b-4234-8c0e-a529ee43eaaf\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq"
Sep 17 17:34:50.708020 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:50.707913    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq"
Sep 17 17:34:50.996448 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:50.996401    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq"]
Sep 17 17:34:50.997081 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:34:50.997053    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ad0cf79_454b_4234_8c0e_a529ee43eaaf.slice/crio-cd2bc8b441a90e204cbec4febc559d4fc16f75300e81139e7a9f5f4e4ccf0341 WatchSource:0}: Error finding container cd2bc8b441a90e204cbec4febc559d4fc16f75300e81139e7a9f5f4e4ccf0341: Status 404 returned error can't find the container with id cd2bc8b441a90e204cbec4febc559d4fc16f75300e81139e7a9f5f4e4ccf0341
Sep 17 17:34:51.486161 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:51.486116    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq" event={"ID":"4ad0cf79-454b-4234-8c0e-a529ee43eaaf","Type":"ContainerStarted","Data":"3b1a962cc95f6aec10008f445ac8e4ccfc19f9772f7ad89ed37a978a92a172e6"}
Sep 17 17:34:51.486694 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:51.486177    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq" event={"ID":"4ad0cf79-454b-4234-8c0e-a529ee43eaaf","Type":"ContainerStarted","Data":"cd2bc8b441a90e204cbec4febc559d4fc16f75300e81139e7a9f5f4e4ccf0341"}
Sep 17 17:34:52.693818 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:52.693767    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6" podUID="8eadd7e2-bbf7-462e-b2aa-27bec9c97afd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 17 17:34:54.504047 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:54.504013    2566 generic.go:356] "Generic (PLEG): container finished" podID="4ad0cf79-454b-4234-8c0e-a529ee43eaaf" containerID="3b1a962cc95f6aec10008f445ac8e4ccfc19f9772f7ad89ed37a978a92a172e6" exitCode=0
Sep 17 17:34:54.504529 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:54.504077    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq" event={"ID":"4ad0cf79-454b-4234-8c0e-a529ee43eaaf","Type":"ContainerDied","Data":"3b1a962cc95f6aec10008f445ac8e4ccfc19f9772f7ad89ed37a978a92a172e6"}
Sep 17 17:34:55.512744 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:55.512703    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq" event={"ID":"4ad0cf79-454b-4234-8c0e-a529ee43eaaf","Type":"ContainerStarted","Data":"3c310347031183efb6b16b987be99e6527a7001479616517a4ce3d2c67f21254"}
Sep 17 17:34:55.541680 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:55.541547    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq" podStartSLOduration=2.159701835 podStartE2EDuration="5.54152519s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.381823355s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:55.53962808 +0000 UTC m=+3181.434107229" watchObservedRunningTime="2026-09-17 17:34:55.54152519 +0000 UTC m=+3181.436004330"
Sep 17 17:34:55.708998 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:34:55.708960    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq"
Sep 17 17:35:00.708665 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:35:00.708620    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq"
Sep 17 17:35:00.710486 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:35:00.710452    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq" podUID="4ad0cf79-454b-4234-8c0e-a529ee43eaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 17 17:35:02.693251 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:35:02.693198    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6" podUID="8eadd7e2-bbf7-462e-b2aa-27bec9c97afd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 17 17:35:10.708808 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:35:10.708760    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq" podUID="4ad0cf79-454b-4234-8c0e-a529ee43eaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 17 17:35:12.693487 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:35:12.693420    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6" podUID="8eadd7e2-bbf7-462e-b2aa-27bec9c97afd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 17 17:35:20.708715 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:35:20.708648    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq" podUID="4ad0cf79-454b-4234-8c0e-a529ee43eaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 17 17:35:22.693476 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:35:22.693406    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6" podUID="8eadd7e2-bbf7-462e-b2aa-27bec9c97afd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 17 17:35:30.709388 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:35:30.709330    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq" podUID="4ad0cf79-454b-4234-8c0e-a529ee43eaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 17 17:35:32.693403 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:35:32.693356    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6" podUID="8eadd7e2-bbf7-462e-b2aa-27bec9c97afd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 17 17:35:40.708611 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:35:40.708509    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq" podUID="4ad0cf79-454b-4234-8c0e-a529ee43eaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 17 17:35:42.693200 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:35:42.693147    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6" podUID="8eadd7e2-bbf7-462e-b2aa-27bec9c97afd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 17 17:35:50.708876 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:35:50.708818    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq" podUID="4ad0cf79-454b-4234-8c0e-a529ee43eaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 17 17:35:52.693861 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:35:52.693810    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6" podUID="8eadd7e2-bbf7-462e-b2aa-27bec9c97afd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 17 17:36:00.708749 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:00.708693    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq" podUID="4ad0cf79-454b-4234-8c0e-a529ee43eaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 17 17:36:02.693999 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:02.693942    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6" podUID="8eadd7e2-bbf7-462e-b2aa-27bec9c97afd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 17 17:36:10.708834 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:10.708772    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq" podUID="4ad0cf79-454b-4234-8c0e-a529ee43eaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 17 17:36:12.704100 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:12.704066    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6"
Sep 17 17:36:12.712353 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:12.712322    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6"
Sep 17 17:36:20.718730 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:20.718697    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq"
Sep 17 17:36:20.726864 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:20.726835    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq"
Sep 17 17:36:22.447250 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:22.447215    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6"]
Sep 17 17:36:37.476289 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:37.476208    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6" podUID="8eadd7e2-bbf7-462e-b2aa-27bec9c97afd" containerName="main" containerID="cri-o://65fd760d8aa9c63ae8342c22a375da229acfe07e73f172e6bbe45cbd01aab40c" gracePeriod=45
Sep 17 17:36:38.546627 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:38.546597    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6"
Sep 17 17:36:38.594207 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:38.594100    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gwqp\" (UniqueName: \"kubernetes.io/projected/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-kube-api-access-8gwqp\") pod \"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd\" (UID: \"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd\") "
Sep 17 17:36:38.594207 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:38.594196    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-tls-certs\") pod \"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd\" (UID: \"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd\") "
Sep 17 17:36:38.594207 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:38.594215    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-cabundle-cert\") pod \"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd\" (UID: \"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd\") "
Sep 17 17:36:38.594540 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:38.594252    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-kserve-provision-location\") pod \"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd\" (UID: \"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd\") "
Sep 17 17:36:38.594540 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:38.594277    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-dshm\") pod \"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd\" (UID: \"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd\") "
Sep 17 17:36:38.594540 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:38.594302    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-tmp-dir\") pod \"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd\" (UID: \"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd\") "
Sep 17 17:36:38.594540 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:38.594342    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-home\") pod \"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd\" (UID: \"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd\") "
Sep 17 17:36:38.594540 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:38.594364    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-model-cache\") pod \"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd\" (UID: \"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd\") "
Sep 17 17:36:38.595323 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:38.595252    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8eadd7e2-bbf7-462e-b2aa-27bec9c97afd" (UID: "8eadd7e2-bbf7-462e-b2aa-27bec9c97afd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:36:38.595323 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:38.595309    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-home" (OuterVolumeSpecName: "home") pod "8eadd7e2-bbf7-462e-b2aa-27bec9c97afd" (UID: "8eadd7e2-bbf7-462e-b2aa-27bec9c97afd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:36:38.595531 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:38.595374    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-model-cache" (OuterVolumeSpecName: "model-cache") pod "8eadd7e2-bbf7-462e-b2aa-27bec9c97afd" (UID: "8eadd7e2-bbf7-462e-b2aa-27bec9c97afd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:36:38.597260 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:38.597205    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8eadd7e2-bbf7-462e-b2aa-27bec9c97afd" (UID: "8eadd7e2-bbf7-462e-b2aa-27bec9c97afd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:36:38.597260 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:38.597221    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-dshm" (OuterVolumeSpecName: "dshm") pod "8eadd7e2-bbf7-462e-b2aa-27bec9c97afd" (UID: "8eadd7e2-bbf7-462e-b2aa-27bec9c97afd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:36:38.597260 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:38.597242    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-kube-api-access-8gwqp" (OuterVolumeSpecName: "kube-api-access-8gwqp") pod "8eadd7e2-bbf7-462e-b2aa-27bec9c97afd" (UID: "8eadd7e2-bbf7-462e-b2aa-27bec9c97afd"). InnerVolumeSpecName "kube-api-access-8gwqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:36:38.613910 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:38.613843    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8eadd7e2-bbf7-462e-b2aa-27bec9c97afd" (UID: "8eadd7e2-bbf7-462e-b2aa-27bec9c97afd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:36:38.626028 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:38.625987    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8eadd7e2-bbf7-462e-b2aa-27bec9c97afd" (UID: "8eadd7e2-bbf7-462e-b2aa-27bec9c97afd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:36:38.695168 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:38.695130    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:36:38.695168 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:38.695162    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-cabundle-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:36:38.695168 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:38.695172    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-kserve-provision-location\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:36:38.695403 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:38.695180    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-dshm\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:36:38.695403 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:38.695190    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-tmp-dir\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:36:38.695403 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:38.695198    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-home\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:36:38.695403 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:38.695204    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-model-cache\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:36:38.695403 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:38.695212    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gwqp\" (UniqueName: \"kubernetes.io/projected/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd-kube-api-access-8gwqp\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:36:39.116287 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:39.116241    2566 generic.go:356] "Generic (PLEG): container finished" podID="8eadd7e2-bbf7-462e-b2aa-27bec9c97afd" containerID="65fd760d8aa9c63ae8342c22a375da229acfe07e73f172e6bbe45cbd01aab40c" exitCode=0
Sep 17 17:36:39.116529 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:39.116308    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6"
Sep 17 17:36:39.116529 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:39.116314    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6" event={"ID":"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd","Type":"ContainerDied","Data":"65fd760d8aa9c63ae8342c22a375da229acfe07e73f172e6bbe45cbd01aab40c"}
Sep 17 17:36:39.116529 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:39.116429    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6" event={"ID":"8eadd7e2-bbf7-462e-b2aa-27bec9c97afd","Type":"ContainerDied","Data":"33de892c66ead42c3a6500c6cd07ccbc2507fec9bc03443005166ad40918f0d0"}
Sep 17 17:36:39.116529 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:39.116472    2566 scope.go:117] "RemoveContainer" containerID="65fd760d8aa9c63ae8342c22a375da229acfe07e73f172e6bbe45cbd01aab40c"
Sep 17 17:36:39.126779 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:39.126759    2566 scope.go:117] "RemoveContainer" containerID="97b241a4b7fa653c98764848cd716b8d3977ef2fcd2f860a04d43ba330ace0c8"
Sep 17 17:36:39.137899 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:39.137870    2566 scope.go:117] "RemoveContainer" containerID="65fd760d8aa9c63ae8342c22a375da229acfe07e73f172e6bbe45cbd01aab40c"
Sep 17 17:36:39.138216 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:36:39.138195    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65fd760d8aa9c63ae8342c22a375da229acfe07e73f172e6bbe45cbd01aab40c\": container with ID starting with 65fd760d8aa9c63ae8342c22a375da229acfe07e73f172e6bbe45cbd01aab40c not found: ID does not exist" containerID="65fd760d8aa9c63ae8342c22a375da229acfe07e73f172e6bbe45cbd01aab40c"
Sep 17 17:36:39.138313 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:39.138224    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65fd760d8aa9c63ae8342c22a375da229acfe07e73f172e6bbe45cbd01aab40c"} err="failed to get container status \"65fd760d8aa9c63ae8342c22a375da229acfe07e73f172e6bbe45cbd01aab40c\": rpc error: code = NotFound desc = could not find container \"65fd760d8aa9c63ae8342c22a375da229acfe07e73f172e6bbe45cbd01aab40c\": container with ID starting with 65fd760d8aa9c63ae8342c22a375da229acfe07e73f172e6bbe45cbd01aab40c not found: ID does not exist"
Sep 17 17:36:39.138313 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:39.138247    2566 scope.go:117] "RemoveContainer" containerID="97b241a4b7fa653c98764848cd716b8d3977ef2fcd2f860a04d43ba330ace0c8"
Sep 17 17:36:39.138549 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:36:39.138529    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97b241a4b7fa653c98764848cd716b8d3977ef2fcd2f860a04d43ba330ace0c8\": container with ID starting with 97b241a4b7fa653c98764848cd716b8d3977ef2fcd2f860a04d43ba330ace0c8 not found: ID does not exist" containerID="97b241a4b7fa653c98764848cd716b8d3977ef2fcd2f860a04d43ba330ace0c8"
Sep 17 17:36:39.138604 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:39.138558    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97b241a4b7fa653c98764848cd716b8d3977ef2fcd2f860a04d43ba330ace0c8"} err="failed to get container status \"97b241a4b7fa653c98764848cd716b8d3977ef2fcd2f860a04d43ba330ace0c8\": rpc error: code = NotFound desc = could not find container \"97b241a4b7fa653c98764848cd716b8d3977ef2fcd2f860a04d43ba330ace0c8\": container with ID starting with 97b241a4b7fa653c98764848cd716b8d3977ef2fcd2f860a04d43ba330ace0c8 not found: ID does not exist"
Sep 17 17:36:39.155727 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:39.155687    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6"]
Sep 17 17:36:39.170456 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:39.170401    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-55fd574559-jqsb6"]
Sep 17 17:36:40.346606 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:40.346571    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8eadd7e2-bbf7-462e-b2aa-27bec9c97afd" path="/var/lib/kubelet/pods/8eadd7e2-bbf7-462e-b2aa-27bec9c97afd/volumes"
Sep 17 17:36:49.976843 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:49.976805    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq_4ad0cf79-454b-4234-8c0e-a529ee43eaaf/storage-initializer/0.log"
Sep 17 17:36:49.994047 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:49.994011    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq_4ad0cf79-454b-4234-8c0e-a529ee43eaaf/main/0.log"
Sep 17 17:36:51.430485 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:51.430390    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq"]
Sep 17 17:36:56.597341 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:56.597314    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gblhr_91477834-6648-4e75-8530-e01bdee5e57e/ovn-acl-logging/0.log"
Sep 17 17:36:56.620121 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:36:56.620094    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gblhr_91477834-6648-4e75-8530-e01bdee5e57e/ovn-acl-logging/0.log"
Sep 17 17:37:06.454018 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:06.453890    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq" podUID="4ad0cf79-454b-4234-8c0e-a529ee43eaaf" containerName="main" containerID="cri-o://3c310347031183efb6b16b987be99e6527a7001479616517a4ce3d2c67f21254" gracePeriod=45
Sep 17 17:37:06.720081 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:06.719973    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq" podUID="4ad0cf79-454b-4234-8c0e-a529ee43eaaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 17 17:37:07.113393 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:07.113364    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq"
Sep 17 17:37:07.242523 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:07.242401    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-dshm\") pod \"4ad0cf79-454b-4234-8c0e-a529ee43eaaf\" (UID: \"4ad0cf79-454b-4234-8c0e-a529ee43eaaf\") "
Sep 17 17:37:07.242523 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:07.242458    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-cabundle-cert\") pod \"4ad0cf79-454b-4234-8c0e-a529ee43eaaf\" (UID: \"4ad0cf79-454b-4234-8c0e-a529ee43eaaf\") "
Sep 17 17:37:07.242523 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:07.242514    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-tmp-dir\") pod \"4ad0cf79-454b-4234-8c0e-a529ee43eaaf\" (UID: \"4ad0cf79-454b-4234-8c0e-a529ee43eaaf\") "
Sep 17 17:37:07.242823 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:07.242553    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-model-cache\") pod \"4ad0cf79-454b-4234-8c0e-a529ee43eaaf\" (UID: \"4ad0cf79-454b-4234-8c0e-a529ee43eaaf\") "
Sep 17 17:37:07.242823 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:07.242582    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-home\") pod \"4ad0cf79-454b-4234-8c0e-a529ee43eaaf\" (UID: \"4ad0cf79-454b-4234-8c0e-a529ee43eaaf\") "
Sep 17 17:37:07.242823 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:07.242610    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-kserve-provision-location\") pod \"4ad0cf79-454b-4234-8c0e-a529ee43eaaf\" (UID: \"4ad0cf79-454b-4234-8c0e-a529ee43eaaf\") "
Sep 17 17:37:07.242823 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:07.242792    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5r8g\" (UniqueName: \"kubernetes.io/projected/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-kube-api-access-m5r8g\") pod \"4ad0cf79-454b-4234-8c0e-a529ee43eaaf\" (UID: \"4ad0cf79-454b-4234-8c0e-a529ee43eaaf\") "
Sep 17 17:37:07.243051 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:07.242843    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-tls-certs\") pod \"4ad0cf79-454b-4234-8c0e-a529ee43eaaf\" (UID: \"4ad0cf79-454b-4234-8c0e-a529ee43eaaf\") "
Sep 17 17:37:07.243051 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:07.242870    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-model-cache" (OuterVolumeSpecName: "model-cache") pod "4ad0cf79-454b-4234-8c0e-a529ee43eaaf" (UID: "4ad0cf79-454b-4234-8c0e-a529ee43eaaf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:37:07.243169 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:07.243056    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4ad0cf79-454b-4234-8c0e-a529ee43eaaf" (UID: "4ad0cf79-454b-4234-8c0e-a529ee43eaaf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:37:07.243169 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:07.243087    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-home" (OuterVolumeSpecName: "home") pod "4ad0cf79-454b-4234-8c0e-a529ee43eaaf" (UID: "4ad0cf79-454b-4234-8c0e-a529ee43eaaf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:37:07.243169 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:07.243119    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-model-cache\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:37:07.245394 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:07.245365    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-dshm" (OuterVolumeSpecName: "dshm") pod "4ad0cf79-454b-4234-8c0e-a529ee43eaaf" (UID: "4ad0cf79-454b-4234-8c0e-a529ee43eaaf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:37:07.245730 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:07.245703    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4ad0cf79-454b-4234-8c0e-a529ee43eaaf" (UID: "4ad0cf79-454b-4234-8c0e-a529ee43eaaf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:37:07.245856 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:07.245741    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-kube-api-access-m5r8g" (OuterVolumeSpecName: "kube-api-access-m5r8g") pod "4ad0cf79-454b-4234-8c0e-a529ee43eaaf" (UID: "4ad0cf79-454b-4234-8c0e-a529ee43eaaf"). InnerVolumeSpecName "kube-api-access-m5r8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:37:07.258530 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:07.258495    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4ad0cf79-454b-4234-8c0e-a529ee43eaaf" (UID: "4ad0cf79-454b-4234-8c0e-a529ee43eaaf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:37:07.272665 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:07.272634    2566 generic.go:356] "Generic (PLEG): container finished" podID="4ad0cf79-454b-4234-8c0e-a529ee43eaaf" containerID="3c310347031183efb6b16b987be99e6527a7001479616517a4ce3d2c67f21254" exitCode=0
Sep 17 17:37:07.272830 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:07.272697    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq"
Sep 17 17:37:07.272830 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:07.272714    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq" event={"ID":"4ad0cf79-454b-4234-8c0e-a529ee43eaaf","Type":"ContainerDied","Data":"3c310347031183efb6b16b987be99e6527a7001479616517a4ce3d2c67f21254"}
Sep 17 17:37:07.272830 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:07.272786    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq" event={"ID":"4ad0cf79-454b-4234-8c0e-a529ee43eaaf","Type":"ContainerDied","Data":"cd2bc8b441a90e204cbec4febc559d4fc16f75300e81139e7a9f5f4e4ccf0341"}
Sep 17 17:37:07.272830 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:07.272819    2566 scope.go:117] "RemoveContainer" containerID="3c310347031183efb6b16b987be99e6527a7001479616517a4ce3d2c67f21254"
Sep 17 17:37:07.274719 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:07.274693    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4ad0cf79-454b-4234-8c0e-a529ee43eaaf" (UID: "4ad0cf79-454b-4234-8c0e-a529ee43eaaf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:37:07.284015 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:07.283990    2566 scope.go:117] "RemoveContainer" containerID="3b1a962cc95f6aec10008f445ac8e4ccfc19f9772f7ad89ed37a978a92a172e6"
Sep 17 17:37:07.293463 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:07.293416    2566 scope.go:117] "RemoveContainer" containerID="3c310347031183efb6b16b987be99e6527a7001479616517a4ce3d2c67f21254"
Sep 17 17:37:07.293796 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:37:07.293777    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c310347031183efb6b16b987be99e6527a7001479616517a4ce3d2c67f21254\": container with ID starting with 3c310347031183efb6b16b987be99e6527a7001479616517a4ce3d2c67f21254 not found: ID does not exist" containerID="3c310347031183efb6b16b987be99e6527a7001479616517a4ce3d2c67f21254"
Sep 17 17:37:07.293862 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:07.293807    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c310347031183efb6b16b987be99e6527a7001479616517a4ce3d2c67f21254"} err="failed to get container status \"3c310347031183efb6b16b987be99e6527a7001479616517a4ce3d2c67f21254\": rpc error: code = NotFound desc = could not find container \"3c310347031183efb6b16b987be99e6527a7001479616517a4ce3d2c67f21254\": container with ID starting with 3c310347031183efb6b16b987be99e6527a7001479616517a4ce3d2c67f21254 not found: ID does not exist"
Sep 17 17:37:07.293862 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:07.293825    2566 scope.go:117] "RemoveContainer" containerID="3b1a962cc95f6aec10008f445ac8e4ccfc19f9772f7ad89ed37a978a92a172e6"
Sep 17 17:37:07.294078 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:37:07.294065    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b1a962cc95f6aec10008f445ac8e4ccfc19f9772f7ad89ed37a978a92a172e6\": container with ID starting with 3b1a962cc95f6aec10008f445ac8e4ccfc19f9772f7ad89ed37a978a92a172e6 not found: ID does not exist" containerID="3b1a962cc95f6aec10008f445ac8e4ccfc19f9772f7ad89ed37a978a92a172e6"
Sep 17 17:37:07.294123 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:07.294080    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b1a962cc95f6aec10008f445ac8e4ccfc19f9772f7ad89ed37a978a92a172e6"} err="failed to get container status \"3b1a962cc95f6aec10008f445ac8e4ccfc19f9772f7ad89ed37a978a92a172e6\": rpc error: code = NotFound desc = could not find container \"3b1a962cc95f6aec10008f445ac8e4ccfc19f9772f7ad89ed37a978a92a172e6\": container with ID starting with 3b1a962cc95f6aec10008f445ac8e4ccfc19f9772f7ad89ed37a978a92a172e6 not found: ID does not exist"
Sep 17 17:37:07.344298 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:07.344260    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-dshm\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:37:07.344298 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:07.344291    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-cabundle-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:37:07.344298 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:07.344301    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-tmp-dir\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:37:07.344564 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:07.344309    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-home\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:37:07.344564 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:07.344319    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-kserve-provision-location\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:37:07.344564 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:07.344329    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5r8g\" (UniqueName: \"kubernetes.io/projected/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-kube-api-access-m5r8g\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:37:07.344564 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:07.344337    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ad0cf79-454b-4234-8c0e-a529ee43eaaf-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:37:07.598645 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:07.598611    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq"]
Sep 17 17:37:07.606931 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:07.606902    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-79ff4bf5b9-8gvpq"]
Sep 17 17:37:08.346926 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:08.346891    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ad0cf79-454b-4234-8c0e-a529ee43eaaf" path="/var/lib/kubelet/pods/4ad0cf79-454b-4234-8c0e-a529ee43eaaf/volumes"
Sep 17 17:37:15.006652 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:15.006616    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-vwf7h"]
Sep 17 17:37:15.007039 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:15.006948    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ad0cf79-454b-4234-8c0e-a529ee43eaaf" containerName="storage-initializer"
Sep 17 17:37:15.007039 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:15.006960    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ad0cf79-454b-4234-8c0e-a529ee43eaaf" containerName="storage-initializer"
Sep 17 17:37:15.007039 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:15.006968    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ad0cf79-454b-4234-8c0e-a529ee43eaaf" containerName="main"
Sep 17 17:37:15.007039 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:15.006973    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ad0cf79-454b-4234-8c0e-a529ee43eaaf" containerName="main"
Sep 17 17:37:15.007039 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:15.006983    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8eadd7e2-bbf7-462e-b2aa-27bec9c97afd" containerName="main"
Sep 17 17:37:15.007039 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:15.006988    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8eadd7e2-bbf7-462e-b2aa-27bec9c97afd" containerName="main"
Sep 17 17:37:15.007039 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:15.006996    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8eadd7e2-bbf7-462e-b2aa-27bec9c97afd" containerName="storage-initializer"
Sep 17 17:37:15.007039 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:15.007001    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8eadd7e2-bbf7-462e-b2aa-27bec9c97afd" containerName="storage-initializer"
Sep 17 17:37:15.007292 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:15.007051    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ad0cf79-454b-4234-8c0e-a529ee43eaaf" containerName="main"
Sep 17 17:37:15.007292 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:15.007060    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8eadd7e2-bbf7-462e-b2aa-27bec9c97afd" containerName="main"
Sep 17 17:37:15.011652 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:15.011632    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-vwf7h"
Sep 17 17:37:15.014209 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:15.014185    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Sep 17 17:37:15.014878 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:15.014857    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-p624b\""
Sep 17 17:37:15.014878 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:15.014875    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Sep 17 17:37:15.025207 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:15.025178    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-vwf7h"]
Sep 17 17:37:15.025891 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:15.025877    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Sep 17 17:37:15.202795 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:15.202750    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j4zx\" (UniqueName: \"kubernetes.io/projected/daacec1d-4a81-4b3c-8414-b3a53b1264cb-kube-api-access-2j4zx\") pod \"e2e-pvc-lora-download-vwf7h\" (UID: \"daacec1d-4a81-4b3c-8414-b3a53b1264cb\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-vwf7h"
Sep 17 17:37:15.303617 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:15.303579    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j4zx\" (UniqueName: \"kubernetes.io/projected/daacec1d-4a81-4b3c-8414-b3a53b1264cb-kube-api-access-2j4zx\") pod \"e2e-pvc-lora-download-vwf7h\" (UID: \"daacec1d-4a81-4b3c-8414-b3a53b1264cb\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-vwf7h"
Sep 17 17:37:15.314883 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:15.314847    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j4zx\" (UniqueName: \"kubernetes.io/projected/daacec1d-4a81-4b3c-8414-b3a53b1264cb-kube-api-access-2j4zx\") pod \"e2e-pvc-lora-download-vwf7h\" (UID: \"daacec1d-4a81-4b3c-8414-b3a53b1264cb\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-vwf7h"
Sep 17 17:37:17.593512 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:17.593456    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"]
Sep 17 17:37:17.593969 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:17.593742    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9" podUID="c88771f6-48ea-4b56-a47c-fd7b29bfc918" containerName="istio-proxy" containerID="cri-o://02d3b9395251b4565f3849f61a75b38ba0dcc4ae423263a81dd57d8d18974b3a" gracePeriod=30
Sep 17 17:37:20.659352 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:20.659313    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-e1a7e358-29a8-4ae3-8862-a1adb249ecd3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0024c64a9d90292eb\") pod \"e2e-pvc-lora-download-vwf7h\" (UID: \"daacec1d-4a81-4b3c-8414-b3a53b1264cb\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-vwf7h"
Sep 17 17:37:20.673787 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:20.673754    2566 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-e1a7e358-29a8-4ae3-8862-a1adb249ecd3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0024c64a9d90292eb\") pod \"e2e-pvc-lora-download-vwf7h\" (UID: \"daacec1d-4a81-4b3c-8414-b3a53b1264cb\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-vwf7h"
Sep 17 17:37:20.759960 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:20.759916    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-e1a7e358-29a8-4ae3-8862-a1adb249ecd3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0024c64a9d90292eb\") pod \"e2e-pvc-lora-download-vwf7h\" (UID: \"daacec1d-4a81-4b3c-8414-b3a53b1264cb\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-vwf7h"
Sep 17 17:37:20.760154 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:20.760044    2566 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-e1a7e358-29a8-4ae3-8862-a1adb249ecd3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0024c64a9d90292eb\") pod \"e2e-pvc-lora-download-vwf7h\" (UID: \"daacec1d-4a81-4b3c-8414-b3a53b1264cb\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-vwf7h"
Sep 17 17:37:20.763860 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:20.763836    2566 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-e1a7e358-29a8-4ae3-8862-a1adb249ecd3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0024c64a9d90292eb\") pod \"e2e-pvc-lora-download-vwf7h\" (UID: \"daacec1d-4a81-4b3c-8414-b3a53b1264cb\") DevicePath \"csi-fc8ae150a79d7da6e19bc357e70065ddda4dcc641a905ccb9a27942b54e570c7\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-vwf7h"
Sep 17 17:37:21.001813 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:21.001778    2566 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-e1a7e358-29a8-4ae3-8862-a1adb249ecd3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0024c64a9d90292eb\") pod \"e2e-pvc-lora-download-vwf7h\" (UID: \"daacec1d-4a81-4b3c-8414-b3a53b1264cb\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/1538cd1e69012c02b995a1a7e916a3b0f4806d6e9ac041bcec0a276392f2fd5c/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-vwf7h"
Sep 17 17:37:21.017997 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:21.017967    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-e1a7e358-29a8-4ae3-8862-a1adb249ecd3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0024c64a9d90292eb\") pod \"e2e-pvc-lora-download-vwf7h\" (UID: \"daacec1d-4a81-4b3c-8414-b3a53b1264cb\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-vwf7h"
Sep 17 17:37:21.320587 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:21.320546    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-vwf7h"
Sep 17 17:37:21.602454 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:21.602398    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-vwf7h"]
Sep 17 17:37:21.604715 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:37:21.604682    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddaacec1d_4a81_4b3c_8414_b3a53b1264cb.slice/crio-846cb261e362e79d25531a288ae5e7b1a2da3c2bb47d48962ab8b553e949b20b WatchSource:0}: Error finding container 846cb261e362e79d25531a288ae5e7b1a2da3c2bb47d48962ab8b553e949b20b: Status 404 returned error can't find the container with id 846cb261e362e79d25531a288ae5e7b1a2da3c2bb47d48962ab8b553e949b20b
Sep 17 17:37:21.966264 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:21.966170    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9" podUID="c88771f6-48ea-4b56-a47c-fd7b29bfc918" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:37:22.354068 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:22.354030    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-vwf7h" event={"ID":"daacec1d-4a81-4b3c-8414-b3a53b1264cb","Type":"ContainerStarted","Data":"837ea866b5aa8732866a0f20a7d7a317cf38e2c59a23cd41e301d9732aa3ee95"}
Sep 17 17:37:22.354068 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:22.354076    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-vwf7h" event={"ID":"daacec1d-4a81-4b3c-8414-b3a53b1264cb","Type":"ContainerStarted","Data":"846cb261e362e79d25531a288ae5e7b1a2da3c2bb47d48962ab8b553e949b20b"}
Sep 17 17:37:22.373255 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:22.373193    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-vwf7h" podStartSLOduration=11.373177225 podStartE2EDuration="11.373177225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:22.371214745 +0000 UTC m=+3328.265693887" watchObservedRunningTime="2026-09-17 17:37:22.373177225 +0000 UTC m=+3328.267656363"
Sep 17 17:37:22.873384 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:22.873308    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"
Sep 17 17:37:22.978682 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:22.978605    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7frl7\" (UniqueName: \"kubernetes.io/projected/c88771f6-48ea-4b56-a47c-fd7b29bfc918-kube-api-access-7frl7\") pod \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\" (UID: \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\") "
Sep 17 17:37:22.978682 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:22.978673    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c88771f6-48ea-4b56-a47c-fd7b29bfc918-istio-data\") pod \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\" (UID: \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\") "
Sep 17 17:37:22.978682 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:22.978698    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c88771f6-48ea-4b56-a47c-fd7b29bfc918-istio-envoy\") pod \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\" (UID: \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\") "
Sep 17 17:37:22.979276 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:22.978742    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c88771f6-48ea-4b56-a47c-fd7b29bfc918-workload-certs\") pod \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\" (UID: \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\") "
Sep 17 17:37:22.979276 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:22.978775    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c88771f6-48ea-4b56-a47c-fd7b29bfc918-istiod-ca-cert\") pod \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\" (UID: \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\") "
Sep 17 17:37:22.979276 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:22.978831    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c88771f6-48ea-4b56-a47c-fd7b29bfc918-istio-podinfo\") pod \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\" (UID: \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\") "
Sep 17 17:37:22.979276 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:22.978861    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88771f6-48ea-4b56-a47c-fd7b29bfc918-credential-socket\") pod \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\" (UID: \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\") "
Sep 17 17:37:22.979276 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:22.978887    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88771f6-48ea-4b56-a47c-fd7b29bfc918-workload-socket\") pod \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\" (UID: \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\") "
Sep 17 17:37:22.979276 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:22.978915    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c88771f6-48ea-4b56-a47c-fd7b29bfc918-istio-token\") pod \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\" (UID: \"c88771f6-48ea-4b56-a47c-fd7b29bfc918\") "
Sep 17 17:37:22.979646 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:22.979462    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c88771f6-48ea-4b56-a47c-fd7b29bfc918-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c88771f6-48ea-4b56-a47c-fd7b29bfc918" (UID: "c88771f6-48ea-4b56-a47c-fd7b29bfc918"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:37:22.979705 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:22.979676    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c88771f6-48ea-4b56-a47c-fd7b29bfc918-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c88771f6-48ea-4b56-a47c-fd7b29bfc918" (UID: "c88771f6-48ea-4b56-a47c-fd7b29bfc918"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:37:22.979781 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:22.979749    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c88771f6-48ea-4b56-a47c-fd7b29bfc918-istio-data" (OuterVolumeSpecName: "istio-data") pod "c88771f6-48ea-4b56-a47c-fd7b29bfc918" (UID: "c88771f6-48ea-4b56-a47c-fd7b29bfc918"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:37:22.980279 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:22.980238    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c88771f6-48ea-4b56-a47c-fd7b29bfc918-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c88771f6-48ea-4b56-a47c-fd7b29bfc918" (UID: "c88771f6-48ea-4b56-a47c-fd7b29bfc918"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:37:22.980405 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:22.980294    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c88771f6-48ea-4b56-a47c-fd7b29bfc918-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c88771f6-48ea-4b56-a47c-fd7b29bfc918" (UID: "c88771f6-48ea-4b56-a47c-fd7b29bfc918"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:37:22.982402 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:22.982375    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c88771f6-48ea-4b56-a47c-fd7b29bfc918-kube-api-access-7frl7" (OuterVolumeSpecName: "kube-api-access-7frl7") pod "c88771f6-48ea-4b56-a47c-fd7b29bfc918" (UID: "c88771f6-48ea-4b56-a47c-fd7b29bfc918"). InnerVolumeSpecName "kube-api-access-7frl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:37:22.983119 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:22.983089    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c88771f6-48ea-4b56-a47c-fd7b29bfc918-istio-token" (OuterVolumeSpecName: "istio-token") pod "c88771f6-48ea-4b56-a47c-fd7b29bfc918" (UID: "c88771f6-48ea-4b56-a47c-fd7b29bfc918"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:37:22.983338 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:22.983315    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c88771f6-48ea-4b56-a47c-fd7b29bfc918-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c88771f6-48ea-4b56-a47c-fd7b29bfc918" (UID: "c88771f6-48ea-4b56-a47c-fd7b29bfc918"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 17:37:22.983429 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:22.983357    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c88771f6-48ea-4b56-a47c-fd7b29bfc918-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c88771f6-48ea-4b56-a47c-fd7b29bfc918" (UID: "c88771f6-48ea-4b56-a47c-fd7b29bfc918"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:37:23.080561 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:23.080521    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c88771f6-48ea-4b56-a47c-fd7b29bfc918-istiod-ca-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:37:23.080561 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:23.080556    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c88771f6-48ea-4b56-a47c-fd7b29bfc918-istio-podinfo\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:37:23.080561 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:23.080565    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88771f6-48ea-4b56-a47c-fd7b29bfc918-credential-socket\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:37:23.080837 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:23.080584    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88771f6-48ea-4b56-a47c-fd7b29bfc918-workload-socket\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:37:23.080837 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:23.080610    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c88771f6-48ea-4b56-a47c-fd7b29bfc918-istio-token\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:37:23.080837 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:23.080624    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7frl7\" (UniqueName: \"kubernetes.io/projected/c88771f6-48ea-4b56-a47c-fd7b29bfc918-kube-api-access-7frl7\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:37:23.080837 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:23.080636    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c88771f6-48ea-4b56-a47c-fd7b29bfc918-istio-data\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:37:23.080837 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:23.080650    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c88771f6-48ea-4b56-a47c-fd7b29bfc918-istio-envoy\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:37:23.080837 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:23.080667    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c88771f6-48ea-4b56-a47c-fd7b29bfc918-workload-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:37:23.361395 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:23.361353    2566 generic.go:356] "Generic (PLEG): container finished" podID="c88771f6-48ea-4b56-a47c-fd7b29bfc918" containerID="02d3b9395251b4565f3849f61a75b38ba0dcc4ae423263a81dd57d8d18974b3a" exitCode=0
Sep 17 17:37:23.361618 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:23.361414    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"
Sep 17 17:37:23.361618 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:23.361467    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9" event={"ID":"c88771f6-48ea-4b56-a47c-fd7b29bfc918","Type":"ContainerDied","Data":"02d3b9395251b4565f3849f61a75b38ba0dcc4ae423263a81dd57d8d18974b3a"}
Sep 17 17:37:23.361618 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:23.361526    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9" event={"ID":"c88771f6-48ea-4b56-a47c-fd7b29bfc918","Type":"ContainerDied","Data":"e06f96b6dd184498d9c86d6cd7096002caddc9edd4fd8a48f7ffb490887738b3"}
Sep 17 17:37:23.361618 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:23.361557    2566 scope.go:117] "RemoveContainer" containerID="02d3b9395251b4565f3849f61a75b38ba0dcc4ae423263a81dd57d8d18974b3a"
Sep 17 17:37:23.371893 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:23.371870    2566 scope.go:117] "RemoveContainer" containerID="51731d91c2660c23bc6b7385ff84681d13d9851a55d96858dac739147f16c272"
Sep 17 17:37:23.380601 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:23.380577    2566 scope.go:117] "RemoveContainer" containerID="02d3b9395251b4565f3849f61a75b38ba0dcc4ae423263a81dd57d8d18974b3a"
Sep 17 17:37:23.380916 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:37:23.380895    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02d3b9395251b4565f3849f61a75b38ba0dcc4ae423263a81dd57d8d18974b3a\": container with ID starting with 02d3b9395251b4565f3849f61a75b38ba0dcc4ae423263a81dd57d8d18974b3a not found: ID does not exist" containerID="02d3b9395251b4565f3849f61a75b38ba0dcc4ae423263a81dd57d8d18974b3a"
Sep 17 17:37:23.380964 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:23.380929    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02d3b9395251b4565f3849f61a75b38ba0dcc4ae423263a81dd57d8d18974b3a"} err="failed to get container status \"02d3b9395251b4565f3849f61a75b38ba0dcc4ae423263a81dd57d8d18974b3a\": rpc error: code = NotFound desc = could not find container \"02d3b9395251b4565f3849f61a75b38ba0dcc4ae423263a81dd57d8d18974b3a\": container with ID starting with 02d3b9395251b4565f3849f61a75b38ba0dcc4ae423263a81dd57d8d18974b3a not found: ID does not exist"
Sep 17 17:37:23.380964 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:23.380949    2566 scope.go:117] "RemoveContainer" containerID="51731d91c2660c23bc6b7385ff84681d13d9851a55d96858dac739147f16c272"
Sep 17 17:37:23.381196 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:37:23.381181    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51731d91c2660c23bc6b7385ff84681d13d9851a55d96858dac739147f16c272\": container with ID starting with 51731d91c2660c23bc6b7385ff84681d13d9851a55d96858dac739147f16c272 not found: ID does not exist" containerID="51731d91c2660c23bc6b7385ff84681d13d9851a55d96858dac739147f16c272"
Sep 17 17:37:23.381237 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:23.381201    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51731d91c2660c23bc6b7385ff84681d13d9851a55d96858dac739147f16c272"} err="failed to get container status \"51731d91c2660c23bc6b7385ff84681d13d9851a55d96858dac739147f16c272\": rpc error: code = NotFound desc = could not find container \"51731d91c2660c23bc6b7385ff84681d13d9851a55d96858dac739147f16c272\": container with ID starting with 51731d91c2660c23bc6b7385ff84681d13d9851a55d96858dac739147f16c272 not found: ID does not exist"
Sep 17 17:37:23.390369 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:23.390341    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"]
Sep 17 17:37:23.394522 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:23.394497    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-5s9b9"]
Sep 17 17:37:24.346949 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:24.346913    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c88771f6-48ea-4b56-a47c-fd7b29bfc918" path="/var/lib/kubelet/pods/c88771f6-48ea-4b56-a47c-fd7b29bfc918/volumes"
Sep 17 17:37:24.368980 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:24.368944    2566 generic.go:356] "Generic (PLEG): container finished" podID="daacec1d-4a81-4b3c-8414-b3a53b1264cb" containerID="837ea866b5aa8732866a0f20a7d7a317cf38e2c59a23cd41e301d9732aa3ee95" exitCode=0
Sep 17 17:37:24.369150 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:24.369017    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-vwf7h" event={"ID":"daacec1d-4a81-4b3c-8414-b3a53b1264cb","Type":"ContainerDied","Data":"837ea866b5aa8732866a0f20a7d7a317cf38e2c59a23cd41e301d9732aa3ee95"}
Sep 17 17:37:25.532076 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:25.532048    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-vwf7h"
Sep 17 17:37:25.702324 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:25.702235    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2j4zx\" (UniqueName: \"kubernetes.io/projected/daacec1d-4a81-4b3c-8414-b3a53b1264cb-kube-api-access-2j4zx\") pod \"daacec1d-4a81-4b3c-8414-b3a53b1264cb\" (UID: \"daacec1d-4a81-4b3c-8414-b3a53b1264cb\") "
Sep 17 17:37:25.702493 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:25.702414    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0024c64a9d90292eb\") pod \"daacec1d-4a81-4b3c-8414-b3a53b1264cb\" (UID: \"daacec1d-4a81-4b3c-8414-b3a53b1264cb\") "
Sep 17 17:37:25.704812 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:25.704785    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daacec1d-4a81-4b3c-8414-b3a53b1264cb-kube-api-access-2j4zx" (OuterVolumeSpecName: "kube-api-access-2j4zx") pod "daacec1d-4a81-4b3c-8414-b3a53b1264cb" (UID: "daacec1d-4a81-4b3c-8414-b3a53b1264cb"). InnerVolumeSpecName "kube-api-access-2j4zx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:37:25.706866 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:25.706844    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0024c64a9d90292eb" (OuterVolumeSpecName: "model-storage") pod "daacec1d-4a81-4b3c-8414-b3a53b1264cb" (UID: "daacec1d-4a81-4b3c-8414-b3a53b1264cb"). InnerVolumeSpecName "pvc-e1a7e358-29a8-4ae3-8862-a1adb249ecd3". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 17 17:37:25.803521 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:25.803480    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j4zx\" (UniqueName: \"kubernetes.io/projected/daacec1d-4a81-4b3c-8414-b3a53b1264cb-kube-api-access-2j4zx\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:37:25.803729 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:25.803542    2566 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-e1a7e358-29a8-4ae3-8862-a1adb249ecd3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0024c64a9d90292eb\") on node \"ip-10-0-141-66.ec2.internal\" "
Sep 17 17:37:26.384715 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:26.384686    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-vwf7h"
Sep 17 17:37:26.384885 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:26.384682    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-vwf7h" event={"ID":"daacec1d-4a81-4b3c-8414-b3a53b1264cb","Type":"ContainerDied","Data":"846cb261e362e79d25531a288ae5e7b1a2da3c2bb47d48962ab8b553e949b20b"}
Sep 17 17:37:26.384885 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:26.384801    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="846cb261e362e79d25531a288ae5e7b1a2da3c2bb47d48962ab8b553e949b20b"
Sep 17 17:37:26.777507 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:26.777482    2566 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-e1a7e358-29a8-4ae3-8862-a1adb249ecd3" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0024c64a9d90292eb") on node "ip-10-0-141-66.ec2.internal"
Sep 17 17:37:26.812490 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:37:26.812420    2566 reconciler_common.go:299] "Volume detached for volume \"pvc-e1a7e358-29a8-4ae3-8862-a1adb249ecd3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0024c64a9d90292eb\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"csi-fc8ae150a79d7da6e19bc357e70065ddda4dcc641a905ccb9a27942b54e570c7\""
Sep 17 17:38:03.993045 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:03.993010    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"]
Sep 17 17:38:03.993508 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:03.993336    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c88771f6-48ea-4b56-a47c-fd7b29bfc918" containerName="istio-proxy"
Sep 17 17:38:03.993508 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:03.993348    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c88771f6-48ea-4b56-a47c-fd7b29bfc918" containerName="istio-proxy"
Sep 17 17:38:03.993508 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:03.993365    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daacec1d-4a81-4b3c-8414-b3a53b1264cb" containerName="storage-initializer"
Sep 17 17:38:03.993508 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:03.993371    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="daacec1d-4a81-4b3c-8414-b3a53b1264cb" containerName="storage-initializer"
Sep 17 17:38:03.993508 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:03.993380    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c88771f6-48ea-4b56-a47c-fd7b29bfc918" containerName="istio-proxy"
Sep 17 17:38:03.993508 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:03.993385    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c88771f6-48ea-4b56-a47c-fd7b29bfc918" containerName="istio-proxy"
Sep 17 17:38:03.993508 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:03.993461    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c88771f6-48ea-4b56-a47c-fd7b29bfc918" containerName="istio-proxy"
Sep 17 17:38:03.993508 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:03.993468    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="daacec1d-4a81-4b3c-8414-b3a53b1264cb" containerName="storage-initializer"
Sep 17 17:38:03.993799 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:03.993562    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c88771f6-48ea-4b56-a47c-fd7b29bfc918" containerName="istio-proxy"
Sep 17 17:38:03.997737 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:03.997711    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"
Sep 17 17:38:04.000001 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:03.999968    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Sep 17 17:38:04.000152 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:04.000100    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:38:04.000374 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:04.000354    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"lora-pvc-single-adapter-test-kserve-self-signed-certs\""
Sep 17 17:38:04.000503 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:04.000480    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Sep 17 17:38:04.000882 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:04.000864    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Sep 17 17:38:04.000976 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:04.000864    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-p624b\""
Sep 17 17:38:04.011173 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:04.011145    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"]
Sep 17 17:38:04.094693 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:04.094649    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq\" (UID: \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"
Sep 17 17:38:04.094693 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:04.094698    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq\" (UID: \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"
Sep 17 17:38:04.094991 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:04.094727    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-dshm\") pod \"lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq\" (UID: \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"
Sep 17 17:38:04.094991 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:04.094787    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq\" (UID: \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"
Sep 17 17:38:04.094991 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:04.094847    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-home\") pod \"lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq\" (UID: \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"
Sep 17 17:38:04.094991 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:04.094901    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq\" (UID: \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"
Sep 17 17:38:04.094991 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:04.094924    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq\" (UID: \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"
Sep 17 17:38:04.195496 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:04.195419    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-dshm\") pod \"lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq\" (UID: \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"
Sep 17 17:38:04.195684 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:04.195524    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq\" (UID: \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"
Sep 17 17:38:04.195684 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:04.195566    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-home\") pod \"lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq\" (UID: \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"
Sep 17 17:38:04.195684 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:04.195598    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xbsv\" (UniqueName: \"kubernetes.io/projected/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-kube-api-access-8xbsv\") pod \"lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq\" (UID: \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"
Sep 17 17:38:04.195684 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:04.195660    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq\" (UID: \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"
Sep 17 17:38:04.195684 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:04.195681    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq\" (UID: \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"
Sep 17 17:38:04.195985 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:04.195789    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq\" (UID: \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"
Sep 17 17:38:04.195985 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:04.195831    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq\" (UID: \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"
Sep 17 17:38:04.196089 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:04.196011    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq\" (UID: \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"
Sep 17 17:38:04.196089 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:04.196045    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-home\") pod \"lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq\" (UID: \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"
Sep 17 17:38:04.196168 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:04.196144    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq\" (UID: \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"
Sep 17 17:38:04.196293 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:04.196274    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq\" (UID: \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"
Sep 17 17:38:04.196796 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:04.196772    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq\" (UID: \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"
Sep 17 17:38:04.198375 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:04.198352    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-dshm\") pod \"lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq\" (UID: \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"
Sep 17 17:38:04.198584 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:04.198563    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq\" (UID: \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"
Sep 17 17:38:04.296949 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:04.296914    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xbsv\" (UniqueName: \"kubernetes.io/projected/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-kube-api-access-8xbsv\") pod \"lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq\" (UID: \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"
Sep 17 17:38:04.309926 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:04.309895    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xbsv\" (UniqueName: \"kubernetes.io/projected/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-kube-api-access-8xbsv\") pod \"lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq\" (UID: \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"
Sep 17 17:38:11.762224 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:11.762179    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-e1a7e358-29a8-4ae3-8862-a1adb249ecd3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0024c64a9d90292eb\") pod \"lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq\" (UID: \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"
Sep 17 17:38:11.768039 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:11.768005    2566 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-e1a7e358-29a8-4ae3-8862-a1adb249ecd3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0024c64a9d90292eb\") pod \"lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq\" (UID: \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"
Sep 17 17:38:11.863294 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:11.863256    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-e1a7e358-29a8-4ae3-8862-a1adb249ecd3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0024c64a9d90292eb\") pod \"lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq\" (UID: \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"
Sep 17 17:38:11.863482 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:11.863370    2566 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-e1a7e358-29a8-4ae3-8862-a1adb249ecd3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0024c64a9d90292eb\") pod \"lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq\" (UID: \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"
Sep 17 17:38:11.866769 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:11.866749    2566 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-e1a7e358-29a8-4ae3-8862-a1adb249ecd3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0024c64a9d90292eb\") pod \"lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq\" (UID: \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\") DevicePath \"csi-fc8ae150a79d7da6e19bc357e70065ddda4dcc641a905ccb9a27942b54e570c7\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"
Sep 17 17:38:11.959594 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:11.959558    2566 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-e1a7e358-29a8-4ae3-8862-a1adb249ecd3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0024c64a9d90292eb\") pod \"lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq\" (UID: \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/1538cd1e69012c02b995a1a7e916a3b0f4806d6e9ac041bcec0a276392f2fd5c/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"
Sep 17 17:38:11.979162 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:11.979123    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-e1a7e358-29a8-4ae3-8862-a1adb249ecd3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0024c64a9d90292eb\") pod \"lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq\" (UID: \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"
Sep 17 17:38:12.109400 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:12.109358    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"
Sep 17 17:38:12.392692 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:12.392663    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"]
Sep 17 17:38:12.394140 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:38:12.394110    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7b2e079_bc7d_44db_b674_9d0e5fa6afe0.slice/crio-2b57dcbb14e893a060488c30c889674c6928c38679e2e2f0abf4f032e9082fe7 WatchSource:0}: Error finding container 2b57dcbb14e893a060488c30c889674c6928c38679e2e2f0abf4f032e9082fe7: Status 404 returned error can't find the container with id 2b57dcbb14e893a060488c30c889674c6928c38679e2e2f0abf4f032e9082fe7
Sep 17 17:38:12.623112 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:12.623072    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq" event={"ID":"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0","Type":"ContainerStarted","Data":"7e1955bf98e26252a787b93a1826fd7014cf3859e438574c10c2f2488b544c4f"}
Sep 17 17:38:12.623320 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:12.623135    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq" event={"ID":"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0","Type":"ContainerStarted","Data":"2b57dcbb14e893a060488c30c889674c6928c38679e2e2f0abf4f032e9082fe7"}
Sep 17 17:38:15.640313 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:15.640276    2566 generic.go:356] "Generic (PLEG): container finished" podID="c7b2e079-bc7d-44db-b674-9d0e5fa6afe0" containerID="7e1955bf98e26252a787b93a1826fd7014cf3859e438574c10c2f2488b544c4f" exitCode=0
Sep 17 17:38:15.640733 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:15.640354    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq" event={"ID":"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0","Type":"ContainerDied","Data":"7e1955bf98e26252a787b93a1826fd7014cf3859e438574c10c2f2488b544c4f"}
Sep 17 17:38:16.647792 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:16.647753    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq" event={"ID":"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0","Type":"ContainerStarted","Data":"961b7d10286f6e7f4814a3cb9c179f518b3d0a618ed3d393984ca5cfd730109c"}
Sep 17 17:38:16.672617 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:16.672541    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq" podStartSLOduration=11.375585011 podStartE2EDuration="13.672519308s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.296934297s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:38:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:38:16.670681543 +0000 UTC m=+3382.565160683" watchObservedRunningTime="2026-09-17 17:38:16.672519308 +0000 UTC m=+3382.566998451"
Sep 17 17:38:17.110354 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:17.110320    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"
Sep 17 17:38:22.109851 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:22.109814    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"
Sep 17 17:38:22.111383 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:22.111352    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq" podUID="c7b2e079-bc7d-44db-b674-9d0e5fa6afe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 17 17:38:32.110053 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:32.110004    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq" podUID="c7b2e079-bc7d-44db-b674-9d0e5fa6afe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 17 17:38:42.110219 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:42.110120    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq" podUID="c7b2e079-bc7d-44db-b674-9d0e5fa6afe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 17 17:38:52.110597 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:38:52.110546    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq" podUID="c7b2e079-bc7d-44db-b674-9d0e5fa6afe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 17 17:39:02.110275 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:02.110223    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq" podUID="c7b2e079-bc7d-44db-b674-9d0e5fa6afe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 17 17:39:12.110745 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:12.110687    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq" podUID="c7b2e079-bc7d-44db-b674-9d0e5fa6afe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 17 17:39:22.110155 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:22.110105    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq" podUID="c7b2e079-bc7d-44db-b674-9d0e5fa6afe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 17 17:39:32.110137 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:32.110081    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq" podUID="c7b2e079-bc7d-44db-b674-9d0e5fa6afe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 17 17:39:42.120249 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:42.120207    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"
Sep 17 17:39:42.128674 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:42.128642    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"
Sep 17 17:39:48.759929 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:48.759884    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen6n7l"]
Sep 17 17:39:48.763684 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:48.763659    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen6n7l"
Sep 17 17:39:48.770118 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:48.770086    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-dw9cb\""
Sep 17 17:39:48.770565 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:48.770117    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-2ld97\""
Sep 17 17:39:48.770750 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:48.770117    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 17 17:39:48.770895 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:48.770170    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 17 17:39:48.771174 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:48.770171    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 17 17:39:48.776536 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:48.776512    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen6n7l"]
Sep 17 17:39:48.782102 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:48.782074    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/263ccc61-917f-4a02-8108-5f9307cfe728-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen6n7l\" (UID: \"263ccc61-917f-4a02-8108-5f9307cfe728\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen6n7l"
Sep 17 17:39:48.782271 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:48.782123    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zv94\" (UniqueName: \"kubernetes.io/projected/263ccc61-917f-4a02-8108-5f9307cfe728-kube-api-access-4zv94\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen6n7l\" (UID: \"263ccc61-917f-4a02-8108-5f9307cfe728\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen6n7l"
Sep 17 17:39:48.869998 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:48.869957    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 17:39:48.873377 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:48.873356    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 17:39:48.876430 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:48.876403    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:39:48.876430 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:48.876422    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 17 17:39:48.876701 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:48.876686    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-pbrnk\""
Sep 17 17:39:48.882958 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:48.882922    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6e3ccc3-30b7-421a-96b3-65dcd3822747-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c6e3ccc3-30b7-421a-96b3-65dcd3822747\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 17:39:48.882958 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:48.882951    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l42c9\" (UniqueName: \"kubernetes.io/projected/c6e3ccc3-30b7-421a-96b3-65dcd3822747-kube-api-access-l42c9\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c6e3ccc3-30b7-421a-96b3-65dcd3822747\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 17:39:48.883121 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:48.882989    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6e3ccc3-30b7-421a-96b3-65dcd3822747-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c6e3ccc3-30b7-421a-96b3-65dcd3822747\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 17:39:48.883121 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:48.883104    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6e3ccc3-30b7-421a-96b3-65dcd3822747-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c6e3ccc3-30b7-421a-96b3-65dcd3822747\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 17:39:48.883239 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:48.883124    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6e3ccc3-30b7-421a-96b3-65dcd3822747-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c6e3ccc3-30b7-421a-96b3-65dcd3822747\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 17:39:48.883239 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:48.883175    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/263ccc61-917f-4a02-8108-5f9307cfe728-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen6n7l\" (UID: \"263ccc61-917f-4a02-8108-5f9307cfe728\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen6n7l"
Sep 17 17:39:48.883239 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:48.883214    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6e3ccc3-30b7-421a-96b3-65dcd3822747-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c6e3ccc3-30b7-421a-96b3-65dcd3822747\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 17:39:48.883385 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:48.883243    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6e3ccc3-30b7-421a-96b3-65dcd3822747-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c6e3ccc3-30b7-421a-96b3-65dcd3822747\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 17:39:48.883385 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:48.883274    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zv94\" (UniqueName: \"kubernetes.io/projected/263ccc61-917f-4a02-8108-5f9307cfe728-kube-api-access-4zv94\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen6n7l\" (UID: \"263ccc61-917f-4a02-8108-5f9307cfe728\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen6n7l"
Sep 17 17:39:48.883385 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:48.883301    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6e3ccc3-30b7-421a-96b3-65dcd3822747-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c6e3ccc3-30b7-421a-96b3-65dcd3822747\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 17:39:48.886055 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:48.886025    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/263ccc61-917f-4a02-8108-5f9307cfe728-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen6n7l\" (UID: \"263ccc61-917f-4a02-8108-5f9307cfe728\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen6n7l"
Sep 17 17:39:48.900311 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:48.900281    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 17:39:48.902177 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:48.902152    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zv94\" (UniqueName: \"kubernetes.io/projected/263ccc61-917f-4a02-8108-5f9307cfe728-kube-api-access-4zv94\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen6n7l\" (UID: \"263ccc61-917f-4a02-8108-5f9307cfe728\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen6n7l"
Sep 17 17:39:48.984586 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:48.984543    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6e3ccc3-30b7-421a-96b3-65dcd3822747-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c6e3ccc3-30b7-421a-96b3-65dcd3822747\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 17:39:48.984586 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:48.984599    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6e3ccc3-30b7-421a-96b3-65dcd3822747-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c6e3ccc3-30b7-421a-96b3-65dcd3822747\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 17:39:48.984878 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:48.984628    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6e3ccc3-30b7-421a-96b3-65dcd3822747-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c6e3ccc3-30b7-421a-96b3-65dcd3822747\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 17:39:48.984878 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:48.984648    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6e3ccc3-30b7-421a-96b3-65dcd3822747-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c6e3ccc3-30b7-421a-96b3-65dcd3822747\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 17:39:48.984878 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:48.984674    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6e3ccc3-30b7-421a-96b3-65dcd3822747-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c6e3ccc3-30b7-421a-96b3-65dcd3822747\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 17:39:48.984878 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:48.984714    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6e3ccc3-30b7-421a-96b3-65dcd3822747-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c6e3ccc3-30b7-421a-96b3-65dcd3822747\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 17:39:48.984878 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:48.984738    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l42c9\" (UniqueName: \"kubernetes.io/projected/c6e3ccc3-30b7-421a-96b3-65dcd3822747-kube-api-access-l42c9\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c6e3ccc3-30b7-421a-96b3-65dcd3822747\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 17:39:48.984878 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:48.984797    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6e3ccc3-30b7-421a-96b3-65dcd3822747-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c6e3ccc3-30b7-421a-96b3-65dcd3822747\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 17:39:48.985182 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:48.985084    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6e3ccc3-30b7-421a-96b3-65dcd3822747-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c6e3ccc3-30b7-421a-96b3-65dcd3822747\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 17:39:48.985182 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:48.985117    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6e3ccc3-30b7-421a-96b3-65dcd3822747-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c6e3ccc3-30b7-421a-96b3-65dcd3822747\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 17:39:48.985254 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:48.985191    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6e3ccc3-30b7-421a-96b3-65dcd3822747-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c6e3ccc3-30b7-421a-96b3-65dcd3822747\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 17:39:48.985300 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:48.985279    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6e3ccc3-30b7-421a-96b3-65dcd3822747-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c6e3ccc3-30b7-421a-96b3-65dcd3822747\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 17:39:48.985358 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:48.985340    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6e3ccc3-30b7-421a-96b3-65dcd3822747-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c6e3ccc3-30b7-421a-96b3-65dcd3822747\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 17:39:48.987833 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:48.987807    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6e3ccc3-30b7-421a-96b3-65dcd3822747-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c6e3ccc3-30b7-421a-96b3-65dcd3822747\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 17:39:48.987956 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:48.987894    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6e3ccc3-30b7-421a-96b3-65dcd3822747-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c6e3ccc3-30b7-421a-96b3-65dcd3822747\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 17:39:49.000359 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:49.000317    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l42c9\" (UniqueName: \"kubernetes.io/projected/c6e3ccc3-30b7-421a-96b3-65dcd3822747-kube-api-access-l42c9\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c6e3ccc3-30b7-421a-96b3-65dcd3822747\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 17:39:49.078322 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:49.078286    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen6n7l"
Sep 17 17:39:49.183557 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:49.183524    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 17:39:49.372008 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:49.371978    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen6n7l"]
Sep 17 17:39:49.373347 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:39:49.373319    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod263ccc61_917f_4a02_8108_5f9307cfe728.slice/crio-32f30f14a6dd11a49d3c7d88cd28dc18f5cc1c3d7d6d373d14f82391e63fca3b WatchSource:0}: Error finding container 32f30f14a6dd11a49d3c7d88cd28dc18f5cc1c3d7d6d373d14f82391e63fca3b: Status 404 returned error can't find the container with id 32f30f14a6dd11a49d3c7d88cd28dc18f5cc1c3d7d6d373d14f82391e63fca3b
Sep 17 17:39:49.475619 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:49.475590    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 17:39:49.477770 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:39:49.477738    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6e3ccc3_30b7_421a_96b3_65dcd3822747.slice/crio-6b3fbcb2bae307f2130f98e36df08b2a4831de2b9649166bc498a504f122a0bb WatchSource:0}: Error finding container 6b3fbcb2bae307f2130f98e36df08b2a4831de2b9649166bc498a504f122a0bb: Status 404 returned error can't find the container with id 6b3fbcb2bae307f2130f98e36df08b2a4831de2b9649166bc498a504f122a0bb
Sep 17 17:39:50.156548 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:50.156505    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"c6e3ccc3-30b7-421a-96b3-65dcd3822747","Type":"ContainerStarted","Data":"c7550015443ba7593d200e65b2c41ba91c1371d3621b39622626904d0ca2f06b"}
Sep 17 17:39:50.157008 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:50.156578    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"c6e3ccc3-30b7-421a-96b3-65dcd3822747","Type":"ContainerStarted","Data":"6b3fbcb2bae307f2130f98e36df08b2a4831de2b9649166bc498a504f122a0bb"}
Sep 17 17:39:50.158236 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:50.158207    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen6n7l" event={"ID":"263ccc61-917f-4a02-8108-5f9307cfe728","Type":"ContainerStarted","Data":"1cd28cf808825b39f012d8923513dd1a6dc58160d490d1ceb5447da06b0ad530"}
Sep 17 17:39:50.158368 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:50.158259    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen6n7l" event={"ID":"263ccc61-917f-4a02-8108-5f9307cfe728","Type":"ContainerStarted","Data":"32f30f14a6dd11a49d3c7d88cd28dc18f5cc1c3d7d6d373d14f82391e63fca3b"}
Sep 17 17:39:50.158368 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:50.158289    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen6n7l"
Sep 17 17:39:50.224189 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:50.224130    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen6n7l" podStartSLOduration=2.224113317 podStartE2EDuration="2.224113317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:39:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:39:50.221507781 +0000 UTC m=+3476.115986917" watchObservedRunningTime="2026-09-17 17:39:50.224113317 +0000 UTC m=+3476.118592455"
Sep 17 17:39:52.171688 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:52.171654    2566 generic.go:356] "Generic (PLEG): container finished" podID="c6e3ccc3-30b7-421a-96b3-65dcd3822747" containerID="c7550015443ba7593d200e65b2c41ba91c1371d3621b39622626904d0ca2f06b" exitCode=0
Sep 17 17:39:52.172087 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:52.171713    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"c6e3ccc3-30b7-421a-96b3-65dcd3822747","Type":"ContainerDied","Data":"c7550015443ba7593d200e65b2c41ba91c1371d3621b39622626904d0ca2f06b"}
Sep 17 17:39:53.179309 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:53.179263    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"c6e3ccc3-30b7-421a-96b3-65dcd3822747","Type":"ContainerStarted","Data":"4aca4c941d7169f62a00818888cb1832545fa53f941df9507aaa4da275e80015"}
Sep 17 17:39:53.207098 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:39:53.206978    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=2.773526709 podStartE2EDuration="5.206962558s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.433435849s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:39:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:39:53.204566636 +0000 UTC m=+3479.099045774" watchObservedRunningTime="2026-09-17 17:39:53.206962558 +0000 UTC m=+3479.101441693"
Sep 17 17:40:21.166236 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:21.166149    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen6n7l"
Sep 17 17:40:27.879226 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:27.879186    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq_c7b2e079-bc7d-44db-b674-9d0e5fa6afe0/storage-initializer/0.log"
Sep 17 17:40:27.889193 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:27.889166    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq_c7b2e079-bc7d-44db-b674-9d0e5fa6afe0/main/0.log"
Sep 17 17:40:29.401920 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:29.401885    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"]
Sep 17 17:40:44.425642 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:44.425573    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq" podUID="c7b2e079-bc7d-44db-b674-9d0e5fa6afe0" containerName="main" containerID="cri-o://961b7d10286f6e7f4814a3cb9c179f518b3d0a618ed3d393984ca5cfd730109c" gracePeriod=45
Sep 17 17:40:45.096304 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:45.096272    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"
Sep 17 17:40:45.204749 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:45.204636    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-tls-certs\") pod \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\" (UID: \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\") "
Sep 17 17:40:45.204749 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:45.204686    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-dshm\") pod \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\" (UID: \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\") "
Sep 17 17:40:45.204749 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:45.204714    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-model-cache\") pod \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\" (UID: \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\") "
Sep 17 17:40:45.205051 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:45.204852    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"lora-pvc-lora-adapter-1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0024c64a9d90292eb\") pod \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\" (UID: \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\") "
Sep 17 17:40:45.205051 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:45.204895    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xbsv\" (UniqueName: \"kubernetes.io/projected/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-kube-api-access-8xbsv\") pod \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\" (UID: \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\") "
Sep 17 17:40:45.205051 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:45.204948    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-tmp-dir\") pod \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\" (UID: \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\") "
Sep 17 17:40:45.206203 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:45.205563    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-cabundle-cert\") pod \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\" (UID: \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\") "
Sep 17 17:40:45.206203 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:45.205640    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-home\") pod \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\" (UID: \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\") "
Sep 17 17:40:45.206203 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:45.205691    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-kserve-provision-location\") pod \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\" (UID: \"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0\") "
Sep 17 17:40:45.207832 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:45.205023    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-model-cache" (OuterVolumeSpecName: "model-cache") pod "c7b2e079-bc7d-44db-b674-9d0e5fa6afe0" (UID: "c7b2e079-bc7d-44db-b674-9d0e5fa6afe0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:40:45.207832 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:45.206765    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-home" (OuterVolumeSpecName: "home") pod "c7b2e079-bc7d-44db-b674-9d0e5fa6afe0" (UID: "c7b2e079-bc7d-44db-b674-9d0e5fa6afe0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:40:45.210128 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:45.206145    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c7b2e079-bc7d-44db-b674-9d0e5fa6afe0" (UID: "c7b2e079-bc7d-44db-b674-9d0e5fa6afe0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:40:45.210128 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:45.208211    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c7b2e079-bc7d-44db-b674-9d0e5fa6afe0" (UID: "c7b2e079-bc7d-44db-b674-9d0e5fa6afe0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:40:45.215630 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:45.210974    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-cabundle-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:40:45.215630 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:45.211012    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-home\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:40:45.215630 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:45.211025    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:40:45.215630 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:45.211036    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-model-cache\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:40:45.215630 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:45.212597    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-kube-api-access-8xbsv" (OuterVolumeSpecName: "kube-api-access-8xbsv") pod "c7b2e079-bc7d-44db-b674-9d0e5fa6afe0" (UID: "c7b2e079-bc7d-44db-b674-9d0e5fa6afe0"). InnerVolumeSpecName "kube-api-access-8xbsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:40:45.216201 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:45.216171    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0024c64a9d90292eb" (OuterVolumeSpecName: "lora-pvc-lora-adapter-1") pod "c7b2e079-bc7d-44db-b674-9d0e5fa6afe0" (UID: "c7b2e079-bc7d-44db-b674-9d0e5fa6afe0"). InnerVolumeSpecName "pvc-e1a7e358-29a8-4ae3-8862-a1adb249ecd3". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 17 17:40:45.218565 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:45.218512    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-dshm" (OuterVolumeSpecName: "dshm") pod "c7b2e079-bc7d-44db-b674-9d0e5fa6afe0" (UID: "c7b2e079-bc7d-44db-b674-9d0e5fa6afe0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:40:45.225507 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:45.225474    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c7b2e079-bc7d-44db-b674-9d0e5fa6afe0" (UID: "c7b2e079-bc7d-44db-b674-9d0e5fa6afe0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:40:45.241285 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:45.241242    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c7b2e079-bc7d-44db-b674-9d0e5fa6afe0" (UID: "c7b2e079-bc7d-44db-b674-9d0e5fa6afe0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:40:45.311657 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:45.311610    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-tmp-dir\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:40:45.311657 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:45.311650    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-kserve-provision-location\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:40:45.311657 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:45.311665    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-dshm\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:40:45.311979 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:45.311705    2566 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-e1a7e358-29a8-4ae3-8862-a1adb249ecd3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0024c64a9d90292eb\") on node \"ip-10-0-141-66.ec2.internal\" "
Sep 17 17:40:45.311979 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:45.311718    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xbsv\" (UniqueName: \"kubernetes.io/projected/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0-kube-api-access-8xbsv\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:40:45.322341 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:45.322316    2566 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-e1a7e358-29a8-4ae3-8862-a1adb249ecd3" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0024c64a9d90292eb") on node "ip-10-0-141-66.ec2.internal"
Sep 17 17:40:45.413120 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:45.413077    2566 reconciler_common.go:299] "Volume detached for volume \"pvc-e1a7e358-29a8-4ae3-8862-a1adb249ecd3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0024c64a9d90292eb\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"csi-fc8ae150a79d7da6e19bc357e70065ddda4dcc641a905ccb9a27942b54e570c7\""
Sep 17 17:40:45.473628 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:45.473520    2566 generic.go:356] "Generic (PLEG): container finished" podID="c7b2e079-bc7d-44db-b674-9d0e5fa6afe0" containerID="961b7d10286f6e7f4814a3cb9c179f518b3d0a618ed3d393984ca5cfd730109c" exitCode=0
Sep 17 17:40:45.473628 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:45.473605    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"
Sep 17 17:40:45.473628 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:45.473617    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq" event={"ID":"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0","Type":"ContainerDied","Data":"961b7d10286f6e7f4814a3cb9c179f518b3d0a618ed3d393984ca5cfd730109c"}
Sep 17 17:40:45.474213 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:45.473680    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq" event={"ID":"c7b2e079-bc7d-44db-b674-9d0e5fa6afe0","Type":"ContainerDied","Data":"2b57dcbb14e893a060488c30c889674c6928c38679e2e2f0abf4f032e9082fe7"}
Sep 17 17:40:45.474213 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:45.473705    2566 scope.go:117] "RemoveContainer" containerID="961b7d10286f6e7f4814a3cb9c179f518b3d0a618ed3d393984ca5cfd730109c"
Sep 17 17:40:45.485387 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:45.485368    2566 scope.go:117] "RemoveContainer" containerID="7e1955bf98e26252a787b93a1826fd7014cf3859e438574c10c2f2488b544c4f"
Sep 17 17:40:45.496545 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:45.496403    2566 scope.go:117] "RemoveContainer" containerID="961b7d10286f6e7f4814a3cb9c179f518b3d0a618ed3d393984ca5cfd730109c"
Sep 17 17:40:45.496913 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:40:45.496881    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"961b7d10286f6e7f4814a3cb9c179f518b3d0a618ed3d393984ca5cfd730109c\": container with ID starting with 961b7d10286f6e7f4814a3cb9c179f518b3d0a618ed3d393984ca5cfd730109c not found: ID does not exist" containerID="961b7d10286f6e7f4814a3cb9c179f518b3d0a618ed3d393984ca5cfd730109c"
Sep 17 17:40:45.497008 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:45.496924    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"961b7d10286f6e7f4814a3cb9c179f518b3d0a618ed3d393984ca5cfd730109c"} err="failed to get container status \"961b7d10286f6e7f4814a3cb9c179f518b3d0a618ed3d393984ca5cfd730109c\": rpc error: code = NotFound desc = could not find container \"961b7d10286f6e7f4814a3cb9c179f518b3d0a618ed3d393984ca5cfd730109c\": container with ID starting with 961b7d10286f6e7f4814a3cb9c179f518b3d0a618ed3d393984ca5cfd730109c not found: ID does not exist"
Sep 17 17:40:45.497008 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:45.496952    2566 scope.go:117] "RemoveContainer" containerID="7e1955bf98e26252a787b93a1826fd7014cf3859e438574c10c2f2488b544c4f"
Sep 17 17:40:45.497924 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:40:45.497893    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e1955bf98e26252a787b93a1826fd7014cf3859e438574c10c2f2488b544c4f\": container with ID starting with 7e1955bf98e26252a787b93a1826fd7014cf3859e438574c10c2f2488b544c4f not found: ID does not exist" containerID="7e1955bf98e26252a787b93a1826fd7014cf3859e438574c10c2f2488b544c4f"
Sep 17 17:40:45.498031 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:45.497936    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e1955bf98e26252a787b93a1826fd7014cf3859e438574c10c2f2488b544c4f"} err="failed to get container status \"7e1955bf98e26252a787b93a1826fd7014cf3859e438574c10c2f2488b544c4f\": rpc error: code = NotFound desc = could not find container \"7e1955bf98e26252a787b93a1826fd7014cf3859e438574c10c2f2488b544c4f\": container with ID starting with 7e1955bf98e26252a787b93a1826fd7014cf3859e438574c10c2f2488b544c4f not found: ID does not exist"
Sep 17 17:40:45.499078 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:45.499057    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"]
Sep 17 17:40:45.505515 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:45.505489    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-d5d454696-z5zdq"]
Sep 17 17:40:46.347131 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:46.347092    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7b2e079-bc7d-44db-b674-9d0e5fa6afe0" path="/var/lib/kubelet/pods/c7b2e079-bc7d-44db-b674-9d0e5fa6afe0/volumes"
Sep 17 17:40:54.479354 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:54.479320    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-vwf7h"]
Sep 17 17:40:54.482939 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:54.482909    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-vwf7h"]
Sep 17 17:40:56.347541 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:40:56.347504    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="daacec1d-4a81-4b3c-8414-b3a53b1264cb" path="/var/lib/kubelet/pods/daacec1d-4a81-4b3c-8414-b3a53b1264cb/volumes"
Sep 17 17:41:29.887595 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:29.887540    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz"]
Sep 17 17:41:29.888115 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:29.887901    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7b2e079-bc7d-44db-b674-9d0e5fa6afe0" containerName="storage-initializer"
Sep 17 17:41:29.888115 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:29.887912    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7b2e079-bc7d-44db-b674-9d0e5fa6afe0" containerName="storage-initializer"
Sep 17 17:41:29.888115 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:29.887920    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7b2e079-bc7d-44db-b674-9d0e5fa6afe0" containerName="main"
Sep 17 17:41:29.888115 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:29.887925    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7b2e079-bc7d-44db-b674-9d0e5fa6afe0" containerName="main"
Sep 17 17:41:29.888115 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:29.887985    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7b2e079-bc7d-44db-b674-9d0e5fa6afe0" containerName="main"
Sep 17 17:41:29.897720 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:29.897685    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz"
Sep 17 17:41:29.900394 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:29.900367    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 17 17:41:29.900572 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:29.900369    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-fg7b7\""
Sep 17 17:41:29.900647 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:29.900575    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"lora-pvc-adapters-test-kserve-self-signed-certs\""
Sep 17 17:41:29.900712 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:29.900679    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 17 17:41:29.906289 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:29.906264    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz"]
Sep 17 17:41:29.986392 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:29.986347    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa9041a8-ac50-434f-a966-685773e2aff6-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-56mdz\" (UID: \"aa9041a8-ac50-434f-a966-685773e2aff6\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz"
Sep 17 17:41:29.986392 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:29.986392    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa9041a8-ac50-434f-a966-685773e2aff6-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-56mdz\" (UID: \"aa9041a8-ac50-434f-a966-685773e2aff6\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz"
Sep 17 17:41:29.986657 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:29.986423    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa9041a8-ac50-434f-a966-685773e2aff6-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-56mdz\" (UID: \"aa9041a8-ac50-434f-a966-685773e2aff6\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz"
Sep 17 17:41:29.986657 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:29.986476    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa9041a8-ac50-434f-a966-685773e2aff6-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-56mdz\" (UID: \"aa9041a8-ac50-434f-a966-685773e2aff6\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz"
Sep 17 17:41:29.986657 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:29.986516    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa9041a8-ac50-434f-a966-685773e2aff6-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-56mdz\" (UID: \"aa9041a8-ac50-434f-a966-685773e2aff6\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz"
Sep 17 17:41:29.986657 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:29.986575    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8fl7\" (UniqueName: \"kubernetes.io/projected/aa9041a8-ac50-434f-a966-685773e2aff6-kube-api-access-n8fl7\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-56mdz\" (UID: \"aa9041a8-ac50-434f-a966-685773e2aff6\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz"
Sep 17 17:41:30.087219 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:30.087173    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa9041a8-ac50-434f-a966-685773e2aff6-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-56mdz\" (UID: \"aa9041a8-ac50-434f-a966-685773e2aff6\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz"
Sep 17 17:41:30.087219 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:30.087220    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa9041a8-ac50-434f-a966-685773e2aff6-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-56mdz\" (UID: \"aa9041a8-ac50-434f-a966-685773e2aff6\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz"
Sep 17 17:41:30.087535 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:30.087247    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa9041a8-ac50-434f-a966-685773e2aff6-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-56mdz\" (UID: \"aa9041a8-ac50-434f-a966-685773e2aff6\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz"
Sep 17 17:41:30.087535 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:30.087267    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa9041a8-ac50-434f-a966-685773e2aff6-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-56mdz\" (UID: \"aa9041a8-ac50-434f-a966-685773e2aff6\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz"
Sep 17 17:41:30.087535 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:30.087314    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa9041a8-ac50-434f-a966-685773e2aff6-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-56mdz\" (UID: \"aa9041a8-ac50-434f-a966-685773e2aff6\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz"
Sep 17 17:41:30.087535 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:30.087349    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8fl7\" (UniqueName: \"kubernetes.io/projected/aa9041a8-ac50-434f-a966-685773e2aff6-kube-api-access-n8fl7\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-56mdz\" (UID: \"aa9041a8-ac50-434f-a966-685773e2aff6\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz"
Sep 17 17:41:30.087741 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:30.087699    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa9041a8-ac50-434f-a966-685773e2aff6-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-56mdz\" (UID: \"aa9041a8-ac50-434f-a966-685773e2aff6\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz"
Sep 17 17:41:30.087741 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:30.087733    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa9041a8-ac50-434f-a966-685773e2aff6-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-56mdz\" (UID: \"aa9041a8-ac50-434f-a966-685773e2aff6\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz"
Sep 17 17:41:30.087831 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:30.087786    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa9041a8-ac50-434f-a966-685773e2aff6-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-56mdz\" (UID: \"aa9041a8-ac50-434f-a966-685773e2aff6\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz"
Sep 17 17:41:30.089843 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:30.089815    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa9041a8-ac50-434f-a966-685773e2aff6-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-56mdz\" (UID: \"aa9041a8-ac50-434f-a966-685773e2aff6\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz"
Sep 17 17:41:30.090176 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:30.090157    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa9041a8-ac50-434f-a966-685773e2aff6-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-56mdz\" (UID: \"aa9041a8-ac50-434f-a966-685773e2aff6\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz"
Sep 17 17:41:30.096744 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:30.096721    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8fl7\" (UniqueName: \"kubernetes.io/projected/aa9041a8-ac50-434f-a966-685773e2aff6-kube-api-access-n8fl7\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-56mdz\" (UID: \"aa9041a8-ac50-434f-a966-685773e2aff6\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz"
Sep 17 17:41:36.541571 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:36.541463    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-ee8a953e-381f-4eaf-9d9a-b1637e28015e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0200295866b2d3505\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-56mdz\" (UID: \"aa9041a8-ac50-434f-a966-685773e2aff6\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz"
Sep 17 17:41:36.549339 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:36.549313    2566 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-ee8a953e-381f-4eaf-9d9a-b1637e28015e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0200295866b2d3505\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-56mdz\" (UID: \"aa9041a8-ac50-434f-a966-685773e2aff6\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz"
Sep 17 17:41:36.642452 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:36.642408    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-ee8a953e-381f-4eaf-9d9a-b1637e28015e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0200295866b2d3505\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-56mdz\" (UID: \"aa9041a8-ac50-434f-a966-685773e2aff6\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz"
Sep 17 17:41:36.642636 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:36.642557    2566 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-ee8a953e-381f-4eaf-9d9a-b1637e28015e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0200295866b2d3505\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-56mdz\" (UID: \"aa9041a8-ac50-434f-a966-685773e2aff6\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz"
Sep 17 17:41:36.646185 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:36.646162    2566 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-ee8a953e-381f-4eaf-9d9a-b1637e28015e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0200295866b2d3505\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-56mdz\" (UID: \"aa9041a8-ac50-434f-a966-685773e2aff6\") DevicePath \"csi-cd44c6a9b855f897a4e570992996311170e4784841e5044b45da57d8e3bd96e7\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz"
Sep 17 17:41:36.746168 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:36.746122    2566 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-ee8a953e-381f-4eaf-9d9a-b1637e28015e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0200295866b2d3505\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-56mdz\" (UID: \"aa9041a8-ac50-434f-a966-685773e2aff6\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/76b44bf1d5f98fa88e8e38d0816d624e6412f2416e862469cfe9db7b87b3f5e4/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz"
Sep 17 17:41:36.768270 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:36.768239    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-ee8a953e-381f-4eaf-9d9a-b1637e28015e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0200295866b2d3505\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-56mdz\" (UID: \"aa9041a8-ac50-434f-a966-685773e2aff6\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz"
Sep 17 17:41:36.809191 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:36.809097    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz"
Sep 17 17:41:37.092966 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:37.092936    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz"]
Sep 17 17:41:37.095102 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:41:37.095073    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa9041a8_ac50_434f_a966_685773e2aff6.slice/crio-8a5de135a794435416c48cd18e4b4a17b0e2174c91c7a25f15247bf254cc14c0 WatchSource:0}: Error finding container 8a5de135a794435416c48cd18e4b4a17b0e2174c91c7a25f15247bf254cc14c0: Status 404 returned error can't find the container with id 8a5de135a794435416c48cd18e4b4a17b0e2174c91c7a25f15247bf254cc14c0
Sep 17 17:41:37.767086 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:37.767034    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz" event={"ID":"aa9041a8-ac50-434f-a966-685773e2aff6","Type":"ContainerStarted","Data":"92c3f9e77267e2d0c7f9383bea2e8f806bfce07ac33a079b4dff55af3367b697"}
Sep 17 17:41:37.767537 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:37.767124    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz" event={"ID":"aa9041a8-ac50-434f-a966-685773e2aff6","Type":"ContainerStarted","Data":"8a5de135a794435416c48cd18e4b4a17b0e2174c91c7a25f15247bf254cc14c0"}
Sep 17 17:41:37.809268 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:37.809216    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz"
Sep 17 17:41:44.274606 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:44.274539    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz" podStartSLOduration=15.274512328 podStartE2EDuration="15.274512328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:37.804095054 +0000 UTC m=+3583.698574194" watchObservedRunningTime="2026-09-17 17:41:44.274512328 +0000 UTC m=+3590.168991467"
Sep 17 17:41:44.275022 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:44.274760    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen6n7l"]
Sep 17 17:41:46.809491 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:46.809429    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz"
Sep 17 17:41:46.811315 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:46.811279    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz" podUID="aa9041a8-ac50-434f-a966-685773e2aff6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Sep 17 17:41:51.166687 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:51.166643    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen6n7l" podUID="263ccc61-917f-4a02-8108-5f9307cfe728" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:41:56.641214 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:56.641183    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gblhr_91477834-6648-4e75-8530-e01bdee5e57e/ovn-acl-logging/0.log"
Sep 17 17:41:56.658554 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:56.658526    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gblhr_91477834-6648-4e75-8530-e01bdee5e57e/ovn-acl-logging/0.log"
Sep 17 17:41:56.810086 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:56.810031    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz" podUID="aa9041a8-ac50-434f-a966-685773e2aff6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Sep 17 17:41:59.295070 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:41:59.294992    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen6n7l" podUID="263ccc61-917f-4a02-8108-5f9307cfe728" containerName="main" containerID="cri-o://1cd28cf808825b39f012d8923513dd1a6dc58160d490d1ceb5447da06b0ad530" gracePeriod=45
Sep 17 17:42:00.449521 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:00.449478    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 17:42:01.166661 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:01.166616    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen6n7l" podUID="263ccc61-917f-4a02-8108-5f9307cfe728" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:42:06.810200 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:06.810150    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz" podUID="aa9041a8-ac50-434f-a966-685773e2aff6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Sep 17 17:42:11.166295 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:11.166252    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen6n7l" podUID="263ccc61-917f-4a02-8108-5f9307cfe728" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:42:11.166802 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:11.166376    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen6n7l"
Sep 17 17:42:15.472583 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:15.472512    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="c6e3ccc3-30b7-421a-96b3-65dcd3822747" containerName="main" containerID="cri-o://4aca4c941d7169f62a00818888cb1832545fa53f941df9507aaa4da275e80015" gracePeriod=45
Sep 17 17:42:16.241844 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:16.241807    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 17:42:16.282812 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:16.282707    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6e3ccc3-30b7-421a-96b3-65dcd3822747-model-cache\") pod \"c6e3ccc3-30b7-421a-96b3-65dcd3822747\" (UID: \"c6e3ccc3-30b7-421a-96b3-65dcd3822747\") "
Sep 17 17:42:16.282812 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:16.282773    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6e3ccc3-30b7-421a-96b3-65dcd3822747-cabundle-cert\") pod \"c6e3ccc3-30b7-421a-96b3-65dcd3822747\" (UID: \"c6e3ccc3-30b7-421a-96b3-65dcd3822747\") "
Sep 17 17:42:16.283075 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:16.282822    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6e3ccc3-30b7-421a-96b3-65dcd3822747-tmp-dir\") pod \"c6e3ccc3-30b7-421a-96b3-65dcd3822747\" (UID: \"c6e3ccc3-30b7-421a-96b3-65dcd3822747\") "
Sep 17 17:42:16.283075 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:16.282847    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6e3ccc3-30b7-421a-96b3-65dcd3822747-tls-certs\") pod \"c6e3ccc3-30b7-421a-96b3-65dcd3822747\" (UID: \"c6e3ccc3-30b7-421a-96b3-65dcd3822747\") "
Sep 17 17:42:16.283075 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:16.282869    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l42c9\" (UniqueName: \"kubernetes.io/projected/c6e3ccc3-30b7-421a-96b3-65dcd3822747-kube-api-access-l42c9\") pod \"c6e3ccc3-30b7-421a-96b3-65dcd3822747\" (UID: \"c6e3ccc3-30b7-421a-96b3-65dcd3822747\") "
Sep 17 17:42:16.283075 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:16.282910    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6e3ccc3-30b7-421a-96b3-65dcd3822747-kserve-provision-location\") pod \"c6e3ccc3-30b7-421a-96b3-65dcd3822747\" (UID: \"c6e3ccc3-30b7-421a-96b3-65dcd3822747\") "
Sep 17 17:42:16.283075 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:16.282934    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6e3ccc3-30b7-421a-96b3-65dcd3822747-dshm\") pod \"c6e3ccc3-30b7-421a-96b3-65dcd3822747\" (UID: \"c6e3ccc3-30b7-421a-96b3-65dcd3822747\") "
Sep 17 17:42:16.283075 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:16.282981    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6e3ccc3-30b7-421a-96b3-65dcd3822747-home\") pod \"c6e3ccc3-30b7-421a-96b3-65dcd3822747\" (UID: \"c6e3ccc3-30b7-421a-96b3-65dcd3822747\") "
Sep 17 17:42:16.283075 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:16.282994    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e3ccc3-30b7-421a-96b3-65dcd3822747-model-cache" (OuterVolumeSpecName: "model-cache") pod "c6e3ccc3-30b7-421a-96b3-65dcd3822747" (UID: "c6e3ccc3-30b7-421a-96b3-65dcd3822747"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:42:16.283523 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:16.283213    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6e3ccc3-30b7-421a-96b3-65dcd3822747-model-cache\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:42:16.283523 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:16.283225    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6e3ccc3-30b7-421a-96b3-65dcd3822747-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c6e3ccc3-30b7-421a-96b3-65dcd3822747" (UID: "c6e3ccc3-30b7-421a-96b3-65dcd3822747"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:42:16.283827 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:16.283803    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e3ccc3-30b7-421a-96b3-65dcd3822747-home" (OuterVolumeSpecName: "home") pod "c6e3ccc3-30b7-421a-96b3-65dcd3822747" (UID: "c6e3ccc3-30b7-421a-96b3-65dcd3822747"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:42:16.286201 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:16.286153    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e3ccc3-30b7-421a-96b3-65dcd3822747-dshm" (OuterVolumeSpecName: "dshm") pod "c6e3ccc3-30b7-421a-96b3-65dcd3822747" (UID: "c6e3ccc3-30b7-421a-96b3-65dcd3822747"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:42:16.286337 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:16.286210    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6e3ccc3-30b7-421a-96b3-65dcd3822747-kube-api-access-l42c9" (OuterVolumeSpecName: "kube-api-access-l42c9") pod "c6e3ccc3-30b7-421a-96b3-65dcd3822747" (UID: "c6e3ccc3-30b7-421a-96b3-65dcd3822747"). InnerVolumeSpecName "kube-api-access-l42c9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:42:16.286559 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:16.286524    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6e3ccc3-30b7-421a-96b3-65dcd3822747-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c6e3ccc3-30b7-421a-96b3-65dcd3822747" (UID: "c6e3ccc3-30b7-421a-96b3-65dcd3822747"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:42:16.299414 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:16.299371    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e3ccc3-30b7-421a-96b3-65dcd3822747-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c6e3ccc3-30b7-421a-96b3-65dcd3822747" (UID: "c6e3ccc3-30b7-421a-96b3-65dcd3822747"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:42:16.310002 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:16.309963    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e3ccc3-30b7-421a-96b3-65dcd3822747-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c6e3ccc3-30b7-421a-96b3-65dcd3822747" (UID: "c6e3ccc3-30b7-421a-96b3-65dcd3822747"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:42:16.384037 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:16.383991    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6e3ccc3-30b7-421a-96b3-65dcd3822747-cabundle-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:42:16.384037 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:16.384043    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6e3ccc3-30b7-421a-96b3-65dcd3822747-tmp-dir\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:42:16.384346 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:16.384054    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6e3ccc3-30b7-421a-96b3-65dcd3822747-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:42:16.384346 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:16.384062    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l42c9\" (UniqueName: \"kubernetes.io/projected/c6e3ccc3-30b7-421a-96b3-65dcd3822747-kube-api-access-l42c9\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:42:16.384346 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:16.384072    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6e3ccc3-30b7-421a-96b3-65dcd3822747-kserve-provision-location\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:42:16.384346 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:16.384080    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6e3ccc3-30b7-421a-96b3-65dcd3822747-dshm\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:42:16.384346 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:16.384087    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6e3ccc3-30b7-421a-96b3-65dcd3822747-home\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:42:16.810012 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:16.809968    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz" podUID="aa9041a8-ac50-434f-a966-685773e2aff6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Sep 17 17:42:16.988700 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:16.988658    2566 generic.go:356] "Generic (PLEG): container finished" podID="c6e3ccc3-30b7-421a-96b3-65dcd3822747" containerID="4aca4c941d7169f62a00818888cb1832545fa53f941df9507aaa4da275e80015" exitCode=0
Sep 17 17:42:16.988912 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:16.988715    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 17:42:16.988912 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:16.988738    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"c6e3ccc3-30b7-421a-96b3-65dcd3822747","Type":"ContainerDied","Data":"4aca4c941d7169f62a00818888cb1832545fa53f941df9507aaa4da275e80015"}
Sep 17 17:42:16.988912 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:16.988801    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"c6e3ccc3-30b7-421a-96b3-65dcd3822747","Type":"ContainerDied","Data":"6b3fbcb2bae307f2130f98e36df08b2a4831de2b9649166bc498a504f122a0bb"}
Sep 17 17:42:16.988912 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:16.988824    2566 scope.go:117] "RemoveContainer" containerID="4aca4c941d7169f62a00818888cb1832545fa53f941df9507aaa4da275e80015"
Sep 17 17:42:17.000759 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:17.000737    2566 scope.go:117] "RemoveContainer" containerID="c7550015443ba7593d200e65b2c41ba91c1371d3621b39622626904d0ca2f06b"
Sep 17 17:42:17.011176 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:17.011148    2566 scope.go:117] "RemoveContainer" containerID="4aca4c941d7169f62a00818888cb1832545fa53f941df9507aaa4da275e80015"
Sep 17 17:42:17.011525 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:42:17.011504    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aca4c941d7169f62a00818888cb1832545fa53f941df9507aaa4da275e80015\": container with ID starting with 4aca4c941d7169f62a00818888cb1832545fa53f941df9507aaa4da275e80015 not found: ID does not exist" containerID="4aca4c941d7169f62a00818888cb1832545fa53f941df9507aaa4da275e80015"
Sep 17 17:42:17.011613 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:17.011535    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aca4c941d7169f62a00818888cb1832545fa53f941df9507aaa4da275e80015"} err="failed to get container status \"4aca4c941d7169f62a00818888cb1832545fa53f941df9507aaa4da275e80015\": rpc error: code = NotFound desc = could not find container \"4aca4c941d7169f62a00818888cb1832545fa53f941df9507aaa4da275e80015\": container with ID starting with 4aca4c941d7169f62a00818888cb1832545fa53f941df9507aaa4da275e80015 not found: ID does not exist"
Sep 17 17:42:17.011613 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:17.011553    2566 scope.go:117] "RemoveContainer" containerID="c7550015443ba7593d200e65b2c41ba91c1371d3621b39622626904d0ca2f06b"
Sep 17 17:42:17.011872 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:42:17.011847    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7550015443ba7593d200e65b2c41ba91c1371d3621b39622626904d0ca2f06b\": container with ID starting with c7550015443ba7593d200e65b2c41ba91c1371d3621b39622626904d0ca2f06b not found: ID does not exist" containerID="c7550015443ba7593d200e65b2c41ba91c1371d3621b39622626904d0ca2f06b"
Sep 17 17:42:17.011916 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:17.011868    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7550015443ba7593d200e65b2c41ba91c1371d3621b39622626904d0ca2f06b"} err="failed to get container status \"c7550015443ba7593d200e65b2c41ba91c1371d3621b39622626904d0ca2f06b\": rpc error: code = NotFound desc = could not find container \"c7550015443ba7593d200e65b2c41ba91c1371d3621b39622626904d0ca2f06b\": container with ID starting with c7550015443ba7593d200e65b2c41ba91c1371d3621b39622626904d0ca2f06b not found: ID does not exist"
Sep 17 17:42:17.012015 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:17.011995    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 17:42:17.025793 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:17.025759    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 17:42:18.347315 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:18.347278    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6e3ccc3-30b7-421a-96b3-65dcd3822747" path="/var/lib/kubelet/pods/c6e3ccc3-30b7-421a-96b3-65dcd3822747/volumes"
Sep 17 17:42:21.166572 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:21.166529    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen6n7l" podUID="263ccc61-917f-4a02-8108-5f9307cfe728" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:42:26.810232 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:26.810188    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz" podUID="aa9041a8-ac50-434f-a966-685773e2aff6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Sep 17 17:42:29.555704 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:29.555672    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen6n7l"
Sep 17 17:42:29.692752 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:29.692640    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/263ccc61-917f-4a02-8108-5f9307cfe728-tls-certs\") pod \"263ccc61-917f-4a02-8108-5f9307cfe728\" (UID: \"263ccc61-917f-4a02-8108-5f9307cfe728\") "
Sep 17 17:42:29.692937 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:29.692757    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zv94\" (UniqueName: \"kubernetes.io/projected/263ccc61-917f-4a02-8108-5f9307cfe728-kube-api-access-4zv94\") pod \"263ccc61-917f-4a02-8108-5f9307cfe728\" (UID: \"263ccc61-917f-4a02-8108-5f9307cfe728\") "
Sep 17 17:42:29.695093 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:29.695056    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/263ccc61-917f-4a02-8108-5f9307cfe728-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "263ccc61-917f-4a02-8108-5f9307cfe728" (UID: "263ccc61-917f-4a02-8108-5f9307cfe728"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:42:29.695196 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:29.695114    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/263ccc61-917f-4a02-8108-5f9307cfe728-kube-api-access-4zv94" (OuterVolumeSpecName: "kube-api-access-4zv94") pod "263ccc61-917f-4a02-8108-5f9307cfe728" (UID: "263ccc61-917f-4a02-8108-5f9307cfe728"). InnerVolumeSpecName "kube-api-access-4zv94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:42:29.793723 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:29.793679    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/263ccc61-917f-4a02-8108-5f9307cfe728-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:42:29.793723 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:29.793725    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zv94\" (UniqueName: \"kubernetes.io/projected/263ccc61-917f-4a02-8108-5f9307cfe728-kube-api-access-4zv94\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:42:30.065669 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:30.065636    2566 generic.go:356] "Generic (PLEG): container finished" podID="263ccc61-917f-4a02-8108-5f9307cfe728" containerID="1cd28cf808825b39f012d8923513dd1a6dc58160d490d1ceb5447da06b0ad530" exitCode=0
Sep 17 17:42:30.065825 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:30.065683    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen6n7l" event={"ID":"263ccc61-917f-4a02-8108-5f9307cfe728","Type":"ContainerDied","Data":"1cd28cf808825b39f012d8923513dd1a6dc58160d490d1ceb5447da06b0ad530"}
Sep 17 17:42:30.065825 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:30.065691    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen6n7l"
Sep 17 17:42:30.065825 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:30.065754    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen6n7l" event={"ID":"263ccc61-917f-4a02-8108-5f9307cfe728","Type":"ContainerDied","Data":"32f30f14a6dd11a49d3c7d88cd28dc18f5cc1c3d7d6d373d14f82391e63fca3b"}
Sep 17 17:42:30.065825 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:30.065776    2566 scope.go:117] "RemoveContainer" containerID="1cd28cf808825b39f012d8923513dd1a6dc58160d490d1ceb5447da06b0ad530"
Sep 17 17:42:30.074903 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:30.074881    2566 scope.go:117] "RemoveContainer" containerID="1cd28cf808825b39f012d8923513dd1a6dc58160d490d1ceb5447da06b0ad530"
Sep 17 17:42:30.075190 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:42:30.075169    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cd28cf808825b39f012d8923513dd1a6dc58160d490d1ceb5447da06b0ad530\": container with ID starting with 1cd28cf808825b39f012d8923513dd1a6dc58160d490d1ceb5447da06b0ad530 not found: ID does not exist" containerID="1cd28cf808825b39f012d8923513dd1a6dc58160d490d1ceb5447da06b0ad530"
Sep 17 17:42:30.075241 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:30.075201    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cd28cf808825b39f012d8923513dd1a6dc58160d490d1ceb5447da06b0ad530"} err="failed to get container status \"1cd28cf808825b39f012d8923513dd1a6dc58160d490d1ceb5447da06b0ad530\": rpc error: code = NotFound desc = could not find container \"1cd28cf808825b39f012d8923513dd1a6dc58160d490d1ceb5447da06b0ad530\": container with ID starting with 1cd28cf808825b39f012d8923513dd1a6dc58160d490d1ceb5447da06b0ad530 not found: ID does not exist"
Sep 17 17:42:30.089145 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:30.089109    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen6n7l"]
Sep 17 17:42:30.095351 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:30.095324    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen6n7l"]
Sep 17 17:42:30.348040 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:30.347952    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="263ccc61-917f-4a02-8108-5f9307cfe728" path="/var/lib/kubelet/pods/263ccc61-917f-4a02-8108-5f9307cfe728/volumes"
Sep 17 17:42:36.810038 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:36.809990    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz" podUID="aa9041a8-ac50-434f-a966-685773e2aff6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Sep 17 17:42:45.571104 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:45.571050    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cblvb4g"]
Sep 17 17:42:45.572067 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:45.572039    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6e3ccc3-30b7-421a-96b3-65dcd3822747" containerName="main"
Sep 17 17:42:45.572171 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:45.572070    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6e3ccc3-30b7-421a-96b3-65dcd3822747" containerName="main"
Sep 17 17:42:45.572171 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:45.572090    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6e3ccc3-30b7-421a-96b3-65dcd3822747" containerName="storage-initializer"
Sep 17 17:42:45.572171 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:45.572096    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6e3ccc3-30b7-421a-96b3-65dcd3822747" containerName="storage-initializer"
Sep 17 17:42:45.572171 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:45.572107    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="263ccc61-917f-4a02-8108-5f9307cfe728" containerName="main"
Sep 17 17:42:45.572171 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:45.572111    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="263ccc61-917f-4a02-8108-5f9307cfe728" containerName="main"
Sep 17 17:42:45.572171 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:45.572172    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6e3ccc3-30b7-421a-96b3-65dcd3822747" containerName="main"
Sep 17 17:42:45.572542 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:45.572180    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="263ccc61-917f-4a02-8108-5f9307cfe728" containerName="main"
Sep 17 17:42:45.576788 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:45.576766    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cblvb4g"
Sep 17 17:42:45.579803 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:45.579774    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 17 17:42:45.579945 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:45.579826    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-epp-sa-dockercfg-rxbws\""
Sep 17 17:42:45.579945 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:45.579909    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-d4m52\""
Sep 17 17:42:45.579945 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:45.579794    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 17 17:42:45.580163 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:45.579779    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 17 17:42:45.583894 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:45.583869    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cblvb4g"]
Sep 17 17:42:45.728869 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:45.728833    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4160e4c-8713-4aa9-8b73-ded10a82e7aa-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cblvb4g\" (UID: \"d4160e4c-8713-4aa9-8b73-ded10a82e7aa\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cblvb4g"
Sep 17 17:42:45.729045 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:45.728927    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdnx6\" (UniqueName: \"kubernetes.io/projected/d4160e4c-8713-4aa9-8b73-ded10a82e7aa-kube-api-access-pdnx6\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cblvb4g\" (UID: \"d4160e4c-8713-4aa9-8b73-ded10a82e7aa\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cblvb4g"
Sep 17 17:42:45.829338 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:45.829232    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4160e4c-8713-4aa9-8b73-ded10a82e7aa-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cblvb4g\" (UID: \"d4160e4c-8713-4aa9-8b73-ded10a82e7aa\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cblvb4g"
Sep 17 17:42:45.829338 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:45.829308    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdnx6\" (UniqueName: \"kubernetes.io/projected/d4160e4c-8713-4aa9-8b73-ded10a82e7aa-kube-api-access-pdnx6\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cblvb4g\" (UID: \"d4160e4c-8713-4aa9-8b73-ded10a82e7aa\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cblvb4g"
Sep 17 17:42:45.832259 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:45.832220    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4160e4c-8713-4aa9-8b73-ded10a82e7aa-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cblvb4g\" (UID: \"d4160e4c-8713-4aa9-8b73-ded10a82e7aa\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cblvb4g"
Sep 17 17:42:45.839484 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:45.839416    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdnx6\" (UniqueName: \"kubernetes.io/projected/d4160e4c-8713-4aa9-8b73-ded10a82e7aa-kube-api-access-pdnx6\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cblvb4g\" (UID: \"d4160e4c-8713-4aa9-8b73-ded10a82e7aa\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cblvb4g"
Sep 17 17:42:45.890922 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:45.890883    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cblvb4g"
Sep 17 17:42:46.191347 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:46.191313    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cblvb4g"]
Sep 17 17:42:46.193589 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:42:46.193561    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4160e4c_8713_4aa9_8b73_ded10a82e7aa.slice/crio-4c64e734580a53e4d565883c4f6ff0efcaebafdf61418aed84449bcd819e2866 WatchSource:0}: Error finding container 4c64e734580a53e4d565883c4f6ff0efcaebafdf61418aed84449bcd819e2866: Status 404 returned error can't find the container with id 4c64e734580a53e4d565883c4f6ff0efcaebafdf61418aed84449bcd819e2866
Sep 17 17:42:46.810153 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:46.810102    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz" podUID="aa9041a8-ac50-434f-a966-685773e2aff6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Sep 17 17:42:47.163003 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:47.162886    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cblvb4g" event={"ID":"d4160e4c-8713-4aa9-8b73-ded10a82e7aa","Type":"ContainerStarted","Data":"a4ce4184788d39e88a1d7bb724ba6fdcbb1f5a835f03ba78a9503748e1ec89e2"}
Sep 17 17:42:47.163003 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:47.162960    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cblvb4g" event={"ID":"d4160e4c-8713-4aa9-8b73-ded10a82e7aa","Type":"ContainerStarted","Data":"4c64e734580a53e4d565883c4f6ff0efcaebafdf61418aed84449bcd819e2866"}
Sep 17 17:42:47.163003 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:47.162991    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cblvb4g"
Sep 17 17:42:47.182273 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:47.182216    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cblvb4g" podStartSLOduration=2.182199175 podStartE2EDuration="2.182199175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:47.180961563 +0000 UTC m=+3653.075440703" watchObservedRunningTime="2026-09-17 17:42:47.182199175 +0000 UTC m=+3653.076678314"
Sep 17 17:42:56.820134 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:56.820093    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz"
Sep 17 17:42:56.827871 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:42:56.827833    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz"
Sep 17 17:43:18.170241 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:18.170135    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cblvb4g"
Sep 17 17:43:20.256096 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:20.256044    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cblvb4g"]
Sep 17 17:43:27.805850 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:27.805819    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-a290d4b0_lora-pvc-adapters-test-kserve-8cfd5947f-56mdz_aa9041a8-ac50-434f-a966-685773e2aff6/main/0.log"
Sep 17 17:43:28.170293 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:28.170203    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cblvb4g" podUID="d4160e4c-8713-4aa9-8b73-ded10a82e7aa" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:43:29.672586 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:29.672549    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz"]
Sep 17 17:43:35.276819 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:35.276773    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cblvb4g" podUID="d4160e4c-8713-4aa9-8b73-ded10a82e7aa" containerName="main" containerID="cri-o://a4ce4184788d39e88a1d7bb724ba6fdcbb1f5a835f03ba78a9503748e1ec89e2" gracePeriod=45
Sep 17 17:43:38.170018 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:38.169974    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cblvb4g" podUID="d4160e4c-8713-4aa9-8b73-ded10a82e7aa" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:43:44.695664 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:44.695624    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz" podUID="aa9041a8-ac50-434f-a966-685773e2aff6" containerName="main" containerID="cri-o://92c3f9e77267e2d0c7f9383bea2e8f806bfce07ac33a079b4dff55af3367b697" gracePeriod=45
Sep 17 17:43:44.821550 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:44.821498    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz" podUID="aa9041a8-ac50-434f-a966-685773e2aff6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Sep 17 17:43:45.353913 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:45.353884    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz"
Sep 17 17:43:45.440709 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:45.440613    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa9041a8-ac50-434f-a966-685773e2aff6-tls-certs\") pod \"aa9041a8-ac50-434f-a966-685773e2aff6\" (UID: \"aa9041a8-ac50-434f-a966-685773e2aff6\") "
Sep 17 17:43:45.440709 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:45.440667    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa9041a8-ac50-434f-a966-685773e2aff6-dshm\") pod \"aa9041a8-ac50-434f-a966-685773e2aff6\" (UID: \"aa9041a8-ac50-434f-a966-685773e2aff6\") "
Sep 17 17:43:45.440709 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:45.440702    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa9041a8-ac50-434f-a966-685773e2aff6-home\") pod \"aa9041a8-ac50-434f-a966-685773e2aff6\" (UID: \"aa9041a8-ac50-434f-a966-685773e2aff6\") "
Sep 17 17:43:45.440960 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:45.440738    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa9041a8-ac50-434f-a966-685773e2aff6-model-cache\") pod \"aa9041a8-ac50-434f-a966-685773e2aff6\" (UID: \"aa9041a8-ac50-434f-a966-685773e2aff6\") "
Sep 17 17:43:45.440960 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:45.440835    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0200295866b2d3505\") pod \"aa9041a8-ac50-434f-a966-685773e2aff6\" (UID: \"aa9041a8-ac50-434f-a966-685773e2aff6\") "
Sep 17 17:43:45.440960 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:45.440869    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8fl7\" (UniqueName: \"kubernetes.io/projected/aa9041a8-ac50-434f-a966-685773e2aff6-kube-api-access-n8fl7\") pod \"aa9041a8-ac50-434f-a966-685773e2aff6\" (UID: \"aa9041a8-ac50-434f-a966-685773e2aff6\") "
Sep 17 17:43:45.440960 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:45.440909    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa9041a8-ac50-434f-a966-685773e2aff6-tmp-dir\") pod \"aa9041a8-ac50-434f-a966-685773e2aff6\" (UID: \"aa9041a8-ac50-434f-a966-685773e2aff6\") "
Sep 17 17:43:45.441182 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:45.441061    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa9041a8-ac50-434f-a966-685773e2aff6-model-cache" (OuterVolumeSpecName: "model-cache") pod "aa9041a8-ac50-434f-a966-685773e2aff6" (UID: "aa9041a8-ac50-434f-a966-685773e2aff6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:43:45.441182 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:45.441151    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa9041a8-ac50-434f-a966-685773e2aff6-home" (OuterVolumeSpecName: "home") pod "aa9041a8-ac50-434f-a966-685773e2aff6" (UID: "aa9041a8-ac50-434f-a966-685773e2aff6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:43:45.441293 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:45.441233    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa9041a8-ac50-434f-a966-685773e2aff6-model-cache\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:43:45.441293 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:45.441250    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa9041a8-ac50-434f-a966-685773e2aff6-home\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:43:45.443345 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:45.443311    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa9041a8-ac50-434f-a966-685773e2aff6-dshm" (OuterVolumeSpecName: "dshm") pod "aa9041a8-ac50-434f-a966-685773e2aff6" (UID: "aa9041a8-ac50-434f-a966-685773e2aff6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:43:45.443480 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:45.443355    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa9041a8-ac50-434f-a966-685773e2aff6-kube-api-access-n8fl7" (OuterVolumeSpecName: "kube-api-access-n8fl7") pod "aa9041a8-ac50-434f-a966-685773e2aff6" (UID: "aa9041a8-ac50-434f-a966-685773e2aff6"). InnerVolumeSpecName "kube-api-access-n8fl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:43:45.443612 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:45.443587    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa9041a8-ac50-434f-a966-685773e2aff6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aa9041a8-ac50-434f-a966-685773e2aff6" (UID: "aa9041a8-ac50-434f-a966-685773e2aff6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:43:45.449708 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:45.449684    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0200295866b2d3505" (OuterVolumeSpecName: "kserve-pvc-source") pod "aa9041a8-ac50-434f-a966-685773e2aff6" (UID: "aa9041a8-ac50-434f-a966-685773e2aff6"). InnerVolumeSpecName "pvc-ee8a953e-381f-4eaf-9d9a-b1637e28015e". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 17 17:43:45.452012 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:45.451984    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa9041a8-ac50-434f-a966-685773e2aff6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aa9041a8-ac50-434f-a966-685773e2aff6" (UID: "aa9041a8-ac50-434f-a966-685773e2aff6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:43:45.480283 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:45.480250    2566 generic.go:356] "Generic (PLEG): container finished" podID="aa9041a8-ac50-434f-a966-685773e2aff6" containerID="92c3f9e77267e2d0c7f9383bea2e8f806bfce07ac33a079b4dff55af3367b697" exitCode=0
Sep 17 17:43:45.480468 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:45.480298    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz"
Sep 17 17:43:45.480468 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:45.480331    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz" event={"ID":"aa9041a8-ac50-434f-a966-685773e2aff6","Type":"ContainerDied","Data":"92c3f9e77267e2d0c7f9383bea2e8f806bfce07ac33a079b4dff55af3367b697"}
Sep 17 17:43:45.480468 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:45.480385    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz" event={"ID":"aa9041a8-ac50-434f-a966-685773e2aff6","Type":"ContainerDied","Data":"8a5de135a794435416c48cd18e4b4a17b0e2174c91c7a25f15247bf254cc14c0"}
Sep 17 17:43:45.480468 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:45.480408    2566 scope.go:117] "RemoveContainer" containerID="92c3f9e77267e2d0c7f9383bea2e8f806bfce07ac33a079b4dff55af3367b697"
Sep 17 17:43:45.489753 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:45.489731    2566 scope.go:117] "RemoveContainer" containerID="92c3f9e77267e2d0c7f9383bea2e8f806bfce07ac33a079b4dff55af3367b697"
Sep 17 17:43:45.490051 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:43:45.490027    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92c3f9e77267e2d0c7f9383bea2e8f806bfce07ac33a079b4dff55af3367b697\": container with ID starting with 92c3f9e77267e2d0c7f9383bea2e8f806bfce07ac33a079b4dff55af3367b697 not found: ID does not exist" containerID="92c3f9e77267e2d0c7f9383bea2e8f806bfce07ac33a079b4dff55af3367b697"
Sep 17 17:43:45.490148 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:45.490057    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92c3f9e77267e2d0c7f9383bea2e8f806bfce07ac33a079b4dff55af3367b697"} err="failed to get container status \"92c3f9e77267e2d0c7f9383bea2e8f806bfce07ac33a079b4dff55af3367b697\": rpc error: code = NotFound desc = could not find container \"92c3f9e77267e2d0c7f9383bea2e8f806bfce07ac33a079b4dff55af3367b697\": container with ID starting with 92c3f9e77267e2d0c7f9383bea2e8f806bfce07ac33a079b4dff55af3367b697 not found: ID does not exist"
Sep 17 17:43:45.518201 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:45.518168    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz"]
Sep 17 17:43:45.527484 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:45.527428    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-56mdz"]
Sep 17 17:43:45.541700 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:45.541662    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa9041a8-ac50-434f-a966-685773e2aff6-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:43:45.541700 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:45.541692    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa9041a8-ac50-434f-a966-685773e2aff6-dshm\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:43:45.541917 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:45.541727    2566 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-ee8a953e-381f-4eaf-9d9a-b1637e28015e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0200295866b2d3505\") on node \"ip-10-0-141-66.ec2.internal\" "
Sep 17 17:43:45.541917 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:45.541742    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8fl7\" (UniqueName: \"kubernetes.io/projected/aa9041a8-ac50-434f-a966-685773e2aff6-kube-api-access-n8fl7\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:43:45.541917 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:45.541752    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa9041a8-ac50-434f-a966-685773e2aff6-tmp-dir\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:43:45.573679 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:45.573653    2566 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-ee8a953e-381f-4eaf-9d9a-b1637e28015e" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0200295866b2d3505") on node "ip-10-0-141-66.ec2.internal"
Sep 17 17:43:45.642712 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:45.642670    2566 reconciler_common.go:299] "Volume detached for volume \"pvc-ee8a953e-381f-4eaf-9d9a-b1637e28015e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0200295866b2d3505\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"csi-cd44c6a9b855f897a4e570992996311170e4784841e5044b45da57d8e3bd96e7\""
Sep 17 17:43:46.346275 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:46.346236    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa9041a8-ac50-434f-a966-685773e2aff6" path="/var/lib/kubelet/pods/aa9041a8-ac50-434f-a966-685773e2aff6/volumes"
Sep 17 17:43:48.169910 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:48.169862    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cblvb4g" podUID="d4160e4c-8713-4aa9-8b73-ded10a82e7aa" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:43:48.170392 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:48.169965    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cblvb4g"
Sep 17 17:43:56.897674 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:56.897636    2566 scope.go:117] "RemoveContainer" containerID="837ea866b5aa8732866a0f20a7d7a317cf38e2c59a23cd41e301d9732aa3ee95"
Sep 17 17:43:57.085922 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:57.085888    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cr6sh"]
Sep 17 17:43:57.086216 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:57.086205    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa9041a8-ac50-434f-a966-685773e2aff6" containerName="main"
Sep 17 17:43:57.086261 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:57.086217    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa9041a8-ac50-434f-a966-685773e2aff6" containerName="main"
Sep 17 17:43:57.086297 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:57.086286    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa9041a8-ac50-434f-a966-685773e2aff6" containerName="main"
Sep 17 17:43:57.090837 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:57.090818    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cr6sh"
Sep 17 17:43:57.096399 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:57.096370    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-epp-sa-dockercfg-kb8cj\""
Sep 17 17:43:57.097067 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:57.097040    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-l7xw4\""
Sep 17 17:43:57.097693 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:57.097065    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 17 17:43:57.098013 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:57.097981    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 17 17:43:57.098013 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:57.097989    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 17 17:43:57.106837 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:57.106809    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cr6sh"]
Sep 17 17:43:57.244117 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:57.244008    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8crd\" (UniqueName: \"kubernetes.io/projected/f5458154-d36a-404a-9449-8c79d64a6cf9-kube-api-access-m8crd\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-cr6sh\" (UID: \"f5458154-d36a-404a-9449-8c79d64a6cf9\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cr6sh"
Sep 17 17:43:57.244117 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:57.244079    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5458154-d36a-404a-9449-8c79d64a6cf9-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-cr6sh\" (UID: \"f5458154-d36a-404a-9449-8c79d64a6cf9\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cr6sh"
Sep 17 17:43:57.345537 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:57.345490    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5458154-d36a-404a-9449-8c79d64a6cf9-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-cr6sh\" (UID: \"f5458154-d36a-404a-9449-8c79d64a6cf9\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cr6sh"
Sep 17 17:43:57.345752 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:57.345591    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8crd\" (UniqueName: \"kubernetes.io/projected/f5458154-d36a-404a-9449-8c79d64a6cf9-kube-api-access-m8crd\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-cr6sh\" (UID: \"f5458154-d36a-404a-9449-8c79d64a6cf9\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cr6sh"
Sep 17 17:43:57.348315 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:57.348290    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5458154-d36a-404a-9449-8c79d64a6cf9-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-cr6sh\" (UID: \"f5458154-d36a-404a-9449-8c79d64a6cf9\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cr6sh"
Sep 17 17:43:57.358665 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:57.358636    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8crd\" (UniqueName: \"kubernetes.io/projected/f5458154-d36a-404a-9449-8c79d64a6cf9-kube-api-access-m8crd\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-cr6sh\" (UID: \"f5458154-d36a-404a-9449-8c79d64a6cf9\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cr6sh"
Sep 17 17:43:57.401722 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:57.401682    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cr6sh"
Sep 17 17:43:57.685993 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:57.685966    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cr6sh"]
Sep 17 17:43:57.688159 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:43:57.688132    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5458154_d36a_404a_9449_8c79d64a6cf9.slice/crio-339a92ed548fcef508c501f622f3536dd280cb193e09f0adbc6dc6d6a4ea5353 WatchSource:0}: Error finding container 339a92ed548fcef508c501f622f3536dd280cb193e09f0adbc6dc6d6a4ea5353: Status 404 returned error can't find the container with id 339a92ed548fcef508c501f622f3536dd280cb193e09f0adbc6dc6d6a4ea5353
Sep 17 17:43:58.170253 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:58.170207    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cblvb4g" podUID="d4160e4c-8713-4aa9-8b73-ded10a82e7aa" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:43:58.551068 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:58.551026    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cr6sh" event={"ID":"f5458154-d36a-404a-9449-8c79d64a6cf9","Type":"ContainerStarted","Data":"31cf1046e7199ec78a32f1fdef93fb6b2b1bc2b039a085b1ee6dbf4ca2774455"}
Sep 17 17:43:58.551260 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:58.551081    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cr6sh" event={"ID":"f5458154-d36a-404a-9449-8c79d64a6cf9","Type":"ContainerStarted","Data":"339a92ed548fcef508c501f622f3536dd280cb193e09f0adbc6dc6d6a4ea5353"}
Sep 17 17:43:58.551260 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:58.551178    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cr6sh"
Sep 17 17:43:58.576466 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:43:58.576392    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cr6sh" podStartSLOduration=1.576368937 podStartE2EDuration="1.576368937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:43:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:43:58.574127469 +0000 UTC m=+3724.468606608" watchObservedRunningTime="2026-09-17 17:43:58.576368937 +0000 UTC m=+3724.470848077"
Sep 17 17:44:05.535944 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:05.535915    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cblvb4g"
Sep 17 17:44:05.589723 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:05.589627    2566 generic.go:356] "Generic (PLEG): container finished" podID="d4160e4c-8713-4aa9-8b73-ded10a82e7aa" containerID="a4ce4184788d39e88a1d7bb724ba6fdcbb1f5a835f03ba78a9503748e1ec89e2" exitCode=0
Sep 17 17:44:05.589723 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:05.589676    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cblvb4g"
Sep 17 17:44:05.589723 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:05.589686    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cblvb4g" event={"ID":"d4160e4c-8713-4aa9-8b73-ded10a82e7aa","Type":"ContainerDied","Data":"a4ce4184788d39e88a1d7bb724ba6fdcbb1f5a835f03ba78a9503748e1ec89e2"}
Sep 17 17:44:05.590001 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:05.589736    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cblvb4g" event={"ID":"d4160e4c-8713-4aa9-8b73-ded10a82e7aa","Type":"ContainerDied","Data":"4c64e734580a53e4d565883c4f6ff0efcaebafdf61418aed84449bcd819e2866"}
Sep 17 17:44:05.590001 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:05.589759    2566 scope.go:117] "RemoveContainer" containerID="a4ce4184788d39e88a1d7bb724ba6fdcbb1f5a835f03ba78a9503748e1ec89e2"
Sep 17 17:44:05.598839 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:05.598812    2566 scope.go:117] "RemoveContainer" containerID="a4ce4184788d39e88a1d7bb724ba6fdcbb1f5a835f03ba78a9503748e1ec89e2"
Sep 17 17:44:05.599178 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:44:05.599155    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4ce4184788d39e88a1d7bb724ba6fdcbb1f5a835f03ba78a9503748e1ec89e2\": container with ID starting with a4ce4184788d39e88a1d7bb724ba6fdcbb1f5a835f03ba78a9503748e1ec89e2 not found: ID does not exist" containerID="a4ce4184788d39e88a1d7bb724ba6fdcbb1f5a835f03ba78a9503748e1ec89e2"
Sep 17 17:44:05.599235 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:05.599187    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4ce4184788d39e88a1d7bb724ba6fdcbb1f5a835f03ba78a9503748e1ec89e2"} err="failed to get container status \"a4ce4184788d39e88a1d7bb724ba6fdcbb1f5a835f03ba78a9503748e1ec89e2\": rpc error: code = NotFound desc = could not find container \"a4ce4184788d39e88a1d7bb724ba6fdcbb1f5a835f03ba78a9503748e1ec89e2\": container with ID starting with a4ce4184788d39e88a1d7bb724ba6fdcbb1f5a835f03ba78a9503748e1ec89e2 not found: ID does not exist"
Sep 17 17:44:05.617499 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:05.617465    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdnx6\" (UniqueName: \"kubernetes.io/projected/d4160e4c-8713-4aa9-8b73-ded10a82e7aa-kube-api-access-pdnx6\") pod \"d4160e4c-8713-4aa9-8b73-ded10a82e7aa\" (UID: \"d4160e4c-8713-4aa9-8b73-ded10a82e7aa\") "
Sep 17 17:44:05.617635 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:05.617516    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4160e4c-8713-4aa9-8b73-ded10a82e7aa-tls-certs\") pod \"d4160e4c-8713-4aa9-8b73-ded10a82e7aa\" (UID: \"d4160e4c-8713-4aa9-8b73-ded10a82e7aa\") "
Sep 17 17:44:05.620005 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:05.619973    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4160e4c-8713-4aa9-8b73-ded10a82e7aa-kube-api-access-pdnx6" (OuterVolumeSpecName: "kube-api-access-pdnx6") pod "d4160e4c-8713-4aa9-8b73-ded10a82e7aa" (UID: "d4160e4c-8713-4aa9-8b73-ded10a82e7aa"). InnerVolumeSpecName "kube-api-access-pdnx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:44:05.620005 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:05.619978    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4160e4c-8713-4aa9-8b73-ded10a82e7aa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d4160e4c-8713-4aa9-8b73-ded10a82e7aa" (UID: "d4160e4c-8713-4aa9-8b73-ded10a82e7aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:44:05.718278 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:05.718243    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdnx6\" (UniqueName: \"kubernetes.io/projected/d4160e4c-8713-4aa9-8b73-ded10a82e7aa-kube-api-access-pdnx6\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:44:05.718278 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:05.718274    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4160e4c-8713-4aa9-8b73-ded10a82e7aa-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:44:05.929093 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:05.929052    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cblvb4g"]
Sep 17 17:44:05.931760 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:05.931731    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cblvb4g"]
Sep 17 17:44:06.347511 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:06.347470    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4160e4c-8713-4aa9-8b73-ded10a82e7aa" path="/var/lib/kubelet/pods/d4160e4c-8713-4aa9-8b73-ded10a82e7aa/volumes"
Sep 17 17:44:25.681258 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:25.681224    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk"]
Sep 17 17:44:25.681838 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:25.681706    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4160e4c-8713-4aa9-8b73-ded10a82e7aa" containerName="main"
Sep 17 17:44:25.681838 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:25.681728    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4160e4c-8713-4aa9-8b73-ded10a82e7aa" containerName="main"
Sep 17 17:44:25.681838 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:25.681789    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4160e4c-8713-4aa9-8b73-ded10a82e7aa" containerName="main"
Sep 17 17:44:25.686229 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:25.686206    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk"
Sep 17 17:44:25.689450 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:25.689403    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 17 17:44:25.689450 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:25.689418    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-px72s\""
Sep 17 17:44:25.689450 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:25.689403    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 17 17:44:25.689703 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:25.689417    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 17 17:44:25.699335 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:25.699304    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk"]
Sep 17 17:44:25.776424 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:25.776384    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e16ada42-45d9-49bb-8884-270824de42bc-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk\" (UID: \"e16ada42-45d9-49bb-8884-270824de42bc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk"
Sep 17 17:44:25.776636 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:25.776483    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs5xp\" (UniqueName: \"kubernetes.io/projected/e16ada42-45d9-49bb-8884-270824de42bc-kube-api-access-gs5xp\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk\" (UID: \"e16ada42-45d9-49bb-8884-270824de42bc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk"
Sep 17 17:44:25.776636 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:25.776507    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e16ada42-45d9-49bb-8884-270824de42bc-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk\" (UID: \"e16ada42-45d9-49bb-8884-270824de42bc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk"
Sep 17 17:44:25.776636 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:25.776524    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e16ada42-45d9-49bb-8884-270824de42bc-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk\" (UID: \"e16ada42-45d9-49bb-8884-270824de42bc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk"
Sep 17 17:44:25.776636 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:25.776554    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e16ada42-45d9-49bb-8884-270824de42bc-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk\" (UID: \"e16ada42-45d9-49bb-8884-270824de42bc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk"
Sep 17 17:44:25.776636 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:25.776620    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e16ada42-45d9-49bb-8884-270824de42bc-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk\" (UID: \"e16ada42-45d9-49bb-8884-270824de42bc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk"
Sep 17 17:44:25.877245 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:25.877203    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e16ada42-45d9-49bb-8884-270824de42bc-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk\" (UID: \"e16ada42-45d9-49bb-8884-270824de42bc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk"
Sep 17 17:44:25.877462 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:25.877273    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e16ada42-45d9-49bb-8884-270824de42bc-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk\" (UID: \"e16ada42-45d9-49bb-8884-270824de42bc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk"
Sep 17 17:44:25.877462 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:25.877324    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs5xp\" (UniqueName: \"kubernetes.io/projected/e16ada42-45d9-49bb-8884-270824de42bc-kube-api-access-gs5xp\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk\" (UID: \"e16ada42-45d9-49bb-8884-270824de42bc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk"
Sep 17 17:44:25.877574 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:25.877478    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e16ada42-45d9-49bb-8884-270824de42bc-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk\" (UID: \"e16ada42-45d9-49bb-8884-270824de42bc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk"
Sep 17 17:44:25.877574 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:25.877516    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e16ada42-45d9-49bb-8884-270824de42bc-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk\" (UID: \"e16ada42-45d9-49bb-8884-270824de42bc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk"
Sep 17 17:44:25.877574 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:25.877555    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e16ada42-45d9-49bb-8884-270824de42bc-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk\" (UID: \"e16ada42-45d9-49bb-8884-270824de42bc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk"
Sep 17 17:44:25.877857 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:25.877788    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e16ada42-45d9-49bb-8884-270824de42bc-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk\" (UID: \"e16ada42-45d9-49bb-8884-270824de42bc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk"
Sep 17 17:44:25.878029 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:25.878006    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e16ada42-45d9-49bb-8884-270824de42bc-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk\" (UID: \"e16ada42-45d9-49bb-8884-270824de42bc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk"
Sep 17 17:44:25.878179 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:25.877855    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e16ada42-45d9-49bb-8884-270824de42bc-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk\" (UID: \"e16ada42-45d9-49bb-8884-270824de42bc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk"
Sep 17 17:44:25.879933 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:25.879911    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e16ada42-45d9-49bb-8884-270824de42bc-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk\" (UID: \"e16ada42-45d9-49bb-8884-270824de42bc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk"
Sep 17 17:44:25.880125 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:25.880104    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e16ada42-45d9-49bb-8884-270824de42bc-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk\" (UID: \"e16ada42-45d9-49bb-8884-270824de42bc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk"
Sep 17 17:44:25.888520 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:25.888496    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs5xp\" (UniqueName: \"kubernetes.io/projected/e16ada42-45d9-49bb-8884-270824de42bc-kube-api-access-gs5xp\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk\" (UID: \"e16ada42-45d9-49bb-8884-270824de42bc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk"
Sep 17 17:44:25.996852 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:25.996764    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk"
Sep 17 17:44:26.277172 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:26.277141    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk"]
Sep 17 17:44:26.278944 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:44:26.278915    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode16ada42_45d9_49bb_8884_270824de42bc.slice/crio-3d20797f46f9efe3d8bf45103873bfab54d599a652375c6df94c0d11872d7f05 WatchSource:0}: Error finding container 3d20797f46f9efe3d8bf45103873bfab54d599a652375c6df94c0d11872d7f05: Status 404 returned error can't find the container with id 3d20797f46f9efe3d8bf45103873bfab54d599a652375c6df94c0d11872d7f05
Sep 17 17:44:26.696016 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:26.695971    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk" event={"ID":"e16ada42-45d9-49bb-8884-270824de42bc","Type":"ContainerStarted","Data":"ff5cb33a8bfa2a4b527e95271e787c9e29d296f0d2de8c6b0f88dbf82c772c18"}
Sep 17 17:44:26.696461 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:26.696040    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk" event={"ID":"e16ada42-45d9-49bb-8884-270824de42bc","Type":"ContainerStarted","Data":"3d20797f46f9efe3d8bf45103873bfab54d599a652375c6df94c0d11872d7f05"}
Sep 17 17:44:26.739831 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:26.739769    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk" podStartSLOduration=1.739751518 podStartE2EDuration="1.739751518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:44:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:44:26.739580297 +0000 UTC m=+3752.634059436" watchObservedRunningTime="2026-09-17 17:44:26.739751518 +0000 UTC m=+3752.634230656"
Sep 17 17:44:26.997719 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:26.997609    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk"
Sep 17 17:44:29.558228 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:29.558190    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cr6sh"
Sep 17 17:44:35.997676 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:35.997559    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk"
Sep 17 17:44:36.010368 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:36.010327    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk"
Sep 17 17:44:36.758859 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:44:36.758818    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk"
Sep 17 17:45:00.234004 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:45:00.233955    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk"]
Sep 17 17:45:00.249070 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:45:00.249028    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:45:00.249070 ip-10-0-141-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:45:00.249070 ip-10-0-141-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:45:00.249070 ip-10-0-141-66 kubenswrapper[2566]:  > containerID="ff5cb33a8bfa2a4b527e95271e787c9e29d296f0d2de8c6b0f88dbf82c772c18" cmd=["/bin/sleep","15"]
Sep 17 17:45:00.249359 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:45:00.249086    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk" message="" err=<
Sep 17 17:45:00.249359 ip-10-0-141-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:45:00.249359 ip-10-0-141-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:45:00.249359 ip-10-0-141-66 kubenswrapper[2566]:  >
Sep 17 17:45:00.249359 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:45:00.249098    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:45:00.249359 ip-10-0-141-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:45:00.249359 ip-10-0-141-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:45:00.249359 ip-10-0-141-66 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk" podUID="e16ada42-45d9-49bb-8884-270824de42bc" containerName="main" containerID="cri-o://ff5cb33a8bfa2a4b527e95271e787c9e29d296f0d2de8c6b0f88dbf82c772c18"
Sep 17 17:45:00.249359 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:45:00.249145    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk" podUID="e16ada42-45d9-49bb-8884-270824de42bc" containerName="main" containerID="cri-o://ff5cb33a8bfa2a4b527e95271e787c9e29d296f0d2de8c6b0f88dbf82c772c18" gracePeriod=60
Sep 17 17:45:00.602734 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:45:00.602704    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk"
Sep 17 17:45:00.644658 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:45:00.644623    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e16ada42-45d9-49bb-8884-270824de42bc-dshm\") pod \"e16ada42-45d9-49bb-8884-270824de42bc\" (UID: \"e16ada42-45d9-49bb-8884-270824de42bc\") "
Sep 17 17:45:00.644846 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:45:00.644683    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e16ada42-45d9-49bb-8884-270824de42bc-tmp-dir\") pod \"e16ada42-45d9-49bb-8884-270824de42bc\" (UID: \"e16ada42-45d9-49bb-8884-270824de42bc\") "
Sep 17 17:45:00.644846 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:45:00.644721    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gs5xp\" (UniqueName: \"kubernetes.io/projected/e16ada42-45d9-49bb-8884-270824de42bc-kube-api-access-gs5xp\") pod \"e16ada42-45d9-49bb-8884-270824de42bc\" (UID: \"e16ada42-45d9-49bb-8884-270824de42bc\") "
Sep 17 17:45:00.644846 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:45:00.644743    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e16ada42-45d9-49bb-8884-270824de42bc-tls-certs\") pod \"e16ada42-45d9-49bb-8884-270824de42bc\" (UID: \"e16ada42-45d9-49bb-8884-270824de42bc\") "
Sep 17 17:45:00.644846 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:45:00.644776    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e16ada42-45d9-49bb-8884-270824de42bc-model-cache\") pod \"e16ada42-45d9-49bb-8884-270824de42bc\" (UID: \"e16ada42-45d9-49bb-8884-270824de42bc\") "
Sep 17 17:45:00.644846 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:45:00.644798    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e16ada42-45d9-49bb-8884-270824de42bc-home\") pod \"e16ada42-45d9-49bb-8884-270824de42bc\" (UID: \"e16ada42-45d9-49bb-8884-270824de42bc\") "
Sep 17 17:45:00.645128 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:45:00.645018    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e16ada42-45d9-49bb-8884-270824de42bc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e16ada42-45d9-49bb-8884-270824de42bc" (UID: "e16ada42-45d9-49bb-8884-270824de42bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:45:00.645187 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:45:00.645140    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e16ada42-45d9-49bb-8884-270824de42bc-home" (OuterVolumeSpecName: "home") pod "e16ada42-45d9-49bb-8884-270824de42bc" (UID: "e16ada42-45d9-49bb-8884-270824de42bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:45:00.645284 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:45:00.645252    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e16ada42-45d9-49bb-8884-270824de42bc-model-cache" (OuterVolumeSpecName: "model-cache") pod "e16ada42-45d9-49bb-8884-270824de42bc" (UID: "e16ada42-45d9-49bb-8884-270824de42bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:45:00.647391 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:45:00.647364    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e16ada42-45d9-49bb-8884-270824de42bc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e16ada42-45d9-49bb-8884-270824de42bc" (UID: "e16ada42-45d9-49bb-8884-270824de42bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:45:00.647591 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:45:00.647574    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e16ada42-45d9-49bb-8884-270824de42bc-dshm" (OuterVolumeSpecName: "dshm") pod "e16ada42-45d9-49bb-8884-270824de42bc" (UID: "e16ada42-45d9-49bb-8884-270824de42bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:45:00.647658 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:45:00.647588    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e16ada42-45d9-49bb-8884-270824de42bc-kube-api-access-gs5xp" (OuterVolumeSpecName: "kube-api-access-gs5xp") pod "e16ada42-45d9-49bb-8884-270824de42bc" (UID: "e16ada42-45d9-49bb-8884-270824de42bc"). InnerVolumeSpecName "kube-api-access-gs5xp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:45:00.745914 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:45:00.745869    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e16ada42-45d9-49bb-8884-270824de42bc-dshm\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:45:00.745914 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:45:00.745906    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e16ada42-45d9-49bb-8884-270824de42bc-tmp-dir\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:45:00.745914 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:45:00.745916    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs5xp\" (UniqueName: \"kubernetes.io/projected/e16ada42-45d9-49bb-8884-270824de42bc-kube-api-access-gs5xp\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:45:00.745914 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:45:00.745926    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e16ada42-45d9-49bb-8884-270824de42bc-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:45:00.746199 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:45:00.745936    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e16ada42-45d9-49bb-8884-270824de42bc-model-cache\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:45:00.746199 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:45:00.745942    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e16ada42-45d9-49bb-8884-270824de42bc-home\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:45:00.874922 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:45:00.874883    2566 generic.go:356] "Generic (PLEG): container finished" podID="e16ada42-45d9-49bb-8884-270824de42bc" containerID="ff5cb33a8bfa2a4b527e95271e787c9e29d296f0d2de8c6b0f88dbf82c772c18" exitCode=0
Sep 17 17:45:00.875142 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:45:00.874928    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk" event={"ID":"e16ada42-45d9-49bb-8884-270824de42bc","Type":"ContainerDied","Data":"ff5cb33a8bfa2a4b527e95271e787c9e29d296f0d2de8c6b0f88dbf82c772c18"}
Sep 17 17:45:00.875142 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:45:00.874979    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk" event={"ID":"e16ada42-45d9-49bb-8884-270824de42bc","Type":"ContainerDied","Data":"3d20797f46f9efe3d8bf45103873bfab54d599a652375c6df94c0d11872d7f05"}
Sep 17 17:45:00.875142 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:45:00.875010    2566 scope.go:117] "RemoveContainer" containerID="ff5cb33a8bfa2a4b527e95271e787c9e29d296f0d2de8c6b0f88dbf82c772c18"
Sep 17 17:45:00.875142 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:45:00.874933    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk"
Sep 17 17:45:00.885592 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:45:00.885571    2566 scope.go:117] "RemoveContainer" containerID="ff5cb33a8bfa2a4b527e95271e787c9e29d296f0d2de8c6b0f88dbf82c772c18"
Sep 17 17:45:00.885910 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:45:00.885889    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff5cb33a8bfa2a4b527e95271e787c9e29d296f0d2de8c6b0f88dbf82c772c18\": container with ID starting with ff5cb33a8bfa2a4b527e95271e787c9e29d296f0d2de8c6b0f88dbf82c772c18 not found: ID does not exist" containerID="ff5cb33a8bfa2a4b527e95271e787c9e29d296f0d2de8c6b0f88dbf82c772c18"
Sep 17 17:45:00.885984 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:45:00.885924    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff5cb33a8bfa2a4b527e95271e787c9e29d296f0d2de8c6b0f88dbf82c772c18"} err="failed to get container status \"ff5cb33a8bfa2a4b527e95271e787c9e29d296f0d2de8c6b0f88dbf82c772c18\": rpc error: code = NotFound desc = could not find container \"ff5cb33a8bfa2a4b527e95271e787c9e29d296f0d2de8c6b0f88dbf82c772c18\": container with ID starting with ff5cb33a8bfa2a4b527e95271e787c9e29d296f0d2de8c6b0f88dbf82c772c18 not found: ID does not exist"
Sep 17 17:45:00.913513 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:45:00.913473    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk"]
Sep 17 17:45:00.916718 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:45:00.916680    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fp8zhk"]
Sep 17 17:45:02.346824 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:45:02.346784    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e16ada42-45d9-49bb-8884-270824de42bc" path="/var/lib/kubelet/pods/e16ada42-45d9-49bb-8884-270824de42bc/volumes"
Sep 17 17:46:09.442990 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:09.442891    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68"]
Sep 17 17:46:09.443512 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:09.443383    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e16ada42-45d9-49bb-8884-270824de42bc" containerName="main"
Sep 17 17:46:09.443512 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:09.443402    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e16ada42-45d9-49bb-8884-270824de42bc" containerName="main"
Sep 17 17:46:09.443641 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:09.443521    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e16ada42-45d9-49bb-8884-270824de42bc" containerName="main"
Sep 17 17:46:09.446569 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:09.446546    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68"
Sep 17 17:46:09.450072 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:09.450047    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-nwhht\""
Sep 17 17:46:09.450543 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:09.450527    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 17 17:46:09.450796 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:09.450778    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 17 17:46:09.450880 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:09.450847    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 17 17:46:09.468005 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:09.467973    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68"]
Sep 17 17:46:09.483624 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:09.483587    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68\" (UID: \"c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68"
Sep 17 17:46:09.483624 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:09.483629    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68\" (UID: \"c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68"
Sep 17 17:46:09.483815 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:09.483662    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd4tl\" (UniqueName: \"kubernetes.io/projected/c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4-kube-api-access-nd4tl\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68\" (UID: \"c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68"
Sep 17 17:46:09.483815 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:09.483697    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68\" (UID: \"c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68"
Sep 17 17:46:09.483815 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:09.483725    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68\" (UID: \"c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68"
Sep 17 17:46:09.483815 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:09.483740    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68\" (UID: \"c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68"
Sep 17 17:46:09.584808 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:09.584768    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68\" (UID: \"c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68"
Sep 17 17:46:09.585017 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:09.584814    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68\" (UID: \"c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68"
Sep 17 17:46:09.585017 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:09.584847    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd4tl\" (UniqueName: \"kubernetes.io/projected/c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4-kube-api-access-nd4tl\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68\" (UID: \"c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68"
Sep 17 17:46:09.585017 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:09.584877    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68\" (UID: \"c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68"
Sep 17 17:46:09.585017 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:09.584907    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68\" (UID: \"c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68"
Sep 17 17:46:09.585017 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:09.584922    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68\" (UID: \"c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68"
Sep 17 17:46:09.585332 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:09.585245    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68\" (UID: \"c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68"
Sep 17 17:46:09.585332 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:09.585294    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68\" (UID: \"c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68"
Sep 17 17:46:09.585332 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:09.585327    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68\" (UID: \"c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68"
Sep 17 17:46:09.587482 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:09.587460    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68\" (UID: \"c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68"
Sep 17 17:46:09.587655 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:09.587637    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68\" (UID: \"c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68"
Sep 17 17:46:09.597319 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:09.597289    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd4tl\" (UniqueName: \"kubernetes.io/projected/c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4-kube-api-access-nd4tl\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68\" (UID: \"c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68"
Sep 17 17:46:09.757359 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:09.757257    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68"
Sep 17 17:46:10.044584 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:10.044461    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68"]
Sep 17 17:46:10.047496 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:46:10.047461    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc60eb0dc_57ef_4d1d_aebe_ca83ed9268e4.slice/crio-a4f87464a1e704547cd2211277c17208d143515f85fcd280d6bf9308f3e6db1e WatchSource:0}: Error finding container a4f87464a1e704547cd2211277c17208d143515f85fcd280d6bf9308f3e6db1e: Status 404 returned error can't find the container with id a4f87464a1e704547cd2211277c17208d143515f85fcd280d6bf9308f3e6db1e
Sep 17 17:46:10.225112 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:10.225063    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68" event={"ID":"c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4","Type":"ContainerStarted","Data":"e1b9598e101072de07aec353ca5a4d68879f8565d544576786cda8681d1a1150"}
Sep 17 17:46:10.225326 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:10.225126    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68" event={"ID":"c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4","Type":"ContainerStarted","Data":"a4f87464a1e704547cd2211277c17208d143515f85fcd280d6bf9308f3e6db1e"}
Sep 17 17:46:10.252202 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:10.252072    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68" podStartSLOduration=1.252041642 podStartE2EDuration="1.252041642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:46:10.251108315 +0000 UTC m=+3856.145587453" watchObservedRunningTime="2026-09-17 17:46:10.252041642 +0000 UTC m=+3856.146520782"
Sep 17 17:46:10.758024 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:10.757986    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68"
Sep 17 17:46:19.757602 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:19.757549    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68"
Sep 17 17:46:19.770201 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:19.770166    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68"
Sep 17 17:46:20.293776 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:20.293740    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68"
Sep 17 17:46:31.499305 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:31.499265    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cr6sh"]
Sep 17 17:46:39.557868 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:39.557826    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cr6sh" podUID="f5458154-d36a-404a-9449-8c79d64a6cf9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:46:44.800155 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:44.800111    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68"]
Sep 17 17:46:44.809953 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:46:44.809907    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:46:44.809953 ip-10-0-141-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:46:44.809953 ip-10-0-141-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:46:44.809953 ip-10-0-141-66 kubenswrapper[2566]:  > containerID="e1b9598e101072de07aec353ca5a4d68879f8565d544576786cda8681d1a1150" cmd=["/bin/sleep","15"]
Sep 17 17:46:44.810344 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:44.809962    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68" message="" err=<
Sep 17 17:46:44.810344 ip-10-0-141-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:46:44.810344 ip-10-0-141-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:46:44.810344 ip-10-0-141-66 kubenswrapper[2566]:  >
Sep 17 17:46:44.810344 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:46:44.809977    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:46:44.810344 ip-10-0-141-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:46:44.810344 ip-10-0-141-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:46:44.810344 ip-10-0-141-66 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68" podUID="c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4" containerName="main" containerID="cri-o://e1b9598e101072de07aec353ca5a4d68879f8565d544576786cda8681d1a1150"
Sep 17 17:46:44.810344 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:44.810018    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68" podUID="c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4" containerName="main" containerID="cri-o://e1b9598e101072de07aec353ca5a4d68879f8565d544576786cda8681d1a1150" gracePeriod=60
Sep 17 17:46:45.164134 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:45.164106    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68"
Sep 17 17:46:45.287386 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:45.287345    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4-dshm\") pod \"c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4\" (UID: \"c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4\") "
Sep 17 17:46:45.287386 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:45.287395    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4-tls-certs\") pod \"c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4\" (UID: \"c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4\") "
Sep 17 17:46:45.287648 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:45.287480    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4-home\") pod \"c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4\" (UID: \"c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4\") "
Sep 17 17:46:45.287648 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:45.287506    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nd4tl\" (UniqueName: \"kubernetes.io/projected/c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4-kube-api-access-nd4tl\") pod \"c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4\" (UID: \"c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4\") "
Sep 17 17:46:45.287648 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:45.287546    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4-model-cache\") pod \"c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4\" (UID: \"c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4\") "
Sep 17 17:46:45.287648 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:45.287589    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4-tmp-dir\") pod \"c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4\" (UID: \"c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4\") "
Sep 17 17:46:45.287863 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:45.287735    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4-home" (OuterVolumeSpecName: "home") pod "c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4" (UID: "c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:46:45.287863 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:45.287807    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4-home\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:46:45.287968 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:45.287920    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4-model-cache" (OuterVolumeSpecName: "model-cache") pod "c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4" (UID: "c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:46:45.287968 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:45.287935    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4" (UID: "c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:46:45.290002 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:45.289979    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4-dshm" (OuterVolumeSpecName: "dshm") pod "c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4" (UID: "c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:46:45.290098 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:45.289999    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4-kube-api-access-nd4tl" (OuterVolumeSpecName: "kube-api-access-nd4tl") pod "c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4" (UID: "c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4"). InnerVolumeSpecName "kube-api-access-nd4tl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:46:45.290098 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:45.290000    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4" (UID: "c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:46:45.388352 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:45.388254    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:46:45.388352 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:45.388290    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd4tl\" (UniqueName: \"kubernetes.io/projected/c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4-kube-api-access-nd4tl\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:46:45.388352 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:45.388302    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4-model-cache\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:46:45.388352 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:45.388311    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4-tmp-dir\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:46:45.388352 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:45.388319    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4-dshm\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:46:45.420649 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:45.420615    2566 generic.go:356] "Generic (PLEG): container finished" podID="c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4" containerID="e1b9598e101072de07aec353ca5a4d68879f8565d544576786cda8681d1a1150" exitCode=0
Sep 17 17:46:45.420844 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:45.420660    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68"
Sep 17 17:46:45.420844 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:45.420668    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68" event={"ID":"c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4","Type":"ContainerDied","Data":"e1b9598e101072de07aec353ca5a4d68879f8565d544576786cda8681d1a1150"}
Sep 17 17:46:45.420844 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:45.420721    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68" event={"ID":"c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4","Type":"ContainerDied","Data":"a4f87464a1e704547cd2211277c17208d143515f85fcd280d6bf9308f3e6db1e"}
Sep 17 17:46:45.420844 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:45.420748    2566 scope.go:117] "RemoveContainer" containerID="e1b9598e101072de07aec353ca5a4d68879f8565d544576786cda8681d1a1150"
Sep 17 17:46:45.430620 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:45.430603    2566 scope.go:117] "RemoveContainer" containerID="e1b9598e101072de07aec353ca5a4d68879f8565d544576786cda8681d1a1150"
Sep 17 17:46:45.430890 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:46:45.430872    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1b9598e101072de07aec353ca5a4d68879f8565d544576786cda8681d1a1150\": container with ID starting with e1b9598e101072de07aec353ca5a4d68879f8565d544576786cda8681d1a1150 not found: ID does not exist" containerID="e1b9598e101072de07aec353ca5a4d68879f8565d544576786cda8681d1a1150"
Sep 17 17:46:45.430964 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:45.430898    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1b9598e101072de07aec353ca5a4d68879f8565d544576786cda8681d1a1150"} err="failed to get container status \"e1b9598e101072de07aec353ca5a4d68879f8565d544576786cda8681d1a1150\": rpc error: code = NotFound desc = could not find container \"e1b9598e101072de07aec353ca5a4d68879f8565d544576786cda8681d1a1150\": container with ID starting with e1b9598e101072de07aec353ca5a4d68879f8565d544576786cda8681d1a1150 not found: ID does not exist"
Sep 17 17:46:45.448300 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:45.448266    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68"]
Sep 17 17:46:45.454997 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:45.454969    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-wkz68"]
Sep 17 17:46:46.347393 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:46.347349    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4" path="/var/lib/kubelet/pods/c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4/volumes"
Sep 17 17:46:46.519370 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:46.519311    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cr6sh" podUID="f5458154-d36a-404a-9449-8c79d64a6cf9" containerName="main" containerID="cri-o://31cf1046e7199ec78a32f1fdef93fb6b2b1bc2b039a085b1ee6dbf4ca2774455" gracePeriod=45
Sep 17 17:46:49.557493 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:49.557429    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cr6sh" podUID="f5458154-d36a-404a-9449-8c79d64a6cf9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:46:56.675246 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:56.675216    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gblhr_91477834-6648-4e75-8530-e01bdee5e57e/ovn-acl-logging/0.log"
Sep 17 17:46:56.694854 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:56.694820    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gblhr_91477834-6648-4e75-8530-e01bdee5e57e/ovn-acl-logging/0.log"
Sep 17 17:46:59.557290 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:59.557249    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cr6sh" podUID="f5458154-d36a-404a-9449-8c79d64a6cf9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:46:59.557706 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:46:59.557362    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cr6sh"
Sep 17 17:47:09.557875 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:09.557828    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cr6sh" podUID="f5458154-d36a-404a-9449-8c79d64a6cf9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:47:16.774619 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:16.774587    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cr6sh"
Sep 17 17:47:16.825081 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:16.825042    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5458154-d36a-404a-9449-8c79d64a6cf9-tls-certs\") pod \"f5458154-d36a-404a-9449-8c79d64a6cf9\" (UID: \"f5458154-d36a-404a-9449-8c79d64a6cf9\") "
Sep 17 17:47:16.825081 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:16.825086    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8crd\" (UniqueName: \"kubernetes.io/projected/f5458154-d36a-404a-9449-8c79d64a6cf9-kube-api-access-m8crd\") pod \"f5458154-d36a-404a-9449-8c79d64a6cf9\" (UID: \"f5458154-d36a-404a-9449-8c79d64a6cf9\") "
Sep 17 17:47:16.827625 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:16.827595    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5458154-d36a-404a-9449-8c79d64a6cf9-kube-api-access-m8crd" (OuterVolumeSpecName: "kube-api-access-m8crd") pod "f5458154-d36a-404a-9449-8c79d64a6cf9" (UID: "f5458154-d36a-404a-9449-8c79d64a6cf9"). InnerVolumeSpecName "kube-api-access-m8crd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:47:16.827732 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:16.827595    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5458154-d36a-404a-9449-8c79d64a6cf9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f5458154-d36a-404a-9449-8c79d64a6cf9" (UID: "f5458154-d36a-404a-9449-8c79d64a6cf9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:47:16.926345 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:16.926308    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5458154-d36a-404a-9449-8c79d64a6cf9-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:47:16.926345 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:16.926339    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8crd\" (UniqueName: \"kubernetes.io/projected/f5458154-d36a-404a-9449-8c79d64a6cf9-kube-api-access-m8crd\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:47:17.583704 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:17.583669    2566 generic.go:356] "Generic (PLEG): container finished" podID="f5458154-d36a-404a-9449-8c79d64a6cf9" containerID="31cf1046e7199ec78a32f1fdef93fb6b2b1bc2b039a085b1ee6dbf4ca2774455" exitCode=0
Sep 17 17:47:17.583940 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:17.583739    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cr6sh"
Sep 17 17:47:17.583940 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:17.583750    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cr6sh" event={"ID":"f5458154-d36a-404a-9449-8c79d64a6cf9","Type":"ContainerDied","Data":"31cf1046e7199ec78a32f1fdef93fb6b2b1bc2b039a085b1ee6dbf4ca2774455"}
Sep 17 17:47:17.583940 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:17.583806    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cr6sh" event={"ID":"f5458154-d36a-404a-9449-8c79d64a6cf9","Type":"ContainerDied","Data":"339a92ed548fcef508c501f622f3536dd280cb193e09f0adbc6dc6d6a4ea5353"}
Sep 17 17:47:17.583940 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:17.583830    2566 scope.go:117] "RemoveContainer" containerID="31cf1046e7199ec78a32f1fdef93fb6b2b1bc2b039a085b1ee6dbf4ca2774455"
Sep 17 17:47:17.593275 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:17.593255    2566 scope.go:117] "RemoveContainer" containerID="31cf1046e7199ec78a32f1fdef93fb6b2b1bc2b039a085b1ee6dbf4ca2774455"
Sep 17 17:47:17.593578 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:47:17.593558    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31cf1046e7199ec78a32f1fdef93fb6b2b1bc2b039a085b1ee6dbf4ca2774455\": container with ID starting with 31cf1046e7199ec78a32f1fdef93fb6b2b1bc2b039a085b1ee6dbf4ca2774455 not found: ID does not exist" containerID="31cf1046e7199ec78a32f1fdef93fb6b2b1bc2b039a085b1ee6dbf4ca2774455"
Sep 17 17:47:17.593660 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:17.593587    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31cf1046e7199ec78a32f1fdef93fb6b2b1bc2b039a085b1ee6dbf4ca2774455"} err="failed to get container status \"31cf1046e7199ec78a32f1fdef93fb6b2b1bc2b039a085b1ee6dbf4ca2774455\": rpc error: code = NotFound desc = could not find container \"31cf1046e7199ec78a32f1fdef93fb6b2b1bc2b039a085b1ee6dbf4ca2774455\": container with ID starting with 31cf1046e7199ec78a32f1fdef93fb6b2b1bc2b039a085b1ee6dbf4ca2774455 not found: ID does not exist"
Sep 17 17:47:17.608700 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:17.608664    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cr6sh"]
Sep 17 17:47:17.614414 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:17.614388    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cr6sh"]
Sep 17 17:47:18.347964 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:18.347931    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5458154-d36a-404a-9449-8c79d64a6cf9" path="/var/lib/kubelet/pods/f5458154-d36a-404a-9449-8c79d64a6cf9/volumes"
Sep 17 17:47:27.421779 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.421742    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd"]
Sep 17 17:47:27.422279 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.422214    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4" containerName="main"
Sep 17 17:47:27.422279 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.422232    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4" containerName="main"
Sep 17 17:47:27.422279 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.422248    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5458154-d36a-404a-9449-8c79d64a6cf9" containerName="main"
Sep 17 17:47:27.422279 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.422255    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5458154-d36a-404a-9449-8c79d64a6cf9" containerName="main"
Sep 17 17:47:27.422535 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.422348    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c60eb0dc-57ef-4d1d-aebe-ca83ed9268e4" containerName="main"
Sep 17 17:47:27.422535 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.422362    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5458154-d36a-404a-9449-8c79d64a6cf9" containerName="main"
Sep 17 17:47:27.427032 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.427000    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd"
Sep 17 17:47:27.429919 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.429890    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 17 17:47:27.430082 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.429944    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 17 17:47:27.430082 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.429963    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:47:27.430215 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.430189    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-rsrk2\""
Sep 17 17:47:27.430297 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.430198    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 17 17:47:27.430426 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.430308    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 17 17:47:27.441595 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.441565    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd"]
Sep 17 17:47:27.509403 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.509364    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-model-cache\") pod \"prestop-hook-test-kserve-86565fd9b-dh5wd\" (UID: \"b0b1f752-3d88-4a0b-8ba0-ba63081d1164\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd"
Sep 17 17:47:27.509403 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.509406    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-home\") pod \"prestop-hook-test-kserve-86565fd9b-dh5wd\" (UID: \"b0b1f752-3d88-4a0b-8ba0-ba63081d1164\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd"
Sep 17 17:47:27.509647 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.509424    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-dshm\") pod \"prestop-hook-test-kserve-86565fd9b-dh5wd\" (UID: \"b0b1f752-3d88-4a0b-8ba0-ba63081d1164\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd"
Sep 17 17:47:27.509647 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.509475    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-tmp-dir\") pod \"prestop-hook-test-kserve-86565fd9b-dh5wd\" (UID: \"b0b1f752-3d88-4a0b-8ba0-ba63081d1164\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd"
Sep 17 17:47:27.509647 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.509532    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-kserve-provision-location\") pod \"prestop-hook-test-kserve-86565fd9b-dh5wd\" (UID: \"b0b1f752-3d88-4a0b-8ba0-ba63081d1164\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd"
Sep 17 17:47:27.509647 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.509602    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-cabundle-cert\") pod \"prestop-hook-test-kserve-86565fd9b-dh5wd\" (UID: \"b0b1f752-3d88-4a0b-8ba0-ba63081d1164\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd"
Sep 17 17:47:27.509647 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.509632    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgcqh\" (UniqueName: \"kubernetes.io/projected/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-kube-api-access-fgcqh\") pod \"prestop-hook-test-kserve-86565fd9b-dh5wd\" (UID: \"b0b1f752-3d88-4a0b-8ba0-ba63081d1164\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd"
Sep 17 17:47:27.509815 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.509659    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-tls-certs\") pod \"prestop-hook-test-kserve-86565fd9b-dh5wd\" (UID: \"b0b1f752-3d88-4a0b-8ba0-ba63081d1164\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd"
Sep 17 17:47:27.610773 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.610730    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-tls-certs\") pod \"prestop-hook-test-kserve-86565fd9b-dh5wd\" (UID: \"b0b1f752-3d88-4a0b-8ba0-ba63081d1164\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd"
Sep 17 17:47:27.610773 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.610774    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-model-cache\") pod \"prestop-hook-test-kserve-86565fd9b-dh5wd\" (UID: \"b0b1f752-3d88-4a0b-8ba0-ba63081d1164\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd"
Sep 17 17:47:27.611049 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.610797    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-home\") pod \"prestop-hook-test-kserve-86565fd9b-dh5wd\" (UID: \"b0b1f752-3d88-4a0b-8ba0-ba63081d1164\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd"
Sep 17 17:47:27.611049 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.610813    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-dshm\") pod \"prestop-hook-test-kserve-86565fd9b-dh5wd\" (UID: \"b0b1f752-3d88-4a0b-8ba0-ba63081d1164\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd"
Sep 17 17:47:27.611049 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.610831    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-tmp-dir\") pod \"prestop-hook-test-kserve-86565fd9b-dh5wd\" (UID: \"b0b1f752-3d88-4a0b-8ba0-ba63081d1164\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd"
Sep 17 17:47:27.611049 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.610867    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-kserve-provision-location\") pod \"prestop-hook-test-kserve-86565fd9b-dh5wd\" (UID: \"b0b1f752-3d88-4a0b-8ba0-ba63081d1164\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd"
Sep 17 17:47:27.611049 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.610909    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-cabundle-cert\") pod \"prestop-hook-test-kserve-86565fd9b-dh5wd\" (UID: \"b0b1f752-3d88-4a0b-8ba0-ba63081d1164\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd"
Sep 17 17:47:27.611049 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.610924    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgcqh\" (UniqueName: \"kubernetes.io/projected/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-kube-api-access-fgcqh\") pod \"prestop-hook-test-kserve-86565fd9b-dh5wd\" (UID: \"b0b1f752-3d88-4a0b-8ba0-ba63081d1164\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd"
Sep 17 17:47:27.611391 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.611247    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-model-cache\") pod \"prestop-hook-test-kserve-86565fd9b-dh5wd\" (UID: \"b0b1f752-3d88-4a0b-8ba0-ba63081d1164\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd"
Sep 17 17:47:27.611495 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.611380    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-home\") pod \"prestop-hook-test-kserve-86565fd9b-dh5wd\" (UID: \"b0b1f752-3d88-4a0b-8ba0-ba63081d1164\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd"
Sep 17 17:47:27.611495 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.611406    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-kserve-provision-location\") pod \"prestop-hook-test-kserve-86565fd9b-dh5wd\" (UID: \"b0b1f752-3d88-4a0b-8ba0-ba63081d1164\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd"
Sep 17 17:47:27.611618 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.611485    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-tmp-dir\") pod \"prestop-hook-test-kserve-86565fd9b-dh5wd\" (UID: \"b0b1f752-3d88-4a0b-8ba0-ba63081d1164\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd"
Sep 17 17:47:27.611768 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.611746    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-cabundle-cert\") pod \"prestop-hook-test-kserve-86565fd9b-dh5wd\" (UID: \"b0b1f752-3d88-4a0b-8ba0-ba63081d1164\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd"
Sep 17 17:47:27.613500 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.613482    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-dshm\") pod \"prestop-hook-test-kserve-86565fd9b-dh5wd\" (UID: \"b0b1f752-3d88-4a0b-8ba0-ba63081d1164\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd"
Sep 17 17:47:27.613668 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.613653    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-tls-certs\") pod \"prestop-hook-test-kserve-86565fd9b-dh5wd\" (UID: \"b0b1f752-3d88-4a0b-8ba0-ba63081d1164\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd"
Sep 17 17:47:27.621879 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.621855    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgcqh\" (UniqueName: \"kubernetes.io/projected/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-kube-api-access-fgcqh\") pod \"prestop-hook-test-kserve-86565fd9b-dh5wd\" (UID: \"b0b1f752-3d88-4a0b-8ba0-ba63081d1164\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd"
Sep 17 17:47:27.684599 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.684497    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-kxhkp"]
Sep 17 17:47:27.689527 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.689499    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-kxhkp"
Sep 17 17:47:27.692454 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.692412    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-epp-sa-dockercfg-d2r4m\""
Sep 17 17:47:27.698145 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.698117    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-kxhkp"]
Sep 17 17:47:27.739648 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.739616    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd"
Sep 17 17:47:27.811929 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.811893    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x95sf\" (UniqueName: \"kubernetes.io/projected/ed85b818-72f4-42d5-999d-a37470f23e19-kube-api-access-x95sf\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-kxhkp\" (UID: \"ed85b818-72f4-42d5-999d-a37470f23e19\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-kxhkp"
Sep 17 17:47:27.812106 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.811936    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed85b818-72f4-42d5-999d-a37470f23e19-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-kxhkp\" (UID: \"ed85b818-72f4-42d5-999d-a37470f23e19\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-kxhkp"
Sep 17 17:47:27.913188 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.913146    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x95sf\" (UniqueName: \"kubernetes.io/projected/ed85b818-72f4-42d5-999d-a37470f23e19-kube-api-access-x95sf\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-kxhkp\" (UID: \"ed85b818-72f4-42d5-999d-a37470f23e19\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-kxhkp"
Sep 17 17:47:27.913188 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.913192    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed85b818-72f4-42d5-999d-a37470f23e19-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-kxhkp\" (UID: \"ed85b818-72f4-42d5-999d-a37470f23e19\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-kxhkp"
Sep 17 17:47:27.915953 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.915930    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed85b818-72f4-42d5-999d-a37470f23e19-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-kxhkp\" (UID: \"ed85b818-72f4-42d5-999d-a37470f23e19\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-kxhkp"
Sep 17 17:47:27.923613 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:27.923583    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x95sf\" (UniqueName: \"kubernetes.io/projected/ed85b818-72f4-42d5-999d-a37470f23e19-kube-api-access-x95sf\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-kxhkp\" (UID: \"ed85b818-72f4-42d5-999d-a37470f23e19\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-kxhkp"
Sep 17 17:47:28.002791 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:28.002711    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-kxhkp"
Sep 17 17:47:28.028236 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:28.028203    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd"]
Sep 17 17:47:28.030285 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:47:28.030256    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0b1f752_3d88_4a0b_8ba0_ba63081d1164.slice/crio-2792486ff9d4a58d9dfa172d2fd23d627a13924439a9cb545e6253cceb65a525 WatchSource:0}: Error finding container 2792486ff9d4a58d9dfa172d2fd23d627a13924439a9cb545e6253cceb65a525: Status 404 returned error can't find the container with id 2792486ff9d4a58d9dfa172d2fd23d627a13924439a9cb545e6253cceb65a525
Sep 17 17:47:28.297242 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:28.297211    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-kxhkp"]
Sep 17 17:47:28.298759 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:47:28.298714    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded85b818_72f4_42d5_999d_a37470f23e19.slice/crio-db900f28c554289678f8f63fb70cfb31c12412b582a993f61e771642d57c4112 WatchSource:0}: Error finding container db900f28c554289678f8f63fb70cfb31c12412b582a993f61e771642d57c4112: Status 404 returned error can't find the container with id db900f28c554289678f8f63fb70cfb31c12412b582a993f61e771642d57c4112
Sep 17 17:47:28.640800 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:28.640684    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd" event={"ID":"b0b1f752-3d88-4a0b-8ba0-ba63081d1164","Type":"ContainerStarted","Data":"e1df344bd5b451f85fd25caa60c33040d824b98c9f871ed1b307585ecea0998d"}
Sep 17 17:47:28.640800 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:28.640762    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd" event={"ID":"b0b1f752-3d88-4a0b-8ba0-ba63081d1164","Type":"ContainerStarted","Data":"2792486ff9d4a58d9dfa172d2fd23d627a13924439a9cb545e6253cceb65a525"}
Sep 17 17:47:28.642493 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:28.642460    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-kxhkp" event={"ID":"ed85b818-72f4-42d5-999d-a37470f23e19","Type":"ContainerStarted","Data":"0f61e26eda060acd042bfd30eeb0d776e71dd77067a7fde79d7f8c4a7cbdcc02"}
Sep 17 17:47:28.642691 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:28.642510    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-kxhkp" event={"ID":"ed85b818-72f4-42d5-999d-a37470f23e19","Type":"ContainerStarted","Data":"db900f28c554289678f8f63fb70cfb31c12412b582a993f61e771642d57c4112"}
Sep 17 17:47:28.642691 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:28.642531    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-kxhkp"
Sep 17 17:47:28.703252 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:28.703172    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-kxhkp" podStartSLOduration=1.703147209 podStartE2EDuration="1.703147209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:47:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:47:28.699544789 +0000 UTC m=+3934.594023930" watchObservedRunningTime="2026-09-17 17:47:28.703147209 +0000 UTC m=+3934.597626349"
Sep 17 17:47:30.655084 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:30.655045    2566 generic.go:356] "Generic (PLEG): container finished" podID="b0b1f752-3d88-4a0b-8ba0-ba63081d1164" containerID="e1df344bd5b451f85fd25caa60c33040d824b98c9f871ed1b307585ecea0998d" exitCode=0
Sep 17 17:47:30.655532 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:30.655123    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd" event={"ID":"b0b1f752-3d88-4a0b-8ba0-ba63081d1164","Type":"ContainerDied","Data":"e1df344bd5b451f85fd25caa60c33040d824b98c9f871ed1b307585ecea0998d"}
Sep 17 17:47:31.663244 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:31.663198    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd" event={"ID":"b0b1f752-3d88-4a0b-8ba0-ba63081d1164","Type":"ContainerStarted","Data":"7504b7a2186d6e32d2c1048386d43e5ec0f7ad86d2e5a8c64251622e216d3370"}
Sep 17 17:47:31.693883 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:31.693805    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd" podStartSLOduration=2.38425344 podStartE2EDuration="4.693778888s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.309525448s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:47:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:47:31.689095888 +0000 UTC m=+3937.583575034" watchObservedRunningTime="2026-09-17 17:47:31.693778888 +0000 UTC m=+3937.588258038"
Sep 17 17:47:31.740236 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:31.740193    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd"
Sep 17 17:47:37.740765 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:37.740671    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd"
Sep 17 17:47:37.742354 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:37.742324    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd" podUID="b0b1f752-3d88-4a0b-8ba0-ba63081d1164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": dial tcp 10.133.0.85:8000: connect: connection refused"
Sep 17 17:47:47.740400 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:47.740338    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd" podUID="b0b1f752-3d88-4a0b-8ba0-ba63081d1164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": dial tcp 10.133.0.85:8000: connect: connection refused"
Sep 17 17:47:57.740428 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:57.740363    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd" podUID="b0b1f752-3d88-4a0b-8ba0-ba63081d1164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": dial tcp 10.133.0.85:8000: connect: connection refused"
Sep 17 17:47:59.650548 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:47:59.650512    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-kxhkp"
Sep 17 17:48:07.740791 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:07.740734    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd" podUID="b0b1f752-3d88-4a0b-8ba0-ba63081d1164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": dial tcp 10.133.0.85:8000: connect: connection refused"
Sep 17 17:48:17.740907 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:17.740849    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd" podUID="b0b1f752-3d88-4a0b-8ba0-ba63081d1164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": dial tcp 10.133.0.85:8000: connect: connection refused"
Sep 17 17:48:27.740186 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:27.740122    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd" podUID="b0b1f752-3d88-4a0b-8ba0-ba63081d1164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": dial tcp 10.133.0.85:8000: connect: connection refused"
Sep 17 17:48:37.741029 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:37.740982    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd" podUID="b0b1f752-3d88-4a0b-8ba0-ba63081d1164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": dial tcp 10.133.0.85:8000: connect: connection refused"
Sep 17 17:48:44.732902 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:44.732857    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dgsz9"]
Sep 17 17:48:44.736851 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:44.736827    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dgsz9"
Sep 17 17:48:44.740305 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:44.740276    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 17 17:48:44.741122 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:44.741100    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-pjwpj\""
Sep 17 17:48:44.741220 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:44.741116    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 17 17:48:44.741220 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:44.741100    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 17 17:48:44.749578 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:44.749548    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dgsz9"]
Sep 17 17:48:44.882714 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:44.882673    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d604d4d-7439-49b9-adc3-31f853df502b-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-dgsz9\" (UID: \"1d604d4d-7439-49b9-adc3-31f853df502b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dgsz9"
Sep 17 17:48:44.882714 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:44.882721    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d604d4d-7439-49b9-adc3-31f853df502b-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-dgsz9\" (UID: \"1d604d4d-7439-49b9-adc3-31f853df502b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dgsz9"
Sep 17 17:48:44.882966 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:44.882740    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d604d4d-7439-49b9-adc3-31f853df502b-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-dgsz9\" (UID: \"1d604d4d-7439-49b9-adc3-31f853df502b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dgsz9"
Sep 17 17:48:44.882966 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:44.882789    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d604d4d-7439-49b9-adc3-31f853df502b-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-dgsz9\" (UID: \"1d604d4d-7439-49b9-adc3-31f853df502b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dgsz9"
Sep 17 17:48:44.882966 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:44.882829    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtwt4\" (UniqueName: \"kubernetes.io/projected/1d604d4d-7439-49b9-adc3-31f853df502b-kube-api-access-wtwt4\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-dgsz9\" (UID: \"1d604d4d-7439-49b9-adc3-31f853df502b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dgsz9"
Sep 17 17:48:44.882966 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:44.882866    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d604d4d-7439-49b9-adc3-31f853df502b-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-dgsz9\" (UID: \"1d604d4d-7439-49b9-adc3-31f853df502b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dgsz9"
Sep 17 17:48:44.984174 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:44.984053    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d604d4d-7439-49b9-adc3-31f853df502b-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-dgsz9\" (UID: \"1d604d4d-7439-49b9-adc3-31f853df502b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dgsz9"
Sep 17 17:48:44.984174 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:44.984102    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtwt4\" (UniqueName: \"kubernetes.io/projected/1d604d4d-7439-49b9-adc3-31f853df502b-kube-api-access-wtwt4\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-dgsz9\" (UID: \"1d604d4d-7439-49b9-adc3-31f853df502b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dgsz9"
Sep 17 17:48:44.984174 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:44.984135    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d604d4d-7439-49b9-adc3-31f853df502b-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-dgsz9\" (UID: \"1d604d4d-7439-49b9-adc3-31f853df502b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dgsz9"
Sep 17 17:48:44.984512 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:44.984227    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d604d4d-7439-49b9-adc3-31f853df502b-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-dgsz9\" (UID: \"1d604d4d-7439-49b9-adc3-31f853df502b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dgsz9"
Sep 17 17:48:44.984512 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:44.984262    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d604d4d-7439-49b9-adc3-31f853df502b-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-dgsz9\" (UID: \"1d604d4d-7439-49b9-adc3-31f853df502b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dgsz9"
Sep 17 17:48:44.984512 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:44.984287    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d604d4d-7439-49b9-adc3-31f853df502b-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-dgsz9\" (UID: \"1d604d4d-7439-49b9-adc3-31f853df502b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dgsz9"
Sep 17 17:48:44.984685 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:44.984593    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d604d4d-7439-49b9-adc3-31f853df502b-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-dgsz9\" (UID: \"1d604d4d-7439-49b9-adc3-31f853df502b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dgsz9"
Sep 17 17:48:44.984685 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:44.984602    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d604d4d-7439-49b9-adc3-31f853df502b-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-dgsz9\" (UID: \"1d604d4d-7439-49b9-adc3-31f853df502b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dgsz9"
Sep 17 17:48:44.984770 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:44.984694    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d604d4d-7439-49b9-adc3-31f853df502b-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-dgsz9\" (UID: \"1d604d4d-7439-49b9-adc3-31f853df502b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dgsz9"
Sep 17 17:48:44.986901 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:44.986878    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d604d4d-7439-49b9-adc3-31f853df502b-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-dgsz9\" (UID: \"1d604d4d-7439-49b9-adc3-31f853df502b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dgsz9"
Sep 17 17:48:44.987206 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:44.987183    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d604d4d-7439-49b9-adc3-31f853df502b-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-dgsz9\" (UID: \"1d604d4d-7439-49b9-adc3-31f853df502b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dgsz9"
Sep 17 17:48:45.004012 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:45.003975    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtwt4\" (UniqueName: \"kubernetes.io/projected/1d604d4d-7439-49b9-adc3-31f853df502b-kube-api-access-wtwt4\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-dgsz9\" (UID: \"1d604d4d-7439-49b9-adc3-31f853df502b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dgsz9"
Sep 17 17:48:45.048469 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:45.048396    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dgsz9"
Sep 17 17:48:45.338192 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:45.338137    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dgsz9"]
Sep 17 17:48:45.339985 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:48:45.339955    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d604d4d_7439_49b9_adc3_31f853df502b.slice/crio-d806ccb9253f61ba3284daae9dd48d4ac87f745a9260826385c179f2adcfb9cf WatchSource:0}: Error finding container d806ccb9253f61ba3284daae9dd48d4ac87f745a9260826385c179f2adcfb9cf: Status 404 returned error can't find the container with id d806ccb9253f61ba3284daae9dd48d4ac87f745a9260826385c179f2adcfb9cf
Sep 17 17:48:46.080227 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:46.080187    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dgsz9" event={"ID":"1d604d4d-7439-49b9-adc3-31f853df502b","Type":"ContainerStarted","Data":"1f57d63273ca2327d37d0ec4ff643ae02f5fbf7dabe4687065c3e19cba0470d2"}
Sep 17 17:48:46.080668 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:46.080252    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dgsz9" event={"ID":"1d604d4d-7439-49b9-adc3-31f853df502b","Type":"ContainerStarted","Data":"d806ccb9253f61ba3284daae9dd48d4ac87f745a9260826385c179f2adcfb9cf"}
Sep 17 17:48:46.104994 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:46.104932    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dgsz9" podStartSLOduration=2.104911616 podStartE2EDuration="2.104911616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:48:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:48:46.103844671 +0000 UTC m=+4011.998323814" watchObservedRunningTime="2026-09-17 17:48:46.104911616 +0000 UTC m=+4011.999390781"
Sep 17 17:48:47.048772 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:47.048722    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dgsz9"
Sep 17 17:48:47.741016 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:47.740972    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd" podUID="b0b1f752-3d88-4a0b-8ba0-ba63081d1164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": dial tcp 10.133.0.85:8000: connect: connection refused"
Sep 17 17:48:50.248609 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:50.248559    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dgsz9"]
Sep 17 17:48:50.258645 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:48:50.258604    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:48:50.258645 ip-10-0-141-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:48:50.258645 ip-10-0-141-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:48:50.258645 ip-10-0-141-66 kubenswrapper[2566]:  > containerID="1f57d63273ca2327d37d0ec4ff643ae02f5fbf7dabe4687065c3e19cba0470d2" cmd=["/bin/sleep","15"]
Sep 17 17:48:50.258949 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:50.258661    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dgsz9" message="" err=<
Sep 17 17:48:50.258949 ip-10-0-141-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:48:50.258949 ip-10-0-141-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:48:50.258949 ip-10-0-141-66 kubenswrapper[2566]:  >
Sep 17 17:48:50.258949 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:48:50.258675    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:48:50.258949 ip-10-0-141-66 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:48:50.258949 ip-10-0-141-66 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 17 17:48:50.258949 ip-10-0-141-66 kubenswrapper[2566]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dgsz9" podUID="1d604d4d-7439-49b9-adc3-31f853df502b" containerName="main" containerID="cri-o://1f57d63273ca2327d37d0ec4ff643ae02f5fbf7dabe4687065c3e19cba0470d2"
Sep 17 17:48:50.258949 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:50.258712    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dgsz9" podUID="1d604d4d-7439-49b9-adc3-31f853df502b" containerName="main" containerID="cri-o://1f57d63273ca2327d37d0ec4ff643ae02f5fbf7dabe4687065c3e19cba0470d2" gracePeriod=60
Sep 17 17:48:50.509514 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:50.509415    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dgsz9"
Sep 17 17:48:50.633792 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:50.633751    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d604d4d-7439-49b9-adc3-31f853df502b-tmp-dir\") pod \"1d604d4d-7439-49b9-adc3-31f853df502b\" (UID: \"1d604d4d-7439-49b9-adc3-31f853df502b\") "
Sep 17 17:48:50.633959 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:50.633820    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d604d4d-7439-49b9-adc3-31f853df502b-home\") pod \"1d604d4d-7439-49b9-adc3-31f853df502b\" (UID: \"1d604d4d-7439-49b9-adc3-31f853df502b\") "
Sep 17 17:48:50.633959 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:50.633903    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtwt4\" (UniqueName: \"kubernetes.io/projected/1d604d4d-7439-49b9-adc3-31f853df502b-kube-api-access-wtwt4\") pod \"1d604d4d-7439-49b9-adc3-31f853df502b\" (UID: \"1d604d4d-7439-49b9-adc3-31f853df502b\") "
Sep 17 17:48:50.634063 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:50.633956    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d604d4d-7439-49b9-adc3-31f853df502b-model-cache\") pod \"1d604d4d-7439-49b9-adc3-31f853df502b\" (UID: \"1d604d4d-7439-49b9-adc3-31f853df502b\") "
Sep 17 17:48:50.634063 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:50.633994    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d604d4d-7439-49b9-adc3-31f853df502b-dshm\") pod \"1d604d4d-7439-49b9-adc3-31f853df502b\" (UID: \"1d604d4d-7439-49b9-adc3-31f853df502b\") "
Sep 17 17:48:50.634063 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:50.634022    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d604d4d-7439-49b9-adc3-31f853df502b-tls-certs\") pod \"1d604d4d-7439-49b9-adc3-31f853df502b\" (UID: \"1d604d4d-7439-49b9-adc3-31f853df502b\") "
Sep 17 17:48:50.634229 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:50.634049    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d604d4d-7439-49b9-adc3-31f853df502b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1d604d4d-7439-49b9-adc3-31f853df502b" (UID: "1d604d4d-7439-49b9-adc3-31f853df502b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:48:50.634229 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:50.634075    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d604d4d-7439-49b9-adc3-31f853df502b-home" (OuterVolumeSpecName: "home") pod "1d604d4d-7439-49b9-adc3-31f853df502b" (UID: "1d604d4d-7439-49b9-adc3-31f853df502b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:48:50.634338 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:50.634227    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d604d4d-7439-49b9-adc3-31f853df502b-model-cache" (OuterVolumeSpecName: "model-cache") pod "1d604d4d-7439-49b9-adc3-31f853df502b" (UID: "1d604d4d-7439-49b9-adc3-31f853df502b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:48:50.634338 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:50.634244    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d604d4d-7439-49b9-adc3-31f853df502b-tmp-dir\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:48:50.634338 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:50.634264    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d604d4d-7439-49b9-adc3-31f853df502b-home\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:48:50.636485 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:50.636428    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d604d4d-7439-49b9-adc3-31f853df502b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1d604d4d-7439-49b9-adc3-31f853df502b" (UID: "1d604d4d-7439-49b9-adc3-31f853df502b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:48:50.636677 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:50.636640    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d604d4d-7439-49b9-adc3-31f853df502b-dshm" (OuterVolumeSpecName: "dshm") pod "1d604d4d-7439-49b9-adc3-31f853df502b" (UID: "1d604d4d-7439-49b9-adc3-31f853df502b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:48:50.636884 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:50.636865    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d604d4d-7439-49b9-adc3-31f853df502b-kube-api-access-wtwt4" (OuterVolumeSpecName: "kube-api-access-wtwt4") pod "1d604d4d-7439-49b9-adc3-31f853df502b" (UID: "1d604d4d-7439-49b9-adc3-31f853df502b"). InnerVolumeSpecName "kube-api-access-wtwt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:48:50.734769 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:50.734725    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtwt4\" (UniqueName: \"kubernetes.io/projected/1d604d4d-7439-49b9-adc3-31f853df502b-kube-api-access-wtwt4\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:48:50.734769 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:50.734766    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d604d4d-7439-49b9-adc3-31f853df502b-model-cache\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:48:50.734769 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:50.734781    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d604d4d-7439-49b9-adc3-31f853df502b-dshm\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:48:50.735037 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:50.734790    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d604d4d-7439-49b9-adc3-31f853df502b-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:48:51.109926 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:51.109893    2566 generic.go:356] "Generic (PLEG): container finished" podID="1d604d4d-7439-49b9-adc3-31f853df502b" containerID="1f57d63273ca2327d37d0ec4ff643ae02f5fbf7dabe4687065c3e19cba0470d2" exitCode=0
Sep 17 17:48:51.110128 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:51.109941    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dgsz9"
Sep 17 17:48:51.110128 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:51.109988    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dgsz9" event={"ID":"1d604d4d-7439-49b9-adc3-31f853df502b","Type":"ContainerDied","Data":"1f57d63273ca2327d37d0ec4ff643ae02f5fbf7dabe4687065c3e19cba0470d2"}
Sep 17 17:48:51.110128 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:51.110042    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dgsz9" event={"ID":"1d604d4d-7439-49b9-adc3-31f853df502b","Type":"ContainerDied","Data":"d806ccb9253f61ba3284daae9dd48d4ac87f745a9260826385c179f2adcfb9cf"}
Sep 17 17:48:51.110128 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:51.110066    2566 scope.go:117] "RemoveContainer" containerID="1f57d63273ca2327d37d0ec4ff643ae02f5fbf7dabe4687065c3e19cba0470d2"
Sep 17 17:48:51.121516 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:51.121483    2566 scope.go:117] "RemoveContainer" containerID="1f57d63273ca2327d37d0ec4ff643ae02f5fbf7dabe4687065c3e19cba0470d2"
Sep 17 17:48:51.122271 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:48:51.122249    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f57d63273ca2327d37d0ec4ff643ae02f5fbf7dabe4687065c3e19cba0470d2\": container with ID starting with 1f57d63273ca2327d37d0ec4ff643ae02f5fbf7dabe4687065c3e19cba0470d2 not found: ID does not exist" containerID="1f57d63273ca2327d37d0ec4ff643ae02f5fbf7dabe4687065c3e19cba0470d2"
Sep 17 17:48:51.122352 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:51.122279    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f57d63273ca2327d37d0ec4ff643ae02f5fbf7dabe4687065c3e19cba0470d2"} err="failed to get container status \"1f57d63273ca2327d37d0ec4ff643ae02f5fbf7dabe4687065c3e19cba0470d2\": rpc error: code = NotFound desc = could not find container \"1f57d63273ca2327d37d0ec4ff643ae02f5fbf7dabe4687065c3e19cba0470d2\": container with ID starting with 1f57d63273ca2327d37d0ec4ff643ae02f5fbf7dabe4687065c3e19cba0470d2 not found: ID does not exist"
Sep 17 17:48:51.145653 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:51.145616    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dgsz9"]
Sep 17 17:48:51.153361 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:51.153330    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dgsz9"]
Sep 17 17:48:52.349739 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:52.349697    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d604d4d-7439-49b9-adc3-31f853df502b" path="/var/lib/kubelet/pods/1d604d4d-7439-49b9-adc3-31f853df502b/volumes"
Sep 17 17:48:57.740868 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:48:57.740811    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd" podUID="b0b1f752-3d88-4a0b-8ba0-ba63081d1164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": dial tcp 10.133.0.85:8000: connect: connection refused"
Sep 17 17:49:07.740338 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:07.740282    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd" podUID="b0b1f752-3d88-4a0b-8ba0-ba63081d1164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": dial tcp 10.133.0.85:8000: connect: connection refused"
Sep 17 17:49:17.740603 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:17.740543    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd" podUID="b0b1f752-3d88-4a0b-8ba0-ba63081d1164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": dial tcp 10.133.0.85:8000: connect: connection refused"
Sep 17 17:49:27.750083 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:27.750045    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd"
Sep 17 17:49:27.758344 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:27.758306    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd"
Sep 17 17:49:28.893291 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:28.893250    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd"]
Sep 17 17:49:44.341642 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:44.341595    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd" podUID="b0b1f752-3d88-4a0b-8ba0-ba63081d1164" containerName="main" containerID="cri-o://7504b7a2186d6e32d2c1048386d43e5ec0f7ad86d2e5a8c64251622e216d3370" gracePeriod=45
Sep 17 17:49:44.750948 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:44.750833    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd" podUID="b0b1f752-3d88-4a0b-8ba0-ba63081d1164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": dial tcp 10.133.0.85:8000: connect: connection refused"
Sep 17 17:49:45.090769 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:45.090737    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd"
Sep 17 17:49:45.189515 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:45.189481    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-model-cache\") pod \"b0b1f752-3d88-4a0b-8ba0-ba63081d1164\" (UID: \"b0b1f752-3d88-4a0b-8ba0-ba63081d1164\") "
Sep 17 17:49:45.189515 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:45.189518    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-cabundle-cert\") pod \"b0b1f752-3d88-4a0b-8ba0-ba63081d1164\" (UID: \"b0b1f752-3d88-4a0b-8ba0-ba63081d1164\") "
Sep 17 17:49:45.189749 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:45.189555    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-tls-certs\") pod \"b0b1f752-3d88-4a0b-8ba0-ba63081d1164\" (UID: \"b0b1f752-3d88-4a0b-8ba0-ba63081d1164\") "
Sep 17 17:49:45.189749 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:45.189585    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-home\") pod \"b0b1f752-3d88-4a0b-8ba0-ba63081d1164\" (UID: \"b0b1f752-3d88-4a0b-8ba0-ba63081d1164\") "
Sep 17 17:49:45.189749 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:45.189628    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-kserve-provision-location\") pod \"b0b1f752-3d88-4a0b-8ba0-ba63081d1164\" (UID: \"b0b1f752-3d88-4a0b-8ba0-ba63081d1164\") "
Sep 17 17:49:45.189749 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:45.189672    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-tmp-dir\") pod \"b0b1f752-3d88-4a0b-8ba0-ba63081d1164\" (UID: \"b0b1f752-3d88-4a0b-8ba0-ba63081d1164\") "
Sep 17 17:49:45.189749 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:45.189731    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-dshm\") pod \"b0b1f752-3d88-4a0b-8ba0-ba63081d1164\" (UID: \"b0b1f752-3d88-4a0b-8ba0-ba63081d1164\") "
Sep 17 17:49:45.189994 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:45.189754    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgcqh\" (UniqueName: \"kubernetes.io/projected/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-kube-api-access-fgcqh\") pod \"b0b1f752-3d88-4a0b-8ba0-ba63081d1164\" (UID: \"b0b1f752-3d88-4a0b-8ba0-ba63081d1164\") "
Sep 17 17:49:45.189994 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:45.189859    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-model-cache" (OuterVolumeSpecName: "model-cache") pod "b0b1f752-3d88-4a0b-8ba0-ba63081d1164" (UID: "b0b1f752-3d88-4a0b-8ba0-ba63081d1164"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:49:45.189994 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:45.189941    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b0b1f752-3d88-4a0b-8ba0-ba63081d1164" (UID: "b0b1f752-3d88-4a0b-8ba0-ba63081d1164"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:49:45.190158 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:45.190014    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-model-cache\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:49:45.190158 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:45.190032    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-cabundle-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:49:45.190584 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:45.190551    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-home" (OuterVolumeSpecName: "home") pod "b0b1f752-3d88-4a0b-8ba0-ba63081d1164" (UID: "b0b1f752-3d88-4a0b-8ba0-ba63081d1164"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:49:45.192258 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:45.192220    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-dshm" (OuterVolumeSpecName: "dshm") pod "b0b1f752-3d88-4a0b-8ba0-ba63081d1164" (UID: "b0b1f752-3d88-4a0b-8ba0-ba63081d1164"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:49:45.192258 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:45.192235    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b0b1f752-3d88-4a0b-8ba0-ba63081d1164" (UID: "b0b1f752-3d88-4a0b-8ba0-ba63081d1164"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:49:45.192706 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:45.192684    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-kube-api-access-fgcqh" (OuterVolumeSpecName: "kube-api-access-fgcqh") pod "b0b1f752-3d88-4a0b-8ba0-ba63081d1164" (UID: "b0b1f752-3d88-4a0b-8ba0-ba63081d1164"). InnerVolumeSpecName "kube-api-access-fgcqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:49:45.203240 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:45.203194    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b0b1f752-3d88-4a0b-8ba0-ba63081d1164" (UID: "b0b1f752-3d88-4a0b-8ba0-ba63081d1164"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:49:45.213604 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:45.213570    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b0b1f752-3d88-4a0b-8ba0-ba63081d1164" (UID: "b0b1f752-3d88-4a0b-8ba0-ba63081d1164"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:49:45.290728 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:45.290691    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:49:45.290728 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:45.290727    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-home\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:49:45.290936 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:45.290737    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-kserve-provision-location\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:49:45.290936 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:45.290746    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-tmp-dir\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:49:45.290936 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:45.290754    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-dshm\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:49:45.290936 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:45.290763    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgcqh\" (UniqueName: \"kubernetes.io/projected/b0b1f752-3d88-4a0b-8ba0-ba63081d1164-kube-api-access-fgcqh\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:49:45.405806 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:45.405763    2566 generic.go:356] "Generic (PLEG): container finished" podID="b0b1f752-3d88-4a0b-8ba0-ba63081d1164" containerID="7504b7a2186d6e32d2c1048386d43e5ec0f7ad86d2e5a8c64251622e216d3370" exitCode=0
Sep 17 17:49:45.406319 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:45.405826    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd"
Sep 17 17:49:45.406319 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:45.405856    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd" event={"ID":"b0b1f752-3d88-4a0b-8ba0-ba63081d1164","Type":"ContainerDied","Data":"7504b7a2186d6e32d2c1048386d43e5ec0f7ad86d2e5a8c64251622e216d3370"}
Sep 17 17:49:45.406319 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:45.405936    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd" event={"ID":"b0b1f752-3d88-4a0b-8ba0-ba63081d1164","Type":"ContainerDied","Data":"2792486ff9d4a58d9dfa172d2fd23d627a13924439a9cb545e6253cceb65a525"}
Sep 17 17:49:45.406319 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:45.405971    2566 scope.go:117] "RemoveContainer" containerID="7504b7a2186d6e32d2c1048386d43e5ec0f7ad86d2e5a8c64251622e216d3370"
Sep 17 17:49:45.416072 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:45.416052    2566 scope.go:117] "RemoveContainer" containerID="e1df344bd5b451f85fd25caa60c33040d824b98c9f871ed1b307585ecea0998d"
Sep 17 17:49:45.424327 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:45.424307    2566 scope.go:117] "RemoveContainer" containerID="7504b7a2186d6e32d2c1048386d43e5ec0f7ad86d2e5a8c64251622e216d3370"
Sep 17 17:49:45.424622 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:49:45.424601    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7504b7a2186d6e32d2c1048386d43e5ec0f7ad86d2e5a8c64251622e216d3370\": container with ID starting with 7504b7a2186d6e32d2c1048386d43e5ec0f7ad86d2e5a8c64251622e216d3370 not found: ID does not exist" containerID="7504b7a2186d6e32d2c1048386d43e5ec0f7ad86d2e5a8c64251622e216d3370"
Sep 17 17:49:45.424677 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:45.424634    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7504b7a2186d6e32d2c1048386d43e5ec0f7ad86d2e5a8c64251622e216d3370"} err="failed to get container status \"7504b7a2186d6e32d2c1048386d43e5ec0f7ad86d2e5a8c64251622e216d3370\": rpc error: code = NotFound desc = could not find container \"7504b7a2186d6e32d2c1048386d43e5ec0f7ad86d2e5a8c64251622e216d3370\": container with ID starting with 7504b7a2186d6e32d2c1048386d43e5ec0f7ad86d2e5a8c64251622e216d3370 not found: ID does not exist"
Sep 17 17:49:45.424677 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:45.424652    2566 scope.go:117] "RemoveContainer" containerID="e1df344bd5b451f85fd25caa60c33040d824b98c9f871ed1b307585ecea0998d"
Sep 17 17:49:45.424924 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:49:45.424904    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1df344bd5b451f85fd25caa60c33040d824b98c9f871ed1b307585ecea0998d\": container with ID starting with e1df344bd5b451f85fd25caa60c33040d824b98c9f871ed1b307585ecea0998d not found: ID does not exist" containerID="e1df344bd5b451f85fd25caa60c33040d824b98c9f871ed1b307585ecea0998d"
Sep 17 17:49:45.424965 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:45.424932    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1df344bd5b451f85fd25caa60c33040d824b98c9f871ed1b307585ecea0998d"} err="failed to get container status \"e1df344bd5b451f85fd25caa60c33040d824b98c9f871ed1b307585ecea0998d\": rpc error: code = NotFound desc = could not find container \"e1df344bd5b451f85fd25caa60c33040d824b98c9f871ed1b307585ecea0998d\": container with ID starting with e1df344bd5b451f85fd25caa60c33040d824b98c9f871ed1b307585ecea0998d not found: ID does not exist"
Sep 17 17:49:45.432184 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:45.432145    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd"]
Sep 17 17:49:45.441298 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:45.441269    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-dh5wd"]
Sep 17 17:49:46.347407 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:46.347367    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0b1f752-3d88-4a0b-8ba0-ba63081d1164" path="/var/lib/kubelet/pods/b0b1f752-3d88-4a0b-8ba0-ba63081d1164/volumes"
Sep 17 17:49:47.462600 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:47.462558    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-kxhkp"]
Sep 17 17:49:49.650888 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:49.650841    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-kxhkp" podUID="ed85b818-72f4-42d5-999d-a37470f23e19" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:49:59.650232 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:49:59.650186    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-kxhkp" podUID="ed85b818-72f4-42d5-999d-a37470f23e19" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:50:02.486097 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:02.486024    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-kxhkp" podUID="ed85b818-72f4-42d5-999d-a37470f23e19" containerName="main" containerID="cri-o://0f61e26eda060acd042bfd30eeb0d776e71dd77067a7fde79d7f8c4a7cbdcc02" gracePeriod=45
Sep 17 17:50:09.650410 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:09.650367    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-kxhkp" podUID="ed85b818-72f4-42d5-999d-a37470f23e19" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:50:09.650903 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:09.650523    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-kxhkp"
Sep 17 17:50:19.650328 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:19.650280    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-kxhkp" podUID="ed85b818-72f4-42d5-999d-a37470f23e19" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:50:29.650837 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:29.650790    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-kxhkp" podUID="ed85b818-72f4-42d5-999d-a37470f23e19" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:50:32.650377 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:32.650338    2566 generic.go:356] "Generic (PLEG): container finished" podID="ed85b818-72f4-42d5-999d-a37470f23e19" containerID="0f61e26eda060acd042bfd30eeb0d776e71dd77067a7fde79d7f8c4a7cbdcc02" exitCode=0
Sep 17 17:50:32.650803 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:32.650404    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-kxhkp" event={"ID":"ed85b818-72f4-42d5-999d-a37470f23e19","Type":"ContainerDied","Data":"0f61e26eda060acd042bfd30eeb0d776e71dd77067a7fde79d7f8c4a7cbdcc02"}
Sep 17 17:50:32.743115 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:32.743085    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-kxhkp"
Sep 17 17:50:32.847340 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:32.847301    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x95sf\" (UniqueName: \"kubernetes.io/projected/ed85b818-72f4-42d5-999d-a37470f23e19-kube-api-access-x95sf\") pod \"ed85b818-72f4-42d5-999d-a37470f23e19\" (UID: \"ed85b818-72f4-42d5-999d-a37470f23e19\") "
Sep 17 17:50:32.847548 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:32.847351    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed85b818-72f4-42d5-999d-a37470f23e19-tls-certs\") pod \"ed85b818-72f4-42d5-999d-a37470f23e19\" (UID: \"ed85b818-72f4-42d5-999d-a37470f23e19\") "
Sep 17 17:50:32.849720 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:32.849684    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed85b818-72f4-42d5-999d-a37470f23e19-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ed85b818-72f4-42d5-999d-a37470f23e19" (UID: "ed85b818-72f4-42d5-999d-a37470f23e19"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:50:32.849720 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:32.849693    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed85b818-72f4-42d5-999d-a37470f23e19-kube-api-access-x95sf" (OuterVolumeSpecName: "kube-api-access-x95sf") pod "ed85b818-72f4-42d5-999d-a37470f23e19" (UID: "ed85b818-72f4-42d5-999d-a37470f23e19"). InnerVolumeSpecName "kube-api-access-x95sf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:50:32.947880 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:32.947839    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x95sf\" (UniqueName: \"kubernetes.io/projected/ed85b818-72f4-42d5-999d-a37470f23e19-kube-api-access-x95sf\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:50:32.947880 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:32.947874    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed85b818-72f4-42d5-999d-a37470f23e19-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:50:33.657076 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:33.657042    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-kxhkp"
Sep 17 17:50:33.657595 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:33.657041    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-kxhkp" event={"ID":"ed85b818-72f4-42d5-999d-a37470f23e19","Type":"ContainerDied","Data":"db900f28c554289678f8f63fb70cfb31c12412b582a993f61e771642d57c4112"}
Sep 17 17:50:33.657595 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:33.657180    2566 scope.go:117] "RemoveContainer" containerID="0f61e26eda060acd042bfd30eeb0d776e71dd77067a7fde79d7f8c4a7cbdcc02"
Sep 17 17:50:33.699246 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:33.699199    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-kxhkp"]
Sep 17 17:50:33.706153 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:33.706118    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-kxhkp"]
Sep 17 17:50:34.346328 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:34.346295    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed85b818-72f4-42d5-999d-a37470f23e19" path="/var/lib/kubelet/pods/ed85b818-72f4-42d5-999d-a37470f23e19/volumes"
Sep 17 17:50:54.151073 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:54.151027    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jkqp9/must-gather-pqgvj"]
Sep 17 17:50:54.151663 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:54.151519    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0b1f752-3d88-4a0b-8ba0-ba63081d1164" containerName="storage-initializer"
Sep 17 17:50:54.151663 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:54.151540    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0b1f752-3d88-4a0b-8ba0-ba63081d1164" containerName="storage-initializer"
Sep 17 17:50:54.151663 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:54.151557    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed85b818-72f4-42d5-999d-a37470f23e19" containerName="main"
Sep 17 17:50:54.151663 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:54.151566    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed85b818-72f4-42d5-999d-a37470f23e19" containerName="main"
Sep 17 17:50:54.151663 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:54.151579    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0b1f752-3d88-4a0b-8ba0-ba63081d1164" containerName="main"
Sep 17 17:50:54.151663 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:54.151589    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0b1f752-3d88-4a0b-8ba0-ba63081d1164" containerName="main"
Sep 17 17:50:54.151663 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:54.151615    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d604d4d-7439-49b9-adc3-31f853df502b" containerName="main"
Sep 17 17:50:54.151663 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:54.151623    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d604d4d-7439-49b9-adc3-31f853df502b" containerName="main"
Sep 17 17:50:54.152068 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:54.151702    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed85b818-72f4-42d5-999d-a37470f23e19" containerName="main"
Sep 17 17:50:54.152068 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:54.151715    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d604d4d-7439-49b9-adc3-31f853df502b" containerName="main"
Sep 17 17:50:54.152068 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:54.151726    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0b1f752-3d88-4a0b-8ba0-ba63081d1164" containerName="main"
Sep 17 17:50:54.156491 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:54.156468    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jkqp9/must-gather-pqgvj"
Sep 17 17:50:54.159085 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:54.159065    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jkqp9\"/\"kube-root-ca.crt\""
Sep 17 17:50:54.159332 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:54.159315    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jkqp9\"/\"default-dockercfg-rljjp\""
Sep 17 17:50:54.160009 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:54.159986    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jkqp9\"/\"openshift-service-ca.crt\""
Sep 17 17:50:54.169513 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:54.169488    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jkqp9/must-gather-pqgvj"]
Sep 17 17:50:54.217715 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:54.217672    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5x9d\" (UniqueName: \"kubernetes.io/projected/08421a1f-7214-4714-bc45-60d1b7eeab04-kube-api-access-b5x9d\") pod \"must-gather-pqgvj\" (UID: \"08421a1f-7214-4714-bc45-60d1b7eeab04\") " pod="openshift-must-gather-jkqp9/must-gather-pqgvj"
Sep 17 17:50:54.217912 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:54.217739    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/08421a1f-7214-4714-bc45-60d1b7eeab04-must-gather-output\") pod \"must-gather-pqgvj\" (UID: \"08421a1f-7214-4714-bc45-60d1b7eeab04\") " pod="openshift-must-gather-jkqp9/must-gather-pqgvj"
Sep 17 17:50:54.318339 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:54.318290    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5x9d\" (UniqueName: \"kubernetes.io/projected/08421a1f-7214-4714-bc45-60d1b7eeab04-kube-api-access-b5x9d\") pod \"must-gather-pqgvj\" (UID: \"08421a1f-7214-4714-bc45-60d1b7eeab04\") " pod="openshift-must-gather-jkqp9/must-gather-pqgvj"
Sep 17 17:50:54.318597 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:54.318385    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/08421a1f-7214-4714-bc45-60d1b7eeab04-must-gather-output\") pod \"must-gather-pqgvj\" (UID: \"08421a1f-7214-4714-bc45-60d1b7eeab04\") " pod="openshift-must-gather-jkqp9/must-gather-pqgvj"
Sep 17 17:50:54.318776 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:54.318755    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/08421a1f-7214-4714-bc45-60d1b7eeab04-must-gather-output\") pod \"must-gather-pqgvj\" (UID: \"08421a1f-7214-4714-bc45-60d1b7eeab04\") " pod="openshift-must-gather-jkqp9/must-gather-pqgvj"
Sep 17 17:50:54.340477 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:54.340430    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5x9d\" (UniqueName: \"kubernetes.io/projected/08421a1f-7214-4714-bc45-60d1b7eeab04-kube-api-access-b5x9d\") pod \"must-gather-pqgvj\" (UID: \"08421a1f-7214-4714-bc45-60d1b7eeab04\") " pod="openshift-must-gather-jkqp9/must-gather-pqgvj"
Sep 17 17:50:54.465932 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:54.465835    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jkqp9/must-gather-pqgvj"
Sep 17 17:50:54.749651 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:54.749624    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jkqp9/must-gather-pqgvj"]
Sep 17 17:50:54.751863 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:50:54.751830    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08421a1f_7214_4714_bc45_60d1b7eeab04.slice/crio-30336ad00ae0a20ace2fadb0ea51b38da838946761b0577beace05b1eb24cb47 WatchSource:0}: Error finding container 30336ad00ae0a20ace2fadb0ea51b38da838946761b0577beace05b1eb24cb47: Status 404 returned error can't find the container with id 30336ad00ae0a20ace2fadb0ea51b38da838946761b0577beace05b1eb24cb47
Sep 17 17:50:54.753736 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:54.753718    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:50:54.768132 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:50:54.768098    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jkqp9/must-gather-pqgvj" event={"ID":"08421a1f-7214-4714-bc45-60d1b7eeab04","Type":"ContainerStarted","Data":"30336ad00ae0a20ace2fadb0ea51b38da838946761b0577beace05b1eb24cb47"}
Sep 17 17:51:01.820795 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:01.820755    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jkqp9/must-gather-pqgvj" event={"ID":"08421a1f-7214-4714-bc45-60d1b7eeab04","Type":"ContainerStarted","Data":"d948ba8e7a768ed4a8907cf1ea4306d715102813c09569acf5e673640ca03e31"}
Sep 17 17:51:01.820795 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:01.820801    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jkqp9/must-gather-pqgvj" event={"ID":"08421a1f-7214-4714-bc45-60d1b7eeab04","Type":"ContainerStarted","Data":"9d35c43d82b607525aadffbc88c74de4ee8f3fa2203c2c803af9dfc59b6370ef"}
Sep 17 17:51:01.842143 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:01.842084    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jkqp9/must-gather-pqgvj" podStartSLOduration=1.433632548 podStartE2EDuration="7.842067555s" totalImagesPullingTime="6.408435007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:50:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:51:01.841613971 +0000 UTC m=+4147.736093111" watchObservedRunningTime="2026-09-17 17:51:01.842067555 +0000 UTC m=+4147.736546694"
Sep 17 17:51:22.402761 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:22.402721    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-qnzll_fbc29aee-02c6-4de1-ba3f-69063895858d/manager/0.log"
Sep 17 17:51:22.636154 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:22.636116    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-sh727_c18cf2f8-66d1-4492-bf48-0defd23dc1c7/limitador/0.log"
Sep 17 17:51:22.659709 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:22.659626    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-5bhzb_d7125f90-b9d9-48ba-a300-1678738e5ee8/manager/0.log"
Sep 17 17:51:23.953183 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:23.953143    2566 generic.go:356] "Generic (PLEG): container finished" podID="08421a1f-7214-4714-bc45-60d1b7eeab04" containerID="9d35c43d82b607525aadffbc88c74de4ee8f3fa2203c2c803af9dfc59b6370ef" exitCode=0
Sep 17 17:51:23.953671 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:23.953220    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jkqp9/must-gather-pqgvj" event={"ID":"08421a1f-7214-4714-bc45-60d1b7eeab04","Type":"ContainerDied","Data":"9d35c43d82b607525aadffbc88c74de4ee8f3fa2203c2c803af9dfc59b6370ef"}
Sep 17 17:51:23.953671 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:23.953591    2566 scope.go:117] "RemoveContainer" containerID="9d35c43d82b607525aadffbc88c74de4ee8f3fa2203c2c803af9dfc59b6370ef"
Sep 17 17:51:24.358713 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:24.358678    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-jkqp9_must-gather-pqgvj_08421a1f-7214-4714-bc45-60d1b7eeab04/gather/0.log"
Sep 17 17:51:28.655777 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:28.655740    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xbdp6_2a0d6b45-1f66-4c14-988a-1ec74eac67ad/global-pull-secret-syncer/0.log"
Sep 17 17:51:28.819981 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:28.819946    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-z8fgx_c10930b3-04d3-4b91-8931-a9638b8c5ea7/konnectivity-agent/0.log"
Sep 17 17:51:28.915599 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:28.915505    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-66.ec2.internal_dcbcba84d85e3930495a76b3760ea833/haproxy/0.log"
Sep 17 17:51:29.898753 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:29.898714    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-jkqp9/must-gather-pqgvj"]
Sep 17 17:51:29.899221 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:29.898895    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-jkqp9/must-gather-pqgvj" podUID="08421a1f-7214-4714-bc45-60d1b7eeab04" containerName="copy" containerID="cri-o://d948ba8e7a768ed4a8907cf1ea4306d715102813c09569acf5e673640ca03e31" gracePeriod=2
Sep 17 17:51:29.906873 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:29.906845    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-jkqp9/must-gather-pqgvj"]
Sep 17 17:51:30.133815 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:30.133792    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-jkqp9_must-gather-pqgvj_08421a1f-7214-4714-bc45-60d1b7eeab04/copy/0.log"
Sep 17 17:51:30.142821 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:30.136981    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jkqp9/must-gather-pqgvj"
Sep 17 17:51:30.142821 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:51:30.142760    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-pqgvj\" is forbidden: User \"system:node:ip-10-0-141-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-jkqp9\": no relationship found between node 'ip-10-0-141-66.ec2.internal' and this object" podUID="08421a1f-7214-4714-bc45-60d1b7eeab04" pod="openshift-must-gather-jkqp9/must-gather-pqgvj"
Sep 17 17:51:30.230894 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:30.230789    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/08421a1f-7214-4714-bc45-60d1b7eeab04-must-gather-output\") pod \"08421a1f-7214-4714-bc45-60d1b7eeab04\" (UID: \"08421a1f-7214-4714-bc45-60d1b7eeab04\") "
Sep 17 17:51:30.236321 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:30.236287    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08421a1f-7214-4714-bc45-60d1b7eeab04-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "08421a1f-7214-4714-bc45-60d1b7eeab04" (UID: "08421a1f-7214-4714-bc45-60d1b7eeab04"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:51:30.331234 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:30.331194    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5x9d\" (UniqueName: \"kubernetes.io/projected/08421a1f-7214-4714-bc45-60d1b7eeab04-kube-api-access-b5x9d\") pod \"08421a1f-7214-4714-bc45-60d1b7eeab04\" (UID: \"08421a1f-7214-4714-bc45-60d1b7eeab04\") "
Sep 17 17:51:30.331390 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:30.331324    2566 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/08421a1f-7214-4714-bc45-60d1b7eeab04-must-gather-output\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:51:30.333742 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:30.333703    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08421a1f-7214-4714-bc45-60d1b7eeab04-kube-api-access-b5x9d" (OuterVolumeSpecName: "kube-api-access-b5x9d") pod "08421a1f-7214-4714-bc45-60d1b7eeab04" (UID: "08421a1f-7214-4714-bc45-60d1b7eeab04"). InnerVolumeSpecName "kube-api-access-b5x9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:51:30.346353 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:30.346321    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08421a1f-7214-4714-bc45-60d1b7eeab04" path="/var/lib/kubelet/pods/08421a1f-7214-4714-bc45-60d1b7eeab04/volumes"
Sep 17 17:51:30.431871 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:30.431835    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5x9d\" (UniqueName: \"kubernetes.io/projected/08421a1f-7214-4714-bc45-60d1b7eeab04-kube-api-access-b5x9d\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\""
Sep 17 17:51:30.990853 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:30.990824    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-jkqp9_must-gather-pqgvj_08421a1f-7214-4714-bc45-60d1b7eeab04/copy/0.log"
Sep 17 17:51:30.991311 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:30.991159    2566 generic.go:356] "Generic (PLEG): container finished" podID="08421a1f-7214-4714-bc45-60d1b7eeab04" containerID="d948ba8e7a768ed4a8907cf1ea4306d715102813c09569acf5e673640ca03e31" exitCode=143
Sep 17 17:51:30.991311 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:30.991230    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jkqp9/must-gather-pqgvj"
Sep 17 17:51:30.991311 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:30.991267    2566 scope.go:117] "RemoveContainer" containerID="d948ba8e7a768ed4a8907cf1ea4306d715102813c09569acf5e673640ca03e31"
Sep 17 17:51:31.000712 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:31.000691    2566 scope.go:117] "RemoveContainer" containerID="9d35c43d82b607525aadffbc88c74de4ee8f3fa2203c2c803af9dfc59b6370ef"
Sep 17 17:51:31.015383 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:31.015359    2566 scope.go:117] "RemoveContainer" containerID="d948ba8e7a768ed4a8907cf1ea4306d715102813c09569acf5e673640ca03e31"
Sep 17 17:51:31.015744 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:51:31.015723    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d948ba8e7a768ed4a8907cf1ea4306d715102813c09569acf5e673640ca03e31\": container with ID starting with d948ba8e7a768ed4a8907cf1ea4306d715102813c09569acf5e673640ca03e31 not found: ID does not exist" containerID="d948ba8e7a768ed4a8907cf1ea4306d715102813c09569acf5e673640ca03e31"
Sep 17 17:51:31.015826 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:31.015754    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d948ba8e7a768ed4a8907cf1ea4306d715102813c09569acf5e673640ca03e31"} err="failed to get container status \"d948ba8e7a768ed4a8907cf1ea4306d715102813c09569acf5e673640ca03e31\": rpc error: code = NotFound desc = could not find container \"d948ba8e7a768ed4a8907cf1ea4306d715102813c09569acf5e673640ca03e31\": container with ID starting with d948ba8e7a768ed4a8907cf1ea4306d715102813c09569acf5e673640ca03e31 not found: ID does not exist"
Sep 17 17:51:31.015826 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:31.015775    2566 scope.go:117] "RemoveContainer" containerID="9d35c43d82b607525aadffbc88c74de4ee8f3fa2203c2c803af9dfc59b6370ef"
Sep 17 17:51:31.016071 ip-10-0-141-66 kubenswrapper[2566]: E0917 17:51:31.016044    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d35c43d82b607525aadffbc88c74de4ee8f3fa2203c2c803af9dfc59b6370ef\": container with ID starting with 9d35c43d82b607525aadffbc88c74de4ee8f3fa2203c2c803af9dfc59b6370ef not found: ID does not exist" containerID="9d35c43d82b607525aadffbc88c74de4ee8f3fa2203c2c803af9dfc59b6370ef"
Sep 17 17:51:31.016138 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:31.016066    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d35c43d82b607525aadffbc88c74de4ee8f3fa2203c2c803af9dfc59b6370ef"} err="failed to get container status \"9d35c43d82b607525aadffbc88c74de4ee8f3fa2203c2c803af9dfc59b6370ef\": rpc error: code = NotFound desc = could not find container \"9d35c43d82b607525aadffbc88c74de4ee8f3fa2203c2c803af9dfc59b6370ef\": container with ID starting with 9d35c43d82b607525aadffbc88c74de4ee8f3fa2203c2c803af9dfc59b6370ef not found: ID does not exist"
Sep 17 17:51:33.567721 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:33.567690    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-qnzll_fbc29aee-02c6-4de1-ba3f-69063895858d/manager/0.log"
Sep 17 17:51:33.952374 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:33.952292    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-sh727_c18cf2f8-66d1-4492-bf48-0defd23dc1c7/limitador/0.log"
Sep 17 17:51:34.010971 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:34.010934    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-5bhzb_d7125f90-b9d9-48ba-a300-1678738e5ee8/manager/0.log"
Sep 17 17:51:36.298620 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:36.298593    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l5ppz_406c1ffc-6544-4566-aac6-afd21bbe6071/node-exporter/0.log"
Sep 17 17:51:36.330837 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:36.330804    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l5ppz_406c1ffc-6544-4566-aac6-afd21bbe6071/kube-rbac-proxy/0.log"
Sep 17 17:51:36.359184 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:36.359152    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l5ppz_406c1ffc-6544-4566-aac6-afd21bbe6071/init-textfile/0.log"
Sep 17 17:51:36.619929 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:36.619812    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-pxb89_63f1b2a4-fb40-43bf-9875-97364d40351e/kube-rbac-proxy-main/0.log"
Sep 17 17:51:36.654373 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:36.654341    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-pxb89_63f1b2a4-fb40-43bf-9875-97364d40351e/kube-rbac-proxy-self/0.log"
Sep 17 17:51:36.686383 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:36.686344    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-pxb89_63f1b2a4-fb40-43bf-9875-97364d40351e/openshift-state-metrics/0.log"
Sep 17 17:51:37.473304 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:37.473255    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2krt9/perf-node-gather-daemonset-5tsnr"]
Sep 17 17:51:37.474694 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:37.473952    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08421a1f-7214-4714-bc45-60d1b7eeab04" containerName="gather"
Sep 17 17:51:37.474694 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:37.473977    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="08421a1f-7214-4714-bc45-60d1b7eeab04" containerName="gather"
Sep 17 17:51:37.474694 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:37.473995    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08421a1f-7214-4714-bc45-60d1b7eeab04" containerName="copy"
Sep 17 17:51:37.474694 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:37.474004    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="08421a1f-7214-4714-bc45-60d1b7eeab04" containerName="copy"
Sep 17 17:51:37.474694 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:37.474170    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="08421a1f-7214-4714-bc45-60d1b7eeab04" containerName="gather"
Sep 17 17:51:37.474694 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:37.474187    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="08421a1f-7214-4714-bc45-60d1b7eeab04" containerName="copy"
Sep 17 17:51:37.478020 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:37.477997    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2krt9/perf-node-gather-daemonset-5tsnr"
Sep 17 17:51:37.480251 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:37.480231    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2krt9\"/\"default-dockercfg-9ssl8\""
Sep 17 17:51:37.480839 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:37.480820    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2krt9\"/\"kube-root-ca.crt\""
Sep 17 17:51:37.480925 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:37.480910    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2krt9\"/\"openshift-service-ca.crt\""
Sep 17 17:51:37.484444 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:37.484413    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dmkr\" (UniqueName: \"kubernetes.io/projected/bec015ac-ae3a-4333-aade-7f6a45d0e664-kube-api-access-7dmkr\") pod \"perf-node-gather-daemonset-5tsnr\" (UID: \"bec015ac-ae3a-4333-aade-7f6a45d0e664\") " pod="openshift-must-gather-2krt9/perf-node-gather-daemonset-5tsnr"
Sep 17 17:51:37.484575 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:37.484489    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bec015ac-ae3a-4333-aade-7f6a45d0e664-podres\") pod \"perf-node-gather-daemonset-5tsnr\" (UID: \"bec015ac-ae3a-4333-aade-7f6a45d0e664\") " pod="openshift-must-gather-2krt9/perf-node-gather-daemonset-5tsnr"
Sep 17 17:51:37.484575 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:37.484519    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bec015ac-ae3a-4333-aade-7f6a45d0e664-sys\") pod \"perf-node-gather-daemonset-5tsnr\" (UID: \"bec015ac-ae3a-4333-aade-7f6a45d0e664\") " pod="openshift-must-gather-2krt9/perf-node-gather-daemonset-5tsnr"
Sep 17 17:51:37.484682 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:37.484617    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bec015ac-ae3a-4333-aade-7f6a45d0e664-proc\") pod \"perf-node-gather-daemonset-5tsnr\" (UID: \"bec015ac-ae3a-4333-aade-7f6a45d0e664\") " pod="openshift-must-gather-2krt9/perf-node-gather-daemonset-5tsnr"
Sep 17 17:51:37.484682 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:37.484649    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bec015ac-ae3a-4333-aade-7f6a45d0e664-lib-modules\") pod \"perf-node-gather-daemonset-5tsnr\" (UID: \"bec015ac-ae3a-4333-aade-7f6a45d0e664\") " pod="openshift-must-gather-2krt9/perf-node-gather-daemonset-5tsnr"
Sep 17 17:51:37.486090 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:37.486063    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2krt9/perf-node-gather-daemonset-5tsnr"]
Sep 17 17:51:37.585103 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:37.585044    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bec015ac-ae3a-4333-aade-7f6a45d0e664-proc\") pod \"perf-node-gather-daemonset-5tsnr\" (UID: \"bec015ac-ae3a-4333-aade-7f6a45d0e664\") " pod="openshift-must-gather-2krt9/perf-node-gather-daemonset-5tsnr"
Sep 17 17:51:37.585103 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:37.585110    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bec015ac-ae3a-4333-aade-7f6a45d0e664-lib-modules\") pod \"perf-node-gather-daemonset-5tsnr\" (UID: \"bec015ac-ae3a-4333-aade-7f6a45d0e664\") " pod="openshift-must-gather-2krt9/perf-node-gather-daemonset-5tsnr"
Sep 17 17:51:37.585345 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:37.585137    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dmkr\" (UniqueName: \"kubernetes.io/projected/bec015ac-ae3a-4333-aade-7f6a45d0e664-kube-api-access-7dmkr\") pod \"perf-node-gather-daemonset-5tsnr\" (UID: \"bec015ac-ae3a-4333-aade-7f6a45d0e664\") " pod="openshift-must-gather-2krt9/perf-node-gather-daemonset-5tsnr"
Sep 17 17:51:37.585345 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:37.585161    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bec015ac-ae3a-4333-aade-7f6a45d0e664-podres\") pod \"perf-node-gather-daemonset-5tsnr\" (UID: \"bec015ac-ae3a-4333-aade-7f6a45d0e664\") " pod="openshift-must-gather-2krt9/perf-node-gather-daemonset-5tsnr"
Sep 17 17:51:37.585345 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:37.585182    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bec015ac-ae3a-4333-aade-7f6a45d0e664-proc\") pod \"perf-node-gather-daemonset-5tsnr\" (UID: \"bec015ac-ae3a-4333-aade-7f6a45d0e664\") " pod="openshift-must-gather-2krt9/perf-node-gather-daemonset-5tsnr"
Sep 17 17:51:37.585345 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:37.585183    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bec015ac-ae3a-4333-aade-7f6a45d0e664-sys\") pod \"perf-node-gather-daemonset-5tsnr\" (UID: \"bec015ac-ae3a-4333-aade-7f6a45d0e664\") " pod="openshift-must-gather-2krt9/perf-node-gather-daemonset-5tsnr"
Sep 17 17:51:37.585345 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:37.585234    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bec015ac-ae3a-4333-aade-7f6a45d0e664-sys\") pod \"perf-node-gather-daemonset-5tsnr\" (UID: \"bec015ac-ae3a-4333-aade-7f6a45d0e664\") " pod="openshift-must-gather-2krt9/perf-node-gather-daemonset-5tsnr"
Sep 17 17:51:37.585345 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:37.585324    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bec015ac-ae3a-4333-aade-7f6a45d0e664-lib-modules\") pod \"perf-node-gather-daemonset-5tsnr\" (UID: \"bec015ac-ae3a-4333-aade-7f6a45d0e664\") " pod="openshift-must-gather-2krt9/perf-node-gather-daemonset-5tsnr"
Sep 17 17:51:37.585583 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:37.585324    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bec015ac-ae3a-4333-aade-7f6a45d0e664-podres\") pod \"perf-node-gather-daemonset-5tsnr\" (UID: \"bec015ac-ae3a-4333-aade-7f6a45d0e664\") " pod="openshift-must-gather-2krt9/perf-node-gather-daemonset-5tsnr"
Sep 17 17:51:37.595530 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:37.595497    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dmkr\" (UniqueName: \"kubernetes.io/projected/bec015ac-ae3a-4333-aade-7f6a45d0e664-kube-api-access-7dmkr\") pod \"perf-node-gather-daemonset-5tsnr\" (UID: \"bec015ac-ae3a-4333-aade-7f6a45d0e664\") " pod="openshift-must-gather-2krt9/perf-node-gather-daemonset-5tsnr"
Sep 17 17:51:37.789230 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:37.789195    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2krt9/perf-node-gather-daemonset-5tsnr"
Sep 17 17:51:38.075855 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:38.075824    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2krt9/perf-node-gather-daemonset-5tsnr"]
Sep 17 17:51:38.078426 ip-10-0-141-66 kubenswrapper[2566]: W0917 17:51:38.078397    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbec015ac_ae3a_4333_aade_7f6a45d0e664.slice/crio-7af134c9d3aea42d286bcb362f3c6ff84b2ef31a3b5eeffdcd72c731b6836d78 WatchSource:0}: Error finding container 7af134c9d3aea42d286bcb362f3c6ff84b2ef31a3b5eeffdcd72c731b6836d78: Status 404 returned error can't find the container with id 7af134c9d3aea42d286bcb362f3c6ff84b2ef31a3b5eeffdcd72c731b6836d78
Sep 17 17:51:39.035819 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:39.035776    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2krt9/perf-node-gather-daemonset-5tsnr" event={"ID":"bec015ac-ae3a-4333-aade-7f6a45d0e664","Type":"ContainerStarted","Data":"a8c0a6af30c08d1c1430487f02d85da11825bc38c4181425570768587f9caaf7"}
Sep 17 17:51:39.036243 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:39.035833    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2krt9/perf-node-gather-daemonset-5tsnr" event={"ID":"bec015ac-ae3a-4333-aade-7f6a45d0e664","Type":"ContainerStarted","Data":"7af134c9d3aea42d286bcb362f3c6ff84b2ef31a3b5eeffdcd72c731b6836d78"}
Sep 17 17:51:39.036243 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:39.035874    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2krt9/perf-node-gather-daemonset-5tsnr"
Sep 17 17:51:39.059512 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:39.059453    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2krt9/perf-node-gather-daemonset-5tsnr" podStartSLOduration=2.059420191 podStartE2EDuration="2.059420191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:51:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:51:39.057664906 +0000 UTC m=+4184.952144044" watchObservedRunningTime="2026-09-17 17:51:39.059420191 +0000 UTC m=+4184.953899330"
Sep 17 17:51:41.538227 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:41.538190    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dmhng_7ce87b87-3574-43de-9939-9be53e78b3e3/dns/0.log"
Sep 17 17:51:41.574830 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:41.574777    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dmhng_7ce87b87-3574-43de-9939-9be53e78b3e3/kube-rbac-proxy/0.log"
Sep 17 17:51:41.918475 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:41.918421    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gnzg4_17d178d9-d43e-468a-9605-359baa45dc72/dns-node-resolver/0.log"
Sep 17 17:51:42.796552 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:42.796517    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-54cc4_96bf0c2f-8a48-491a-bb3d-73e417abfa26/node-ca/0.log"
Sep 17 17:51:45.051329 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:45.051297    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2krt9/perf-node-gather-daemonset-5tsnr"
Sep 17 17:51:45.259509 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:45.259476    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vbswf_310fde3a-70b0-4bc6-8dd8-021b49da2f53/serve-healthcheck-canary/0.log"
Sep 17 17:51:46.362647 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:46.362567    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nnsgj_a654b952-b3e3-4827-8d9d-9e346f66857f/kube-rbac-proxy/0.log"
Sep 17 17:51:46.404714 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:46.404679    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nnsgj_a654b952-b3e3-4827-8d9d-9e346f66857f/exporter/0.log"
Sep 17 17:51:46.455821 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:46.455791    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nnsgj_a654b952-b3e3-4827-8d9d-9e346f66857f/extractor/0.log"
Sep 17 17:51:51.238028 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:51.237989    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-gkcdn_c19fe3f1-8cde-4773-875b-c125d01d964a/openshift-lws-operator/0.log"
Sep 17 17:51:52.058192 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:52.058158    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-846cf5d849-ndw56_65d593a7-e622-40a5-81b9-ff549faf9546/manager/0.log"
Sep 17 17:51:52.511056 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:52.510973    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-758fb496fc-sj9bf_6c28f291-1afd-4f74-b4dc-b87c905f7f79/manager/0.log"
Sep 17 17:51:52.578297 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:52.578265    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-lssj5_9a3b6e7c-75d9-4ab6-9319-35a0bb32bebb/server/0.log"
Sep 17 17:51:53.149006 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:53.148965    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-5d6kl_bb6aa6ec-c87f-4948-9617-1c512c866fd3/manager/0.log"
Sep 17 17:51:53.209194 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:53.209153    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-9vs5m_2453727a-9cf4-444a-8e22-d7d2daae63e4/s3-init/0.log"
Sep 17 17:51:53.239313 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:53.239285    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-9vs5m_2453727a-9cf4-444a-8e22-d7d2daae63e4/download-hf-model/0.log"
Sep 17 17:51:53.281346 ip-10-0-141-66 kubenswrapper[2566]: I0917 17:51:53.281315    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-w8wpl_1c4b7e49-7705-4fca-8c49-33ca5789eadd/seaweedfs/0.log"