Sep 16 18:38:34.838833 ip-10-0-128-247 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 16 18:38:34.838864 ip-10-0-128-247 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 16 18:38:34.838875 ip-10-0-128-247 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 16 18:38:34.839058 ip-10-0-128-247 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 16 18:38:44.983677 ip-10-0-128-247 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 16 18:38:44.983697 ip-10-0-128-247 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot ce74e621a65c4330bba586c2df2dc66c --
Sep 16 18:40:41.393342 ip-10-0-128-247 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 18:40:41.834624 ip-10-0-128-247 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:40:41.834624 ip-10-0-128-247 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 18:40:41.834624 ip-10-0-128-247 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:40:41.834624 ip-10-0-128-247 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 18:40:41.834624 ip-10-0-128-247 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:40:41.835331 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.834663    2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 18:40:41.837721 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837704    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:41.837721 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837721    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:41.837782 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837725    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:41.837782 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837728    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:41.837782 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837731    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:41.837782 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837733    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:41.837782 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837736    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:41.837782 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837739    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:41.837782 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837742    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:41.837782 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837744    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:41.837782 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837747    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:41.837782 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837749    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:41.837782 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837752    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:41.837782 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837754    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:41.837782 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837757    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:41.837782 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837760    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:41.837782 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837763    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:41.837782 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837765    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:41.837782 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837767    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:41.837782 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837770    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:41.837782 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837773    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:41.837782 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837775    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:41.838342 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837778    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:41.838342 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837780    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:41.838342 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837782    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:41.838342 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837785    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:41.838342 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837787    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:41.838342 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837790    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:41.838342 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837793    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:41.838342 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837796    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:41.838342 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837799    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:41.838342 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837801    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:41.838342 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837806    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:41.838342 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837809    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:41.838342 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837813    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:41.838342 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837816    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:41.838342 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837819    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:41.838342 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837822    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:41.838342 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837825    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:41.838342 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837828    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:41.838342 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837832    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:41.839119 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837835    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:41.839119 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837837    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:41.839119 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837840    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:41.839119 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837843    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:41.839119 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837845    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:41.839119 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837848    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:41.839119 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837856    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:41.839119 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837859    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:41.839119 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837862    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:41.839119 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837864    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:41.839119 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837867    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:41.839119 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837870    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:41.839119 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837872    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:41.839119 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837878    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:41.839119 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837881    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:41.839119 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837883    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:41.839119 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837886    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:41.839119 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837888    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:41.839119 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837891    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:41.839747 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837893    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:41.839747 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837897    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:41.839747 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837899    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:41.839747 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837901    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:41.839747 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837904    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:41.839747 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837906    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:41.839747 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837909    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:41.839747 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837911    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:41.839747 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837913    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:41.839747 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837916    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:41.839747 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837918    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:41.839747 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837920    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:41.839747 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837923    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:41.839747 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837925    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:41.839747 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837927    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:41.839747 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837930    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:41.839747 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837932    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:41.839747 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837934    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:41.839747 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837937    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:41.839747 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837939    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:41.840261 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837944    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:41.840261 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837946    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:41.840261 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837949    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:41.840261 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837953    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:41.840261 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837956    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:41.840261 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837959    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:41.840261 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837962    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:41.840261 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837965    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:41.840261 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837968    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:41.840261 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837970    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:41.840261 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837973    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:41.840261 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837975    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:41.840261 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837978    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:41.840261 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837980    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:41.840261 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837982    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:41.840261 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837985    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:41.840261 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837987    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:41.840261 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837990    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:41.840261 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.837992    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:41.840261 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.839945    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:41.840742 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.839957    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:41.840742 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.839960    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:41.840742 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.839963    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:41.840742 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.839966    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:41.840742 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.839969    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:41.840742 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.839973    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:41.840742 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.839976    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:41.840742 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.839979    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:41.840742 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.839981    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:41.840742 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.839984    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:41.840742 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.839987    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:41.840742 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.839990    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:41.840742 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.839992    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:41.840742 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.839995    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:41.840742 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.839997    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:41.840742 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840000    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:41.840742 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840002    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:41.840742 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840007    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:41.840742 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840010    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:41.841262 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840013    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:41.841262 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840016    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:41.841262 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840019    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:41.841262 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840023    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:41.841262 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840027    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:41.841262 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840030    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:41.841262 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840034    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:41.841262 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840037    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:41.841262 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840039    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:41.841262 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840042    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:41.841262 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840045    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:41.841262 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840047    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:41.841262 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840050    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:41.841262 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840052    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:41.841262 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840076    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:41.841262 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840080    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:41.841262 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840083    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:41.841262 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840086    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:41.841262 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840088    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:41.841771 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840092    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:41.841771 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840096    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:41.841771 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840099    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:41.841771 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840102    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:41.841771 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840104    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:41.841771 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840107    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:41.841771 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840109    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:41.841771 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840112    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:41.841771 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840114    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:41.841771 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840117    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:41.841771 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840119    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:41.841771 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840122    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:41.841771 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840125    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:41.841771 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840127    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:41.841771 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840130    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:41.841771 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840132    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:41.841771 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840134    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:41.841771 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840137    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:41.841771 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840140    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:41.841771 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840143    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:41.842396 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840146    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:41.842396 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840148    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:41.842396 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840151    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:41.842396 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840153    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:41.842396 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840156    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:41.842396 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840158    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:41.842396 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840161    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:41.842396 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840163    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:41.842396 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840166    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:41.842396 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840170    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:41.842396 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840172    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:41.842396 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840175    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:41.842396 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840177    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:41.842396 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840180    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:41.842396 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840182    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:41.842396 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840185    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:41.842396 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840187    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:41.842396 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840189    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:41.842396 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840192    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:41.842396 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840194    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:41.842890 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840196    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:41.842890 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840199    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:41.842890 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840201    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:41.842890 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840204    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:41.842890 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840213    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:41.842890 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840216    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:41.842890 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840218    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:41.842890 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840221    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:41.842890 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840223    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:41.842890 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840226    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:41.842890 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840229    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:41.842890 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840232    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:41.842890 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840235    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:41.842890 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840238    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:41.842890 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840240    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:41.842890 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840243    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:41.842890 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840245    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:41.842890 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840247    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:41.842890 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840250    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:41.843391 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840252    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:41.843391 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840629    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:41.843391 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840634    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:41.843391 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840636    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:41.843391 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840639    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:41.843391 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840641    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:41.843391 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840644    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:41.843391 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840647    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:41.843391 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840649    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:41.843391 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840652    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:41.843391 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840654    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:41.843391 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840657    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:41.843391 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840659    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:41.843391 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840662    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:41.843391 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840664    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:41.843391 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840667    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:41.843391 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840669    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:41.843391 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840672    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:41.843391 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840675    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:41.843391 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840678    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:41.843872 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840680    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:41.843872 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840683    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:41.843872 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840685    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:41.843872 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840688    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:41.843872 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840691    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:41.843872 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840693    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:41.843872 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840696    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:41.843872 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840698    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:41.843872 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840701    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:41.843872 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840704    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:41.843872 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840706    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:41.843872 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840709    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:41.843872 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840711    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:41.843872 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840713    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:41.843872 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840716    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:41.843872 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840718    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:41.843872 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840720    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:41.843872 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840722    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:41.843872 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840725    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:41.844368 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840728    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:41.844368 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840730    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:41.844368 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840733    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:41.844368 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840735    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:41.844368 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840738    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:41.844368 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840740    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:41.844368 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840744    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:41.844368 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840747    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:41.844368 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840750    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:41.844368 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840753    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:41.844368 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840755    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:41.844368 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840758    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:41.844368 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840762    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:41.844368 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840765    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:41.844368 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840768    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:41.844368 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840771    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:41.844368 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840775    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:41.844368 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840778    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:41.844368 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840781    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:41.844836 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840784    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:41.844836 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840787    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:41.844836 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840789    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:41.844836 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840792    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:41.844836 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840794    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:41.844836 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840797    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:41.844836 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840799    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:41.844836 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840802    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:41.844836 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840804    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:41.844836 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840807    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:41.844836 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840809    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:41.844836 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840812    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:41.844836 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840814    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:41.844836 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840817    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:41.844836 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840819    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:41.844836 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840821    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:41.844836 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840824    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:41.844836 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840826    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:41.844836 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840829    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:41.844836 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840831    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:41.845344 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840833    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:41.845344 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840836    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:41.845344 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840838    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:41.845344 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840841    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:41.845344 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840843    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:41.845344 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840846    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:41.845344 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840849    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:41.845344 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840852    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:41.845344 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840855    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:41.845344 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840858    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:41.845344 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840861    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:41.845344 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840863    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:41.845344 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840866    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:41.845344 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840868    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:41.845344 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840870    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:41.845344 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840873    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:41.845344 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840875    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:41.845344 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840879    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:41.845344 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840882    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:41.845800 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840885    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:41.845800 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840887    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:41.845800 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.840890    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:41.845800 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.840960    2577 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 18:40:41.845800 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841000    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 18:40:41.845800 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841009    2577 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 18:40:41.845800 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841014    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 18:40:41.845800 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841018    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 18:40:41.845800 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841021    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 18:40:41.845800 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841025    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 18:40:41.845800 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841030    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 18:40:41.845800 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841033    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 18:40:41.845800 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841035    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 18:40:41.845800 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841039    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 18:40:41.845800 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841042    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 18:40:41.845800 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841045    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 18:40:41.845800 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841048    2577 flags.go:64] FLAG: --cgroup-root=""
Sep 16 18:40:41.845800 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841051    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 18:40:41.845800 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841073    2577 flags.go:64] FLAG: --client-ca-file=""
Sep 16 18:40:41.845800 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841078    2577 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 18:40:41.845800 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841081    2577 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 18:40:41.845800 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841085    2577 flags.go:64] FLAG: --cluster-domain=""
Sep 16 18:40:41.845800 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841088    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 18:40:41.845800 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841091    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 18:40:41.846443 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841094    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 18:40:41.846443 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841097    2577 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 18:40:41.846443 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841101    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 18:40:41.846443 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841104    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 18:40:41.846443 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841107    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 18:40:41.846443 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841110    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 18:40:41.846443 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841113    2577 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 18:40:41.846443 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841116    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 18:40:41.846443 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841119    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 18:40:41.846443 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841122    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 18:40:41.846443 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841125    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 18:40:41.846443 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841129    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 18:40:41.846443 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841132    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 18:40:41.846443 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841134    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 18:40:41.846443 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841137    2577 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 18:40:41.846443 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841140    2577 flags.go:64] FLAG: --enable-server="true"
Sep 16 18:40:41.846443 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841143    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 18:40:41.846443 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841147    2577 flags.go:64] FLAG: --event-burst="100"
Sep 16 18:40:41.846443 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841150    2577 flags.go:64] FLAG: --event-qps="50"
Sep 16 18:40:41.846443 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841153    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 18:40:41.846443 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841156    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 18:40:41.846443 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841158    2577 flags.go:64] FLAG: --eviction-hard=""
Sep 16 18:40:41.846443 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841165    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 18:40:41.846443 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841168    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 18:40:41.847019 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841171    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 18:40:41.847019 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841174    2577 flags.go:64] FLAG: --eviction-soft=""
Sep 16 18:40:41.847019 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841177    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 18:40:41.847019 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841180    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 18:40:41.847019 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841184    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 18:40:41.847019 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841187    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 18:40:41.847019 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841190    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 18:40:41.847019 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841193    2577 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 18:40:41.847019 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841196    2577 flags.go:64] FLAG: --feature-gates=""
Sep 16 18:40:41.847019 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841199    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 18:40:41.847019 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841202    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 18:40:41.847019 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841205    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 18:40:41.847019 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841208    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 18:40:41.847019 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841211    2577 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 18:40:41.847019 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841214    2577 flags.go:64] FLAG: --help="false"
Sep 16 18:40:41.847019 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841217    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-128-247.ec2.internal"
Sep 16 18:40:41.847019 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841219    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 18:40:41.847019 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841222    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 18:40:41.847019 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841225    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 18:40:41.847019 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841228    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 18:40:41.847019 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841232    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 18:40:41.847019 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841234    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 18:40:41.847019 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841237    2577 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 18:40:41.847654 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841240    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 18:40:41.847654 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841243    2577 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 18:40:41.847654 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841246    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 18:40:41.847654 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841249    2577 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 18:40:41.847654 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841251    2577 flags.go:64] FLAG: --kube-reserved=""
Sep 16 18:40:41.847654 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841254    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 18:40:41.847654 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841257    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 18:40:41.847654 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841261    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 18:40:41.847654 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841264    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 18:40:41.847654 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841266    2577 flags.go:64] FLAG: --lock-file=""
Sep 16 18:40:41.847654 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841269    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 18:40:41.847654 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841272    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 18:40:41.847654 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841275    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 18:40:41.847654 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841280    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 18:40:41.847654 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841284    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 18:40:41.847654 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841287    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 18:40:41.847654 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841290    2577 flags.go:64] FLAG: --logging-format="text"
Sep 16 18:40:41.847654 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841293    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 18:40:41.847654 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841296    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 18:40:41.847654 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841299    2577 flags.go:64] FLAG: --manifest-url=""
Sep 16 18:40:41.847654 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841301    2577 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 18:40:41.847654 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841305    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 18:40:41.847654 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841308    2577 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 18:40:41.847654 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841312    2577 flags.go:64] FLAG: --max-pods="110"
Sep 16 18:40:41.847654 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841315    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 18:40:41.848293 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841317    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 18:40:41.848293 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841320    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 18:40:41.848293 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841323    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 18:40:41.848293 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841326    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 18:40:41.848293 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841328    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 18:40:41.848293 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841331    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 18:40:41.848293 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841339    2577 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 18:40:41.848293 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841341    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 18:40:41.848293 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841344    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 18:40:41.848293 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841347    2577 flags.go:64] FLAG: --pod-cidr=""
Sep 16 18:40:41.848293 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841350    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 18:40:41.848293 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841355    2577 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 18:40:41.848293 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841358    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 18:40:41.848293 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841361    2577 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 18:40:41.848293 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841364    2577 flags.go:64] FLAG: --port="10250"
Sep 16 18:40:41.848293 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841367    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 18:40:41.848293 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841370    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c0ad7a50bfd48aa4"
Sep 16 18:40:41.848293 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841373    2577 flags.go:64] FLAG: --qos-reserved=""
Sep 16 18:40:41.848293 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841376    2577 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 18:40:41.848293 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841379    2577 flags.go:64] FLAG: --register-node="true"
Sep 16 18:40:41.848293 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841381    2577 flags.go:64] FLAG: --register-with-taints=""
Sep 16 18:40:41.848293 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841385    2577 flags.go:64] FLAG: --registry-burst="10"
Sep 16 18:40:41.848293 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841390    2577 flags.go:64] FLAG: --registry-qps="5"
Sep 16 18:40:41.848293 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841393    2577 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 18:40:41.848896 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841395    2577 flags.go:64] FLAG: --reserved-memory=""
Sep 16 18:40:41.848896 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841399    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 18:40:41.848896 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841402    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 18:40:41.848896 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841405    2577 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 18:40:41.848896 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841408    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 18:40:41.848896 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841410    2577 flags.go:64] FLAG: --runonce="false"
Sep 16 18:40:41.848896 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841413    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 18:40:41.848896 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841416    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 18:40:41.848896 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841419    2577 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 18:40:41.848896 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841421    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 18:40:41.848896 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841424    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 18:40:41.848896 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841427    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 18:40:41.848896 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841430    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 18:40:41.848896 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841433    2577 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 18:40:41.848896 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841436    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 18:40:41.848896 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841439    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 18:40:41.848896 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841442    2577 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 18:40:41.848896 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841444    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 18:40:41.848896 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841447    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 18:40:41.848896 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841450    2577 flags.go:64] FLAG: --system-cgroups=""
Sep 16 18:40:41.848896 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841453    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 18:40:41.848896 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841458    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 18:40:41.848896 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841461    2577 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 18:40:41.848896 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841463    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 18:40:41.848896 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841468    2577 flags.go:64] FLAG: --tls-min-version=""
Sep 16 18:40:41.849562 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841470    2577 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 18:40:41.849562 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841473    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 18:40:41.849562 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841476    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 18:40:41.849562 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841478    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 18:40:41.849562 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841481    2577 flags.go:64] FLAG: --v="2"
Sep 16 18:40:41.849562 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841487    2577 flags.go:64] FLAG: --version="false"
Sep 16 18:40:41.849562 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841492    2577 flags.go:64] FLAG: --vmodule=""
Sep 16 18:40:41.849562 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841496    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 18:40:41.849562 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.841499    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 18:40:41.849562 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841615    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:41.849562 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841620    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:41.849562 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841623    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:41.849562 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841626    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:41.849562 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841629    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:41.849562 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841632    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:41.849562 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841635    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:41.849562 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841638    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:41.849562 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841641    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:41.849562 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841644    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:41.849562 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841647    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:41.849562 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841650    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:41.850106 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841653    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:41.850106 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841655    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:41.850106 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841657    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:41.850106 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841660    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:41.850106 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841662    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:41.850106 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841665    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:41.850106 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841667    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:41.850106 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841670    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:41.850106 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841673    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:41.850106 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841675    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:41.850106 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841678    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:41.850106 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841680    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:41.850106 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841683    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:41.850106 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841685    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:41.850106 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841688    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:41.850106 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841690    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:41.850106 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841694    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:41.850106 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841696    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:41.850106 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841699    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:41.850106 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841701    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:41.850600 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841704    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:41.850600 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841706    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:41.850600 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841711    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:41.850600 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841714    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:41.850600 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841717    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:41.850600 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841719    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:41.850600 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841722    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:41.850600 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841725    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:41.850600 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841727    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:41.850600 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841730    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:41.850600 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841732    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:41.850600 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841734    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:41.850600 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841737    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:41.850600 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841740    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:41.850600 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841742    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:41.850600 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841745    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:41.850600 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841747    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:41.850600 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841750    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:41.850600 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841753    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:41.850600 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841755    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:41.851133 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841757    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:41.851133 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841760    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:41.851133 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841763    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:41.851133 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841765    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:41.851133 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841767    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:41.851133 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841770    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:41.851133 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841773    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:41.851133 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841775    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:41.851133 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841779    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:41.851133 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841781    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:41.851133 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841784    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:41.851133 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841787    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:41.851133 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841790    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:41.851133 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841793    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:41.851133 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841795    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:41.851133 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841797    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:41.851133 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841800    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:41.851133 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841802    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:41.851133 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841805    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:41.851133 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841808    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:41.851631 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841810    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:41.851631 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841812    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:41.851631 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841815    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:41.851631 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841818    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:41.851631 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841822    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:41.851631 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841825    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:41.851631 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841827    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:41.851631 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841830    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:41.851631 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841832    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:41.851631 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841834    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:41.851631 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841837    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:41.851631 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841840    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:41.851631 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841844    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:41.851631 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841846    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:41.851631 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841849    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:41.851631 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841851    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:41.851631 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841853    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:41.851631 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841856    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:41.851631 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841858    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:41.851631 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841861    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:41.852152 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841865    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:41.852152 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841868    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:41.852152 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841871    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:41.852152 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841873    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:41.852152 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841877    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:41.852152 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841879    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:41.852152 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.841882    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:41.852152 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.842398    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:40:41.852152 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.850302    2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 18:40:41.852152 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.850317    2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 18:40:41.852152 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.850336    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 18:40:41.852152 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.850342    2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 18:40:41.852152 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850402    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:41.852152 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850407    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:41.852152 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850410    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:41.852539 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850414    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:41.852539 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850417    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:41.852539 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850421    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:41.852539 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850424    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:41.852539 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850427    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:41.852539 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850430    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:41.852539 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850433    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:41.852539 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850435    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:41.852539 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850437    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:41.852539 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850440    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:41.852539 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850443    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:41.852539 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850445    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:41.852539 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850448    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:41.852539 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850450    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:41.852539 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850453    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:41.852539 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850455    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:41.852539 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850457    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:41.852539 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850460    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:41.852539 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850463    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:41.853005 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850465    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:41.853005 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850467    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:41.853005 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850470    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:41.853005 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850473    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:41.853005 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850475    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:41.853005 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850478    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:41.853005 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850480    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:41.853005 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850486    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:41.853005 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850489    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:41.853005 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850492    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:41.853005 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850494    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:41.853005 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850497    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:41.853005 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850499    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:41.853005 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850502    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:41.853005 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850505    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:41.853005 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850507    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:41.853005 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850510    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:41.853005 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850513    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:41.853005 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850515    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:41.853487 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850517    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:41.853487 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850520    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:41.853487 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850522    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:41.853487 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850525    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:41.853487 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850527    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:41.853487 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850530    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:41.853487 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850532    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:41.853487 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850535    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:41.853487 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850537    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:41.853487 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850539    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:41.853487 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850542    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:41.853487 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850544    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:41.853487 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850547    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:41.853487 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850550    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:41.853487 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850552    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:41.853487 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850555    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:41.853487 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850557    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:41.853487 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850560    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:41.853487 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850562    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:41.853487 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850565    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:41.853968 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850567    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:41.853968 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850570    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:41.853968 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850573    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:41.853968 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850577    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:41.853968 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850580    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:41.853968 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850583    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:41.853968 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850586    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:41.853968 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850589    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:41.853968 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850591    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:41.853968 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850594    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:41.853968 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850597    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:41.853968 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850599    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:41.853968 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850602    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:41.853968 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850605    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:41.853968 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850608    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:41.853968 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850611    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:41.853968 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850614    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:41.853968 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850617    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:41.853968 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850619    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:41.853968 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850622    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:41.854504 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850624    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:41.854504 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850627    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:41.854504 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850629    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:41.854504 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850632    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:41.854504 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850634    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:41.854504 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850637    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:41.854504 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850640    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:41.854504 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850642    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:41.854504 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850645    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:41.854504 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850647    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:41.854504 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850651    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:41.854504 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850654    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:41.854504 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850657    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:41.854504 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850659    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:41.854504 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850662    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:41.854504 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850665    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:41.854504 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850667    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:41.854504 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850669    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:41.854944 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.850674    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:40:41.854944 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850802    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:41.854944 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850807    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:41.854944 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850810    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:41.854944 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850812    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:41.854944 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850815    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:41.854944 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850817    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:41.854944 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850820    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:41.854944 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850823    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:41.854944 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850825    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:41.854944 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850828    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:41.854944 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850830    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:41.854944 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850833    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:41.854944 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850835    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:41.854944 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850838    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:41.855348 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850840    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:41.855348 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850842    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:41.855348 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850845    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:41.855348 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850847    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:41.855348 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850850    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:41.855348 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850853    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:41.855348 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850855    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:41.855348 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850858    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:41.855348 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850860    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:41.855348 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850863    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:41.855348 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850866    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:41.855348 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850868    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:41.855348 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850871    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:41.855348 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850873    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:41.855348 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850876    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:41.855348 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850878    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:41.855348 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850881    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:41.855348 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850884    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:41.855348 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850886    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:41.855348 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850888    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:41.855839 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850891    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:41.855839 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850893    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:41.855839 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850896    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:41.855839 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850898    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:41.855839 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850901    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:41.855839 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850903    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:41.855839 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850906    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:41.855839 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850908    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:41.855839 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850911    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:41.855839 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850913    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:41.855839 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850916    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:41.855839 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850919    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:41.855839 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850922    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:41.855839 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850924    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:41.855839 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850927    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:41.855839 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850930    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:41.855839 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850932    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:41.855839 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850935    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:41.855839 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850938    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:41.856327 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850940    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:41.856327 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850943    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:41.856327 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850945    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:41.856327 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850948    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:41.856327 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850950    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:41.856327 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850953    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:41.856327 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850957    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:41.856327 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850960    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:41.856327 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850966    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:41.856327 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850969    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:41.856327 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850973    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:41.856327 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850977    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:41.856327 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850979    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:41.856327 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850982    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:41.856327 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850985    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:41.856327 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850987    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:41.856327 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850990    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:41.856327 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850993    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:41.856327 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850995    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:41.856781 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.850997    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:41.856781 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.851000    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:41.856781 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.851002    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:41.856781 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.851005    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:41.856781 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.851007    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:41.856781 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.851010    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:41.856781 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.851012    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:41.856781 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.851015    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:41.856781 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.851017    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:41.856781 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.851020    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:41.856781 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.851022    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:41.856781 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.851025    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:41.856781 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.851027    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:41.856781 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.851030    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:41.856781 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.851032    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:41.856781 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.851035    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:41.856781 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.851037    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:41.856781 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.851040    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:41.856781 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.851042    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:41.857252 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.851046    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:41.857252 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.851050    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:41.857252 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.851052    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:41.857252 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.851080    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:41.857252 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.851083    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:41.857252 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.851086    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:41.857252 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.851089    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:41.857252 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:41.851091    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:41.857252 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.851096    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:40:41.857252 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.851232    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 18:40:41.857252 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.853194    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 18:40:41.857252 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.854175    2577 server.go:1019] "Starting client certificate rotation"
Sep 16 18:40:41.857252 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.854274    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:40:41.857252 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.855086    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:40:41.857605 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.856490    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:40:41.862026 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.862004    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:40:41.871125 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.871106    2577 log.go:25] "Validated CRI v1 runtime API"
Sep 16 18:40:41.873892 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.873876    2577 log.go:25] "Validated CRI v1 image API"
Sep 16 18:40:41.874708 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.874693    2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 18:40:41.876961 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.876936    2577 fs.go:135] Filesystem UUIDs: map[6d3fe219-3bc9-4c01-b097-8fa86678e13b:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 fb7e6803-c825-4830-8607-ef32ef617e35:/dev/nvme0n1p3]
Sep 16 18:40:41.877027 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.876960    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 18:40:41.881010 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.880991    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:40:41.883126 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.883001    2577 manager.go:216] Machine: {Timestamp:2026-09-16 18:40:41.881770761 +0000 UTC m=+0.377130535 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099385 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2866f88297531598290157f3f5ffdd SystemUUID:ec2866f8-8297-5315-9829-0157f3f5ffdd BootID:ce74e621-a65c-4330-bba5-86c2df2dc66c Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:dd:b0:04:09 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:dd:b0:04:09 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:de:5c:7b:3e:6f:16 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 18:40:41.883126 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.883117    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 18:40:41.883237 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.883189    2577 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 18:40:41.886282 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.886263    2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 18:40:41.886417 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.886285    2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-247.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 18:40:41.887043 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.887034    2577 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 18:40:41.887086 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.887046    2577 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 18:40:41.887115 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.887103    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:40:41.887955 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.887945    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:40:41.888710 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.888701    2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 18:40:41.892316 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.892305    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:40:41.892434 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.892424    2577 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 18:40:41.892469 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.892455    2577 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 18:40:41.892469 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.892468    2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 18:40:41.892523 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.892488    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 18:40:41.892523 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.892498    2577 kubelet.go:398] "Adding apiserver pod source"
Sep 16 18:40:41.892523 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.892510    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 18:40:41.892881 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.892864    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:40:41.892930 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.892888    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:40:41.895071 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.895036    2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 18:40:41.897216 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.897203    2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 18:40:41.897258 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.897233    2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 18:40:41.899016 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.899003    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 18:40:41.899049 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.899028    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 18:40:41.899049 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.899038    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 18:40:41.899049 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.899046    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 18:40:41.899145 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.899075    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 18:40:41.899145 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.899084    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 18:40:41.899145 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.899098    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 18:40:41.899145 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.899103    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 18:40:41.899145 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.899112    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 18:40:41.899145 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.899119    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 18:40:41.899145 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.899134    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 18:40:41.899145 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.899139    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 18:40:41.899357 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.899162    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 18:40:41.899357 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.899170    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 18:40:41.902767 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.902754    2577 server.go:1268] "Started kubelet"
Sep 16 18:40:41.902848 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.902816    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 18:40:41.903411 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:41.903353    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 18:40:41.903411 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.903405    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-247.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 18:40:41.903552 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:41.903509    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-247.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 18:40:41.903620 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.903558    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 18:40:41.903620 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.903616    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 18:40:41.903678 ip-10-0-128-247 systemd[1]: Started Kubernetes Kubelet.
Sep 16 18:40:41.904781 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.904621    2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 18:40:41.908720 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.908699    2577 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 18:40:41.908880 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:41.908856    2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 18:40:41.911214 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:41.910391    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-247.ec2.internal.18d5e1345951d847  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-247.ec2.internal,UID:ip-10-0-128-247.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-247.ec2.internal,},FirstTimestamp:2026-09-16 18:40:41.902725191 +0000 UTC m=+0.398084964,LastTimestamp:2026-09-16 18:40:41.902725191 +0000 UTC m=+0.398084964,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-247.ec2.internal,}"
Sep 16 18:40:41.912660 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.912643    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 18:40:41.913282 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.913261    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 18:40:41.913558 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:41.913538    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-247.ec2.internal\" not found"
Sep 16 18:40:41.913897 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.913879    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 18:40:41.913897 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.913880    2577 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 18:40:41.914031 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.913906    2577 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 18:40:41.914031 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.913992    2577 factory.go:55] Registering systemd factory
Sep 16 18:40:41.914031 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.914007    2577 factory.go:223] Registration of the systemd container factory successfully
Sep 16 18:40:41.914031 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.914015    2577 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 18:40:41.914031 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.914023    2577 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 18:40:41.914267 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.914248    2577 factory.go:153] Registering CRI-O factory
Sep 16 18:40:41.914267 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.914259    2577 factory.go:223] Registration of the crio container factory successfully
Sep 16 18:40:41.914448 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.914306    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 18:40:41.914448 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.914321    2577 factory.go:103] Registering Raw factory
Sep 16 18:40:41.914448 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.914330    2577 manager.go:1196] Started watching for new ooms in manager
Sep 16 18:40:41.914715 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.914702    2577 manager.go:349] Starting recovery of all containers
Sep 16 18:40:41.917448 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.917259    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-42fbm"
Sep 16 18:40:41.924441 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.924420    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-42fbm"
Sep 16 18:40:41.924925 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.924904    2577 manager.go:354] Recovery completed
Sep 16 18:40:41.925711 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:41.925688    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 18:40:41.925799 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:41.925778    2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-247.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 18:40:41.929199 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.929187    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:41.930858 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.930835    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-247.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:41.930858 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.930860    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-247.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:41.930958 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.930869    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-247.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:41.931840 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.931826    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 18:40:41.931840 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.931838    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 18:40:41.931840 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.931841    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:41.932008 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.931858    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:40:41.932535 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.932520    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-247.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:41.932601 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.932546    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-247.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:41.932601 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.932559    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-247.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:41.933129 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.933116    2577 policy_none.go:49] "None policy: Start"
Sep 16 18:40:41.933210 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.933135    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 18:40:41.933210 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.933148    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 18:40:41.934431 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.934416    2577 policy_none.go:47] "Start"
Sep 16 18:40:41.973691 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.973675    2577 manager.go:337] "Starting Device Plugin manager"
Sep 16 18:40:41.987411 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:41.973716    2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 18:40:41.987411 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.974104    2577 server.go:87] "Starting device plugin registration server"
Sep 16 18:40:41.987411 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.974539    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 18:40:41.987411 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.974567    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 18:40:41.987411 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.974909    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 18:40:41.987411 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.974994    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 18:40:41.987411 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:41.975004    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 18:40:41.987411 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:41.976547    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 18:40:41.987411 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:41.976602    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-247.ec2.internal\" not found"
Sep 16 18:40:42.035295 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.035252    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 18:40:42.036616 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.036599    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 18:40:42.036711 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.036645    2577 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 18:40:42.036711 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.036666    2577 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 18:40:42.036711 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:42.036707    2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 18:40:42.038991 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.038969    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:42.074882 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.074852    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:42.075685 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.075672    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-247.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:42.075751 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.075698    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-247.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:42.075751 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.075708    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-247.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:42.075751 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.075731    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-247.ec2.internal"
Sep 16 18:40:42.093435 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.093371    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-247.ec2.internal"
Sep 16 18:40:42.093435 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:42.093392    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-247.ec2.internal\": node \"ip-10-0-128-247.ec2.internal\" not found"
Sep 16 18:40:42.109982 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:42.109952    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-247.ec2.internal\" not found"
Sep 16 18:40:42.137393 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.137371    2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-247.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-247.ec2.internal"]
Sep 16 18:40:42.137453 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.137439    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:42.138852 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.138836    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-247.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:42.138920 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.138863    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-247.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:42.138920 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.138873    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-247.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:42.140354 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.140342    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:42.140512 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.140494    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-247.ec2.internal"
Sep 16 18:40:42.140596 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.140525    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:42.141109 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.141087    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-247.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:42.141197 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.141088    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-247.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:42.141197 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.141135    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-247.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:42.141197 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.141144    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-247.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:42.141197 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.141116    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-247.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:42.141197 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.141188    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-247.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:42.142440 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.142427    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-247.ec2.internal"
Sep 16 18:40:42.142496 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.142449    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:42.143177 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.143159    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-247.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:42.143268 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.143183    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-247.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:42.143268 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.143193    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-247.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:42.159609 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:42.159592    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-247.ec2.internal\" not found" node="ip-10-0-128-247.ec2.internal"
Sep 16 18:40:42.163864 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:42.163849    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-247.ec2.internal\" not found" node="ip-10-0-128-247.ec2.internal"
Sep 16 18:40:42.210546 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:42.210526    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-247.ec2.internal\" not found"
Sep 16 18:40:42.215260 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.215245    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c28a4ec2fbe427bdfc7c14e0446ecb67-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-247.ec2.internal\" (UID: \"c28a4ec2fbe427bdfc7c14e0446ecb67\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-247.ec2.internal"
Sep 16 18:40:42.215333 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.215269    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/146a57ab6e613bbac55bc1f7a3764a51-config\") pod \"kube-apiserver-proxy-ip-10-0-128-247.ec2.internal\" (UID: \"146a57ab6e613bbac55bc1f7a3764a51\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-247.ec2.internal"
Sep 16 18:40:42.215333 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.215293    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c28a4ec2fbe427bdfc7c14e0446ecb67-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-247.ec2.internal\" (UID: \"c28a4ec2fbe427bdfc7c14e0446ecb67\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-247.ec2.internal"
Sep 16 18:40:42.310933 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:42.310908    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-247.ec2.internal\" not found"
Sep 16 18:40:42.316264 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.316247    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c28a4ec2fbe427bdfc7c14e0446ecb67-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-247.ec2.internal\" (UID: \"c28a4ec2fbe427bdfc7c14e0446ecb67\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-247.ec2.internal"
Sep 16 18:40:42.316333 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.316270    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c28a4ec2fbe427bdfc7c14e0446ecb67-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-247.ec2.internal\" (UID: \"c28a4ec2fbe427bdfc7c14e0446ecb67\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-247.ec2.internal"
Sep 16 18:40:42.316333 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.316285    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/146a57ab6e613bbac55bc1f7a3764a51-config\") pod \"kube-apiserver-proxy-ip-10-0-128-247.ec2.internal\" (UID: \"146a57ab6e613bbac55bc1f7a3764a51\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-247.ec2.internal"
Sep 16 18:40:42.316432 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.316348    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/146a57ab6e613bbac55bc1f7a3764a51-config\") pod \"kube-apiserver-proxy-ip-10-0-128-247.ec2.internal\" (UID: \"146a57ab6e613bbac55bc1f7a3764a51\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-247.ec2.internal"
Sep 16 18:40:42.316432 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.316364    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c28a4ec2fbe427bdfc7c14e0446ecb67-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-247.ec2.internal\" (UID: \"c28a4ec2fbe427bdfc7c14e0446ecb67\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-247.ec2.internal"
Sep 16 18:40:42.316432 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.316365    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c28a4ec2fbe427bdfc7c14e0446ecb67-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-247.ec2.internal\" (UID: \"c28a4ec2fbe427bdfc7c14e0446ecb67\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-247.ec2.internal"
Sep 16 18:40:42.411894 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:42.411836    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-247.ec2.internal\" not found"
Sep 16 18:40:42.462272 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.462251    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-247.ec2.internal"
Sep 16 18:40:42.467698 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.467682    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-247.ec2.internal"
Sep 16 18:40:42.512536 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:42.512511    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-247.ec2.internal\" not found"
Sep 16 18:40:42.613016 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:42.612994    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-247.ec2.internal\" not found"
Sep 16 18:40:42.713461 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:42.713411    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-247.ec2.internal\" not found"
Sep 16 18:40:42.813961 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:42.813921    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-247.ec2.internal\" not found"
Sep 16 18:40:42.854272 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.854254    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 18:40:42.854675 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.854415    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:40:42.876035 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.876012    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:42.893436 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.893418    2577 apiserver.go:52] "Watching apiserver"
Sep 16 18:40:42.904240 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.904222    2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 18:40:42.904613 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.904589    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-8cxsn","openshift-network-diagnostics/network-check-target-ggvbf","openshift-network-operator/iptables-alerter-298hg","openshift-ovn-kubernetes/ovnkube-node-wcfql","kube-system/global-pull-secret-syncer-88mzt","openshift-multus/multus-additional-cni-plugins-btfcs","openshift-multus/multus-rlvsc","openshift-multus/network-metrics-daemon-s229p","kube-system/konnectivity-agent-jpq9w","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hv7z5","openshift-cluster-node-tuning-operator/tuned-vlcrr","openshift-dns/node-resolver-wmfjw"]
Sep 16 18:40:42.906228 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.906210    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8cxsn"
Sep 16 18:40:42.907706 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.907687    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggvbf"
Sep 16 18:40:42.907779 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:42.907759    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggvbf" podUID="8e33041b-cc50-41a3-bcc4-34942629f4cc"
Sep 16 18:40:42.908783 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.908762    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 18:40:42.908901 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.908884    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-298hg"
Sep 16 18:40:42.909174 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.909125    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-87cgd\""
Sep 16 18:40:42.909174 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.909132    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 18:40:42.909337 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.909194    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 18:40:42.910513 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.910499    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:42.911259 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.911245    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 18:40:42.911645 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.911631    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-92g55\""
Sep 16 18:40:42.911761 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.911746    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:40:42.911933 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.911916    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 18:40:42.911996 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.911986    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88mzt"
Sep 16 18:40:42.912167 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:42.912043    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88mzt" podUID="2b544a31-67f6-4a93-beeb-a21fd49b00b1"
Sep 16 18:40:42.912775 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.912743    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 18:40:42.912861 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.912847    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 18:40:42.913181 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.913145    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-btfcs"
Sep 16 18:40:42.914945 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.914000    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 18:40:42.914945 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.914756    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-247.ec2.internal"
Sep 16 18:40:42.915104 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.915086    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 18:40:42.915401 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.915381    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 18:40:42.915918 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.915893    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-s6pc4\""
Sep 16 18:40:42.916003 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.915967    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 18:40:42.916869 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.916041    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 18:40:42.917406 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.917090    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 18:40:42.917513 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.917461    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:42.918899 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.918883    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s229p"
Sep 16 18:40:42.918968 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:42.918930    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s229p" podUID="ba8dcba9-de75-4055-8086-3820f53b55e3"
Sep 16 18:40:42.919297 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.919266    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/88b20525-d601-4224-bd43-397308547efd-ovnkube-script-lib\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:42.919369 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.919306    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8049c55c-c4ac-44a1-a1bf-5872846445c4-cnibin\") pod \"multus-additional-cni-plugins-btfcs\" (UID: \"8049c55c-c4ac-44a1-a1bf-5872846445c4\") " pod="openshift-multus/multus-additional-cni-plugins-btfcs"
Sep 16 18:40:42.919369 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.919318    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 18:40:42.919369 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.919332    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 18:40:42.919369 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.919332    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 18:40:42.919369 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.919323    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8049c55c-c4ac-44a1-a1bf-5872846445c4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-btfcs\" (UID: \"8049c55c-c4ac-44a1-a1bf-5872846445c4\") " pod="openshift-multus/multus-additional-cni-plugins-btfcs"
Sep 16 18:40:42.919653 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.919389    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8859cf81-dc3a-4a84-a649-efdaee26a8a2-system-cni-dir\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:42.919653 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.919414    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8859cf81-dc3a-4a84-a649-efdaee26a8a2-cni-binary-copy\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:42.919653 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.919433    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8859cf81-dc3a-4a84-a649-efdaee26a8a2-etc-kubernetes\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:42.919653 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.919472    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clzd5\" (UniqueName: \"kubernetes.io/projected/1e98550f-c964-434b-995d-60ad18764eff-kube-api-access-clzd5\") pod \"iptables-alerter-298hg\" (UID: \"1e98550f-c964-434b-995d-60ad18764eff\") " pod="openshift-network-operator/iptables-alerter-298hg"
Sep 16 18:40:42.919653 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.919499    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs64s\" (UniqueName: \"kubernetes.io/projected/f935c176-f27b-4179-bc5e-802ee61499d3-kube-api-access-rs64s\") pod \"node-ca-8cxsn\" (UID: \"f935c176-f27b-4179-bc5e-802ee61499d3\") " pod="openshift-image-registry/node-ca-8cxsn"
Sep 16 18:40:42.919653 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.919522    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2b544a31-67f6-4a93-beeb-a21fd49b00b1-kubelet-config\") pod \"global-pull-secret-syncer-88mzt\" (UID: \"2b544a31-67f6-4a93-beeb-a21fd49b00b1\") " pod="kube-system/global-pull-secret-syncer-88mzt"
Sep 16 18:40:42.919653 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.919536    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-run-systemd\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:42.919653 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.919555    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-var-lib-openvswitch\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:42.919653 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.919571    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-log-socket\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:42.919653 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.919592    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/88b20525-d601-4224-bd43-397308547efd-env-overrides\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:42.919653 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.919607    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8049c55c-c4ac-44a1-a1bf-5872846445c4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-btfcs\" (UID: \"8049c55c-c4ac-44a1-a1bf-5872846445c4\") " pod="openshift-multus/multus-additional-cni-plugins-btfcs"
Sep 16 18:40:42.919653 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.919620    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2b544a31-67f6-4a93-beeb-a21fd49b00b1-dbus\") pod \"global-pull-secret-syncer-88mzt\" (UID: \"2b544a31-67f6-4a93-beeb-a21fd49b00b1\") " pod="kube-system/global-pull-secret-syncer-88mzt"
Sep 16 18:40:42.919653 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.919637    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-systemd-units\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:42.919653 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.919661    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-host-run-ovn-kubernetes\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:42.920383 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.919686    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw4tm\" (UniqueName: \"kubernetes.io/projected/88b20525-d601-4224-bd43-397308547efd-kube-api-access-vw4tm\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:42.920383 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.919701    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8859cf81-dc3a-4a84-a649-efdaee26a8a2-cnibin\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:42.920383 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.919738    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8859cf81-dc3a-4a84-a649-efdaee26a8a2-host-run-netns\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:42.920383 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.919760    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8859cf81-dc3a-4a84-a649-efdaee26a8a2-host-var-lib-kubelet\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:42.920383 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.919774    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8859cf81-dc3a-4a84-a649-efdaee26a8a2-multus-daemon-config\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:42.920383 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.919787    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f935c176-f27b-4179-bc5e-802ee61499d3-host\") pod \"node-ca-8cxsn\" (UID: \"f935c176-f27b-4179-bc5e-802ee61499d3\") " pod="openshift-image-registry/node-ca-8cxsn"
Sep 16 18:40:42.920383 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.919810    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b544a31-67f6-4a93-beeb-a21fd49b00b1-original-pull-secret\") pod \"global-pull-secret-syncer-88mzt\" (UID: \"2b544a31-67f6-4a93-beeb-a21fd49b00b1\") " pod="kube-system/global-pull-secret-syncer-88mzt"
Sep 16 18:40:42.920383 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.919833    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-host-cni-netd\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:42.920383 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.919851    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8859cf81-dc3a-4a84-a649-efdaee26a8a2-multus-socket-dir-parent\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:42.920383 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.919865    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8859cf81-dc3a-4a84-a649-efdaee26a8a2-host-run-k8s-cni-cncf-io\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:42.920383 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.919877    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8859cf81-dc3a-4a84-a649-efdaee26a8a2-host-var-lib-cni-multus\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:42.920383 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.919891    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8859cf81-dc3a-4a84-a649-efdaee26a8a2-multus-conf-dir\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:42.920383 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.919924    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1e98550f-c964-434b-995d-60ad18764eff-host-slash\") pod \"iptables-alerter-298hg\" (UID: \"1e98550f-c964-434b-995d-60ad18764eff\") " pod="openshift-network-operator/iptables-alerter-298hg"
Sep 16 18:40:42.920383 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.919951    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-host-slash\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:42.920383 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.919976    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-etc-openvswitch\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:42.920383 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.919997    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-node-log\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:42.920383 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.920018    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:42.921009 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.920033    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8049c55c-c4ac-44a1-a1bf-5872846445c4-cni-binary-copy\") pod \"multus-additional-cni-plugins-btfcs\" (UID: \"8049c55c-c4ac-44a1-a1bf-5872846445c4\") " pod="openshift-multus/multus-additional-cni-plugins-btfcs"
Sep 16 18:40:42.921009 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.920081    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8859cf81-dc3a-4a84-a649-efdaee26a8a2-host-var-lib-cni-bin\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:42.921009 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.920104    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt6sq\" (UniqueName: \"kubernetes.io/projected/8859cf81-dc3a-4a84-a649-efdaee26a8a2-kube-api-access-nt6sq\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:42.921009 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.920129    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f935c176-f27b-4179-bc5e-802ee61499d3-serviceca\") pod \"node-ca-8cxsn\" (UID: \"f935c176-f27b-4179-bc5e-802ee61499d3\") " pod="openshift-image-registry/node-ca-8cxsn"
Sep 16 18:40:42.921009 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.920142    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jpq9w"
Sep 16 18:40:42.921009 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.920152    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-run-ovn\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:42.921009 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.920178    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8049c55c-c4ac-44a1-a1bf-5872846445c4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-btfcs\" (UID: \"8049c55c-c4ac-44a1-a1bf-5872846445c4\") " pod="openshift-multus/multus-additional-cni-plugins-btfcs"
Sep 16 18:40:42.921009 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.920205    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plpqb\" (UniqueName: \"kubernetes.io/projected/8049c55c-c4ac-44a1-a1bf-5872846445c4-kube-api-access-plpqb\") pod \"multus-additional-cni-plugins-btfcs\" (UID: \"8049c55c-c4ac-44a1-a1bf-5872846445c4\") " pod="openshift-multus/multus-additional-cni-plugins-btfcs"
Sep 16 18:40:42.921009 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.920229    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8859cf81-dc3a-4a84-a649-efdaee26a8a2-hostroot\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:42.921009 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.920254    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-host-run-netns\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:42.921009 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.920276    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8859cf81-dc3a-4a84-a649-efdaee26a8a2-multus-cni-dir\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:42.921009 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.920301    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8859cf81-dc3a-4a84-a649-efdaee26a8a2-host-run-multus-certs\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:42.921009 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.920378    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-host-kubelet\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:42.921009 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.920404    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-run-openvswitch\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:42.921009 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.920426    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-host-cni-bin\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:42.921009 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.920447    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/88b20525-d601-4224-bd43-397308547efd-ovnkube-config\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:42.921009 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.920470    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/88b20525-d601-4224-bd43-397308547efd-ovn-node-metrics-cert\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:42.921636 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.920491    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8049c55c-c4ac-44a1-a1bf-5872846445c4-system-cni-dir\") pod \"multus-additional-cni-plugins-btfcs\" (UID: \"8049c55c-c4ac-44a1-a1bf-5872846445c4\") " pod="openshift-multus/multus-additional-cni-plugins-btfcs"
Sep 16 18:40:42.921636 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.920513    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5ttq\" (UniqueName: \"kubernetes.io/projected/8e33041b-cc50-41a3-bcc4-34942629f4cc-kube-api-access-r5ttq\") pod \"network-check-target-ggvbf\" (UID: \"8e33041b-cc50-41a3-bcc4-34942629f4cc\") " pod="openshift-network-diagnostics/network-check-target-ggvbf"
Sep 16 18:40:42.921636 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.920553    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1e98550f-c964-434b-995d-60ad18764eff-iptables-alerter-script\") pod \"iptables-alerter-298hg\" (UID: \"1e98550f-c964-434b-995d-60ad18764eff\") " pod="openshift-network-operator/iptables-alerter-298hg"
Sep 16 18:40:42.921636 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.921206    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-c7mv8\""
Sep 16 18:40:42.921636 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.921212    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 18:40:42.922319 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.921801    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hv7z5"
Sep 16 18:40:42.923148 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.923133    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vlcrr"
Sep 16 18:40:42.924519 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.924502    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-crk2x\""
Sep 16 18:40:42.924756 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.924744    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wmfjw"
Sep 16 18:40:42.925288 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.925273    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 18:40:42.925637 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.925624    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mgr49\""
Sep 16 18:40:42.925892 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.925877    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 18:40:42.925944 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.925918    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 18:40:42.927311 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.927295    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 18:40:42.927918 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.927905    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 18:40:42.928374 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.928322    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:35:43 +0000 UTC" deadline="2028-05-26 08:22:47.653597216 +0000 UTC"
Sep 16 18:40:42.928436 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.928376    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14821h42m4.725225262s"
Sep 16 18:40:42.928860 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.928835    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 18:40:42.929701 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.929686    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 18:40:42.929784 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.929731    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 18:40:42.929841 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.929810    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jr6mz\""
Sep 16 18:40:42.929950 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.929933    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:40:42.930044 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.929990    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wh7wb\""
Sep 16 18:40:42.930117 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.930085    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 18:40:42.930245 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.930233    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-p2qjd\""
Sep 16 18:40:42.936109 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.935932    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:40:42.936109 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.936017    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-247.ec2.internal"
Sep 16 18:40:42.936728 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.936714    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-247.ec2.internal"]
Sep 16 18:40:42.949233 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.949210    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:40:42.956172 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.956157    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:40:42.956315 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.956300    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-247.ec2.internal"]
Sep 16 18:40:42.965128 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:42.965076    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146a57ab6e613bbac55bc1f7a3764a51.slice/crio-a1697e0fdd3992ca89d29405baf969943049f9c0e05e11cbc0402699f092c5f6 WatchSource:0}: Error finding container a1697e0fdd3992ca89d29405baf969943049f9c0e05e11cbc0402699f092c5f6: Status 404 returned error can't find the container with id a1697e0fdd3992ca89d29405baf969943049f9c0e05e11cbc0402699f092c5f6
Sep 16 18:40:42.965360 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:42.965330    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc28a4ec2fbe427bdfc7c14e0446ecb67.slice/crio-6d2a332ec2633e1d5d142c671d902bed6dce61aa5a659b541125f25dcc1d6213 WatchSource:0}: Error finding container 6d2a332ec2633e1d5d142c671d902bed6dce61aa5a659b541125f25dcc1d6213: Status 404 returned error can't find the container with id 6d2a332ec2633e1d5d142c671d902bed6dce61aa5a659b541125f25dcc1d6213
Sep 16 18:40:42.970714 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.970698    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:40:42.978308 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.978292    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-s895x"
Sep 16 18:40:42.986316 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:42.986299    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-s895x"
Sep 16 18:40:43.014613 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.014596    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 18:40:43.021048 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021029    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-run-ovn\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:43.021149 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021066    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plpqb\" (UniqueName: \"kubernetes.io/projected/8049c55c-c4ac-44a1-a1bf-5872846445c4-kube-api-access-plpqb\") pod \"multus-additional-cni-plugins-btfcs\" (UID: \"8049c55c-c4ac-44a1-a1bf-5872846445c4\") " pod="openshift-multus/multus-additional-cni-plugins-btfcs"
Sep 16 18:40:43.021149 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021085    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba8dcba9-de75-4055-8086-3820f53b55e3-metrics-certs\") pod \"network-metrics-daemon-s229p\" (UID: \"ba8dcba9-de75-4055-8086-3820f53b55e3\") " pod="openshift-multus/network-metrics-daemon-s229p"
Sep 16 18:40:43.021149 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021102    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8859cf81-dc3a-4a84-a649-efdaee26a8a2-host-run-multus-certs\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:43.021149 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021117    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1d943a0b-9f9f-4b81-bffb-c434be495a76-etc-modprobe-d\") pod \"tuned-vlcrr\" (UID: \"1d943a0b-9f9f-4b81-bffb-c434be495a76\") " pod="openshift-cluster-node-tuning-operator/tuned-vlcrr"
Sep 16 18:40:43.021149 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021138    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8859cf81-dc3a-4a84-a649-efdaee26a8a2-etc-kubernetes\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:43.021408 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021153    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-run-ovn\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:43.021408 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021160    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-host-kubelet\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:43.021408 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021182    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8859cf81-dc3a-4a84-a649-efdaee26a8a2-host-run-multus-certs\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:43.021408 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021193    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8859cf81-dc3a-4a84-a649-efdaee26a8a2-etc-kubernetes\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:43.021408 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021197    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-host-kubelet\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:43.021408 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021211    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-host-cni-bin\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:43.021408 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021241    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/88b20525-d601-4224-bd43-397308547efd-ovnkube-config\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:43.021408 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021260    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-host-cni-bin\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:43.021408 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021289    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/590e4879-7945-4583-b644-0caa6b1c6ebd-tmp-dir\") pod \"node-resolver-wmfjw\" (UID: \"590e4879-7945-4583-b644-0caa6b1c6ebd\") " pod="openshift-dns/node-resolver-wmfjw"
Sep 16 18:40:43.021408 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021317    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5ttq\" (UniqueName: \"kubernetes.io/projected/8e33041b-cc50-41a3-bcc4-34942629f4cc-kube-api-access-r5ttq\") pod \"network-check-target-ggvbf\" (UID: \"8e33041b-cc50-41a3-bcc4-34942629f4cc\") " pod="openshift-network-diagnostics/network-check-target-ggvbf"
Sep 16 18:40:43.021408 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021342    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8049c55c-c4ac-44a1-a1bf-5872846445c4-cnibin\") pod \"multus-additional-cni-plugins-btfcs\" (UID: \"8049c55c-c4ac-44a1-a1bf-5872846445c4\") " pod="openshift-multus/multus-additional-cni-plugins-btfcs"
Sep 16 18:40:43.021408 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021372    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8049c55c-c4ac-44a1-a1bf-5872846445c4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-btfcs\" (UID: \"8049c55c-c4ac-44a1-a1bf-5872846445c4\") " pod="openshift-multus/multus-additional-cni-plugins-btfcs"
Sep 16 18:40:43.021408 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021398    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8859cf81-dc3a-4a84-a649-efdaee26a8a2-system-cni-dir\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:43.022096 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021419    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8859cf81-dc3a-4a84-a649-efdaee26a8a2-cni-binary-copy\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:43.022096 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021375    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8049c55c-c4ac-44a1-a1bf-5872846445c4-cnibin\") pod \"multus-additional-cni-plugins-btfcs\" (UID: \"8049c55c-c4ac-44a1-a1bf-5872846445c4\") " pod="openshift-multus/multus-additional-cni-plugins-btfcs"
Sep 16 18:40:43.022096 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021446    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7ls2\" (UniqueName: \"kubernetes.io/projected/ba8dcba9-de75-4055-8086-3820f53b55e3-kube-api-access-q7ls2\") pod \"network-metrics-daemon-s229p\" (UID: \"ba8dcba9-de75-4055-8086-3820f53b55e3\") " pod="openshift-multus/network-metrics-daemon-s229p"
Sep 16 18:40:43.022096 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021471    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4870db69-091f-43b3-944c-4fa7d195ca2d-registration-dir\") pod \"aws-ebs-csi-driver-node-hv7z5\" (UID: \"4870db69-091f-43b3-944c-4fa7d195ca2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hv7z5"
Sep 16 18:40:43.022096 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021497    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clzd5\" (UniqueName: \"kubernetes.io/projected/1e98550f-c964-434b-995d-60ad18764eff-kube-api-access-clzd5\") pod \"iptables-alerter-298hg\" (UID: \"1e98550f-c964-434b-995d-60ad18764eff\") " pod="openshift-network-operator/iptables-alerter-298hg"
Sep 16 18:40:43.022096 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021521    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2b544a31-67f6-4a93-beeb-a21fd49b00b1-kubelet-config\") pod \"global-pull-secret-syncer-88mzt\" (UID: \"2b544a31-67f6-4a93-beeb-a21fd49b00b1\") " pod="kube-system/global-pull-secret-syncer-88mzt"
Sep 16 18:40:43.022096 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021545    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/88b20525-d601-4224-bd43-397308547efd-env-overrides\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:43.022096 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021564    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8859cf81-dc3a-4a84-a649-efdaee26a8a2-system-cni-dir\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:43.022096 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021568    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8049c55c-c4ac-44a1-a1bf-5872846445c4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-btfcs\" (UID: \"8049c55c-c4ac-44a1-a1bf-5872846445c4\") " pod="openshift-multus/multus-additional-cni-plugins-btfcs"
Sep 16 18:40:43.022096 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021637    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt6sq\" (UniqueName: \"kubernetes.io/projected/8859cf81-dc3a-4a84-a649-efdaee26a8a2-kube-api-access-nt6sq\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:43.022096 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021671    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d943a0b-9f9f-4b81-bffb-c434be495a76-var-lib-kubelet\") pod \"tuned-vlcrr\" (UID: \"1d943a0b-9f9f-4b81-bffb-c434be495a76\") " pod="openshift-cluster-node-tuning-operator/tuned-vlcrr"
Sep 16 18:40:43.022096 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021697    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-host-run-ovn-kubernetes\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:43.022096 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021696    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2b544a31-67f6-4a93-beeb-a21fd49b00b1-kubelet-config\") pod \"global-pull-secret-syncer-88mzt\" (UID: \"2b544a31-67f6-4a93-beeb-a21fd49b00b1\") " pod="kube-system/global-pull-secret-syncer-88mzt"
Sep 16 18:40:43.022096 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021734    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8859cf81-dc3a-4a84-a649-efdaee26a8a2-host-var-lib-kubelet\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:43.022096 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021740    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-host-run-ovn-kubernetes\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:43.022096 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021768    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8859cf81-dc3a-4a84-a649-efdaee26a8a2-multus-daemon-config\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:43.022096 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021775    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8859cf81-dc3a-4a84-a649-efdaee26a8a2-host-var-lib-kubelet\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:43.022764 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021787    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8049c55c-c4ac-44a1-a1bf-5872846445c4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-btfcs\" (UID: \"8049c55c-c4ac-44a1-a1bf-5872846445c4\") " pod="openshift-multus/multus-additional-cni-plugins-btfcs"
Sep 16 18:40:43.022764 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021797    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4870db69-091f-43b3-944c-4fa7d195ca2d-sys-fs\") pod \"aws-ebs-csi-driver-node-hv7z5\" (UID: \"4870db69-091f-43b3-944c-4fa7d195ca2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hv7z5"
Sep 16 18:40:43.022764 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021830    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f935c176-f27b-4179-bc5e-802ee61499d3-host\") pod \"node-ca-8cxsn\" (UID: \"f935c176-f27b-4179-bc5e-802ee61499d3\") " pod="openshift-image-registry/node-ca-8cxsn"
Sep 16 18:40:43.022764 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021872    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b544a31-67f6-4a93-beeb-a21fd49b00b1-original-pull-secret\") pod \"global-pull-secret-syncer-88mzt\" (UID: \"2b544a31-67f6-4a93-beeb-a21fd49b00b1\") " pod="kube-system/global-pull-secret-syncer-88mzt"
Sep 16 18:40:43.022764 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021878    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f935c176-f27b-4179-bc5e-802ee61499d3-host\") pod \"node-ca-8cxsn\" (UID: \"f935c176-f27b-4179-bc5e-802ee61499d3\") " pod="openshift-image-registry/node-ca-8cxsn"
Sep 16 18:40:43.022764 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021897    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8859cf81-dc3a-4a84-a649-efdaee26a8a2-host-var-lib-cni-multus\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:43.022764 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021922    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4870db69-091f-43b3-944c-4fa7d195ca2d-etc-selinux\") pod \"aws-ebs-csi-driver-node-hv7z5\" (UID: \"4870db69-091f-43b3-944c-4fa7d195ca2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hv7z5"
Sep 16 18:40:43.022764 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021946    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1e98550f-c964-434b-995d-60ad18764eff-host-slash\") pod \"iptables-alerter-298hg\" (UID: \"1e98550f-c964-434b-995d-60ad18764eff\") " pod="openshift-network-operator/iptables-alerter-298hg"
Sep 16 18:40:43.022764 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021971    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-node-log\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:43.022764 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021977    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8859cf81-dc3a-4a84-a649-efdaee26a8a2-host-var-lib-cni-multus\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:43.022764 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:43.021949    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:43.022764 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021996    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1e98550f-c964-434b-995d-60ad18764eff-host-slash\") pod \"iptables-alerter-298hg\" (UID: \"1e98550f-c964-434b-995d-60ad18764eff\") " pod="openshift-network-operator/iptables-alerter-298hg"
Sep 16 18:40:43.022764 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022009    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8859cf81-dc3a-4a84-a649-efdaee26a8a2-cni-binary-copy\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:43.022764 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022018    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/88b20525-d601-4224-bd43-397308547efd-env-overrides\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:43.022764 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022027    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8859cf81-dc3a-4a84-a649-efdaee26a8a2-host-var-lib-cni-bin\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:43.022764 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022009    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8049c55c-c4ac-44a1-a1bf-5872846445c4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-btfcs\" (UID: \"8049c55c-c4ac-44a1-a1bf-5872846445c4\") " pod="openshift-multus/multus-additional-cni-plugins-btfcs"
Sep 16 18:40:43.022764 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.021994    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8859cf81-dc3a-4a84-a649-efdaee26a8a2-host-var-lib-cni-bin\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:43.022764 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022039    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-node-log\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:43.023529 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:43.022099    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b544a31-67f6-4a93-beeb-a21fd49b00b1-original-pull-secret podName:2b544a31-67f6-4a93-beeb-a21fd49b00b1 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:43.522045053 +0000 UTC m=+2.017404828 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b544a31-67f6-4a93-beeb-a21fd49b00b1-original-pull-secret") pod "global-pull-secret-syncer-88mzt" (UID: "2b544a31-67f6-4a93-beeb-a21fd49b00b1") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:43.023529 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022142    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/025a78d4-1397-4057-89d3-56bbf1d96e2c-agent-certs\") pod \"konnectivity-agent-jpq9w\" (UID: \"025a78d4-1397-4057-89d3-56bbf1d96e2c\") " pod="kube-system/konnectivity-agent-jpq9w"
Sep 16 18:40:43.023529 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022167    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d943a0b-9f9f-4b81-bffb-c434be495a76-tmp\") pod \"tuned-vlcrr\" (UID: \"1d943a0b-9f9f-4b81-bffb-c434be495a76\") " pod="openshift-cluster-node-tuning-operator/tuned-vlcrr"
Sep 16 18:40:43.023529 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022195    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f935c176-f27b-4179-bc5e-802ee61499d3-serviceca\") pod \"node-ca-8cxsn\" (UID: \"f935c176-f27b-4179-bc5e-802ee61499d3\") " pod="openshift-image-registry/node-ca-8cxsn"
Sep 16 18:40:43.023529 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022224    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8049c55c-c4ac-44a1-a1bf-5872846445c4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-btfcs\" (UID: \"8049c55c-c4ac-44a1-a1bf-5872846445c4\") " pod="openshift-multus/multus-additional-cni-plugins-btfcs"
Sep 16 18:40:43.023529 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022225    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8859cf81-dc3a-4a84-a649-efdaee26a8a2-multus-daemon-config\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:43.023529 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022253    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8859cf81-dc3a-4a84-a649-efdaee26a8a2-hostroot\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:43.023529 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022269    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/88b20525-d601-4224-bd43-397308547efd-ovnkube-config\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:43.023529 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022315    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-host-run-netns\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:43.023529 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022318    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8859cf81-dc3a-4a84-a649-efdaee26a8a2-hostroot\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:43.023529 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022352    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-host-run-netns\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:43.023529 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022358    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8859cf81-dc3a-4a84-a649-efdaee26a8a2-multus-cni-dir\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:43.023529 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022386    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1d943a0b-9f9f-4b81-bffb-c434be495a76-etc-systemd\") pod \"tuned-vlcrr\" (UID: \"1d943a0b-9f9f-4b81-bffb-c434be495a76\") " pod="openshift-cluster-node-tuning-operator/tuned-vlcrr"
Sep 16 18:40:43.023529 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022393    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8859cf81-dc3a-4a84-a649-efdaee26a8a2-multus-cni-dir\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:43.023529 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022411    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7gt4\" (UniqueName: \"kubernetes.io/projected/590e4879-7945-4583-b644-0caa6b1c6ebd-kube-api-access-m7gt4\") pod \"node-resolver-wmfjw\" (UID: \"590e4879-7945-4583-b644-0caa6b1c6ebd\") " pod="openshift-dns/node-resolver-wmfjw"
Sep 16 18:40:43.023529 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022445    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4870db69-091f-43b3-944c-4fa7d195ca2d-socket-dir\") pod \"aws-ebs-csi-driver-node-hv7z5\" (UID: \"4870db69-091f-43b3-944c-4fa7d195ca2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hv7z5"
Sep 16 18:40:43.024000 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022480    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c85w4\" (UniqueName: \"kubernetes.io/projected/4870db69-091f-43b3-944c-4fa7d195ca2d-kube-api-access-c85w4\") pod \"aws-ebs-csi-driver-node-hv7z5\" (UID: \"4870db69-091f-43b3-944c-4fa7d195ca2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hv7z5"
Sep 16 18:40:43.024000 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022509    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-run-openvswitch\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:43.024000 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022533    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/88b20525-d601-4224-bd43-397308547efd-ovn-node-metrics-cert\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:43.024000 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022556    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8049c55c-c4ac-44a1-a1bf-5872846445c4-system-cni-dir\") pod \"multus-additional-cni-plugins-btfcs\" (UID: \"8049c55c-c4ac-44a1-a1bf-5872846445c4\") " pod="openshift-multus/multus-additional-cni-plugins-btfcs"
Sep 16 18:40:43.024000 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022562    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-run-openvswitch\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:43.024000 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022579    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1d943a0b-9f9f-4b81-bffb-c434be495a76-etc-sysctl-conf\") pod \"tuned-vlcrr\" (UID: \"1d943a0b-9f9f-4b81-bffb-c434be495a76\") " pod="openshift-cluster-node-tuning-operator/tuned-vlcrr"
Sep 16 18:40:43.024000 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022588    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f935c176-f27b-4179-bc5e-802ee61499d3-serviceca\") pod \"node-ca-8cxsn\" (UID: \"f935c176-f27b-4179-bc5e-802ee61499d3\") " pod="openshift-image-registry/node-ca-8cxsn"
Sep 16 18:40:43.024000 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022603    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/590e4879-7945-4583-b644-0caa6b1c6ebd-hosts-file\") pod \"node-resolver-wmfjw\" (UID: \"590e4879-7945-4583-b644-0caa6b1c6ebd\") " pod="openshift-dns/node-resolver-wmfjw"
Sep 16 18:40:43.024000 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022613    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8049c55c-c4ac-44a1-a1bf-5872846445c4-system-cni-dir\") pod \"multus-additional-cni-plugins-btfcs\" (UID: \"8049c55c-c4ac-44a1-a1bf-5872846445c4\") " pod="openshift-multus/multus-additional-cni-plugins-btfcs"
Sep 16 18:40:43.024000 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022621    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8049c55c-c4ac-44a1-a1bf-5872846445c4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-btfcs\" (UID: \"8049c55c-c4ac-44a1-a1bf-5872846445c4\") " pod="openshift-multus/multus-additional-cni-plugins-btfcs"
Sep 16 18:40:43.024000 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022627    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1e98550f-c964-434b-995d-60ad18764eff-iptables-alerter-script\") pod \"iptables-alerter-298hg\" (UID: \"1e98550f-c964-434b-995d-60ad18764eff\") " pod="openshift-network-operator/iptables-alerter-298hg"
Sep 16 18:40:43.024000 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022644    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/88b20525-d601-4224-bd43-397308547efd-ovnkube-script-lib\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:43.024000 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022657    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1d943a0b-9f9f-4b81-bffb-c434be495a76-lib-modules\") pod \"tuned-vlcrr\" (UID: \"1d943a0b-9f9f-4b81-bffb-c434be495a76\") " pod="openshift-cluster-node-tuning-operator/tuned-vlcrr"
Sep 16 18:40:43.024000 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022677    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4870db69-091f-43b3-944c-4fa7d195ca2d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hv7z5\" (UID: \"4870db69-091f-43b3-944c-4fa7d195ca2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hv7z5"
Sep 16 18:40:43.024000 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022698    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4870db69-091f-43b3-944c-4fa7d195ca2d-device-dir\") pod \"aws-ebs-csi-driver-node-hv7z5\" (UID: \"4870db69-091f-43b3-944c-4fa7d195ca2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hv7z5"
Sep 16 18:40:43.024000 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022735    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs64s\" (UniqueName: \"kubernetes.io/projected/f935c176-f27b-4179-bc5e-802ee61499d3-kube-api-access-rs64s\") pod \"node-ca-8cxsn\" (UID: \"f935c176-f27b-4179-bc5e-802ee61499d3\") " pod="openshift-image-registry/node-ca-8cxsn"
Sep 16 18:40:43.024000 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022756    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-run-systemd\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:43.024489 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022772    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-var-lib-openvswitch\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:43.024489 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022788    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-log-socket\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:43.024489 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022804    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8859cf81-dc3a-4a84-a649-efdaee26a8a2-cnibin\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:43.024489 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022839    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-run-systemd\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:43.024489 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022856    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8859cf81-dc3a-4a84-a649-efdaee26a8a2-cnibin\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:43.024489 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022869    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/025a78d4-1397-4057-89d3-56bbf1d96e2c-konnectivity-ca\") pod \"konnectivity-agent-jpq9w\" (UID: \"025a78d4-1397-4057-89d3-56bbf1d96e2c\") " pod="kube-system/konnectivity-agent-jpq9w"
Sep 16 18:40:43.024489 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022881    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-log-socket\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:43.024489 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022890    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1d943a0b-9f9f-4b81-bffb-c434be495a76-etc-sysconfig\") pod \"tuned-vlcrr\" (UID: \"1d943a0b-9f9f-4b81-bffb-c434be495a76\") " pod="openshift-cluster-node-tuning-operator/tuned-vlcrr"
Sep 16 18:40:43.024489 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022911    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d943a0b-9f9f-4b81-bffb-c434be495a76-sys\") pod \"tuned-vlcrr\" (UID: \"1d943a0b-9f9f-4b81-bffb-c434be495a76\") " pod="openshift-cluster-node-tuning-operator/tuned-vlcrr"
Sep 16 18:40:43.024489 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022939    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2b544a31-67f6-4a93-beeb-a21fd49b00b1-dbus\") pod \"global-pull-secret-syncer-88mzt\" (UID: \"2b544a31-67f6-4a93-beeb-a21fd49b00b1\") " pod="kube-system/global-pull-secret-syncer-88mzt"
Sep 16 18:40:43.024489 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022961    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-systemd-units\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:43.024489 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.022985    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw4tm\" (UniqueName: \"kubernetes.io/projected/88b20525-d601-4224-bd43-397308547efd-kube-api-access-vw4tm\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:43.024489 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.023007    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8859cf81-dc3a-4a84-a649-efdaee26a8a2-host-run-netns\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:43.024489 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.023028    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2b544a31-67f6-4a93-beeb-a21fd49b00b1-dbus\") pod \"global-pull-secret-syncer-88mzt\" (UID: \"2b544a31-67f6-4a93-beeb-a21fd49b00b1\") " pod="kube-system/global-pull-secret-syncer-88mzt"
Sep 16 18:40:43.024489 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.023029    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d943a0b-9f9f-4b81-bffb-c434be495a76-host\") pod \"tuned-vlcrr\" (UID: \"1d943a0b-9f9f-4b81-bffb-c434be495a76\") " pod="openshift-cluster-node-tuning-operator/tuned-vlcrr"
Sep 16 18:40:43.024489 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.023028    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-systemd-units\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:43.024489 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.023085    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8859cf81-dc3a-4a84-a649-efdaee26a8a2-host-run-netns\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:43.024489 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.023098    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1d943a0b-9f9f-4b81-bffb-c434be495a76-etc-tuned\") pod \"tuned-vlcrr\" (UID: \"1d943a0b-9f9f-4b81-bffb-c434be495a76\") " pod="openshift-cluster-node-tuning-operator/tuned-vlcrr"
Sep 16 18:40:43.025003 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.023109    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/88b20525-d601-4224-bd43-397308547efd-ovnkube-script-lib\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:43.025003 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.023119    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-host-cni-netd\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:43.025003 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.023144    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-var-lib-openvswitch\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:43.025003 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.023146    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 18:40:43.025003 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.023136    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8859cf81-dc3a-4a84-a649-efdaee26a8a2-multus-socket-dir-parent\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:43.025003 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.023149    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1e98550f-c964-434b-995d-60ad18764eff-iptables-alerter-script\") pod \"iptables-alerter-298hg\" (UID: \"1e98550f-c964-434b-995d-60ad18764eff\") " pod="openshift-network-operator/iptables-alerter-298hg"
Sep 16 18:40:43.025003 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.023173    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-host-cni-netd\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:43.025003 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.023204    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8859cf81-dc3a-4a84-a649-efdaee26a8a2-multus-socket-dir-parent\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:43.025003 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.023227    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8859cf81-dc3a-4a84-a649-efdaee26a8a2-host-run-k8s-cni-cncf-io\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:43.025003 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.023251    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8859cf81-dc3a-4a84-a649-efdaee26a8a2-multus-conf-dir\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:43.025003 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.023289    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1d943a0b-9f9f-4b81-bffb-c434be495a76-run\") pod \"tuned-vlcrr\" (UID: \"1d943a0b-9f9f-4b81-bffb-c434be495a76\") " pod="openshift-cluster-node-tuning-operator/tuned-vlcrr"
Sep 16 18:40:43.025003 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.023292    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8859cf81-dc3a-4a84-a649-efdaee26a8a2-host-run-k8s-cni-cncf-io\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:43.025003 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.023314    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-host-slash\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:43.025003 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.023331    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-etc-openvswitch\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:43.025003 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.023347    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:43.025003 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.023355    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-etc-openvswitch\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:43.025003 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.023357    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-host-slash\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:43.025003 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.023365    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8049c55c-c4ac-44a1-a1bf-5872846445c4-cni-binary-copy\") pod \"multus-additional-cni-plugins-btfcs\" (UID: \"8049c55c-c4ac-44a1-a1bf-5872846445c4\") " pod="openshift-multus/multus-additional-cni-plugins-btfcs"
Sep 16 18:40:43.025508 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.023390    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d943a0b-9f9f-4b81-bffb-c434be495a76-etc-kubernetes\") pod \"tuned-vlcrr\" (UID: \"1d943a0b-9f9f-4b81-bffb-c434be495a76\") " pod="openshift-cluster-node-tuning-operator/tuned-vlcrr"
Sep 16 18:40:43.025508 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.023405    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88b20525-d601-4224-bd43-397308547efd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:43.025508 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.023413    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1d943a0b-9f9f-4b81-bffb-c434be495a76-etc-sysctl-d\") pod \"tuned-vlcrr\" (UID: \"1d943a0b-9f9f-4b81-bffb-c434be495a76\") " pod="openshift-cluster-node-tuning-operator/tuned-vlcrr"
Sep 16 18:40:43.025508 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.023331    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8859cf81-dc3a-4a84-a649-efdaee26a8a2-multus-conf-dir\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:43.025508 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.023438    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpfzn\" (UniqueName: \"kubernetes.io/projected/1d943a0b-9f9f-4b81-bffb-c434be495a76-kube-api-access-kpfzn\") pod \"tuned-vlcrr\" (UID: \"1d943a0b-9f9f-4b81-bffb-c434be495a76\") " pod="openshift-cluster-node-tuning-operator/tuned-vlcrr"
Sep 16 18:40:43.025508 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.023730    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8049c55c-c4ac-44a1-a1bf-5872846445c4-cni-binary-copy\") pod \"multus-additional-cni-plugins-btfcs\" (UID: \"8049c55c-c4ac-44a1-a1bf-5872846445c4\") " pod="openshift-multus/multus-additional-cni-plugins-btfcs"
Sep 16 18:40:43.026624 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.026608    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/88b20525-d601-4224-bd43-397308547efd-ovn-node-metrics-cert\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:43.030878 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.030864    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:43.033771 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:43.033758    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:43.033833 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:43.033775    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:43.033833 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:43.033783    2577 projected.go:196] Error preparing data for projected volume kube-api-access-r5ttq for pod openshift-network-diagnostics/network-check-target-ggvbf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:43.033833 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:43.033825    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e33041b-cc50-41a3-bcc4-34942629f4cc-kube-api-access-r5ttq podName:8e33041b-cc50-41a3-bcc4-34942629f4cc nodeName:}" failed. No retries permitted until 2026-09-16 18:40:43.533813323 +0000 UTC m=+2.029173083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-r5ttq" (UniqueName: "kubernetes.io/projected/8e33041b-cc50-41a3-bcc4-34942629f4cc-kube-api-access-r5ttq") pod "network-check-target-ggvbf" (UID: "8e33041b-cc50-41a3-bcc4-34942629f4cc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:43.036040 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.036012    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plpqb\" (UniqueName: \"kubernetes.io/projected/8049c55c-c4ac-44a1-a1bf-5872846445c4-kube-api-access-plpqb\") pod \"multus-additional-cni-plugins-btfcs\" (UID: \"8049c55c-c4ac-44a1-a1bf-5872846445c4\") " pod="openshift-multus/multus-additional-cni-plugins-btfcs"
Sep 16 18:40:43.038329 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.038285    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clzd5\" (UniqueName: \"kubernetes.io/projected/1e98550f-c964-434b-995d-60ad18764eff-kube-api-access-clzd5\") pod \"iptables-alerter-298hg\" (UID: \"1e98550f-c964-434b-995d-60ad18764eff\") " pod="openshift-network-operator/iptables-alerter-298hg"
Sep 16 18:40:43.038413 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.038355    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt6sq\" (UniqueName: \"kubernetes.io/projected/8859cf81-dc3a-4a84-a649-efdaee26a8a2-kube-api-access-nt6sq\") pod \"multus-rlvsc\" (UID: \"8859cf81-dc3a-4a84-a649-efdaee26a8a2\") " pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:43.038501 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.038479    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw4tm\" (UniqueName: \"kubernetes.io/projected/88b20525-d601-4224-bd43-397308547efd-kube-api-access-vw4tm\") pod \"ovnkube-node-wcfql\" (UID: \"88b20525-d601-4224-bd43-397308547efd\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:43.038608 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.038484    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs64s\" (UniqueName: \"kubernetes.io/projected/f935c176-f27b-4179-bc5e-802ee61499d3-kube-api-access-rs64s\") pod \"node-ca-8cxsn\" (UID: \"f935c176-f27b-4179-bc5e-802ee61499d3\") " pod="openshift-image-registry/node-ca-8cxsn"
Sep 16 18:40:43.040217 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.040175    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-247.ec2.internal" event={"ID":"c28a4ec2fbe427bdfc7c14e0446ecb67","Type":"ContainerStarted","Data":"6d2a332ec2633e1d5d142c671d902bed6dce61aa5a659b541125f25dcc1d6213"}
Sep 16 18:40:43.041288 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.041265    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-247.ec2.internal" event={"ID":"146a57ab6e613bbac55bc1f7a3764a51","Type":"ContainerStarted","Data":"a1697e0fdd3992ca89d29405baf969943049f9c0e05e11cbc0402699f092c5f6"}
Sep 16 18:40:43.124103 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.124082    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d943a0b-9f9f-4b81-bffb-c434be495a76-etc-kubernetes\") pod \"tuned-vlcrr\" (UID: \"1d943a0b-9f9f-4b81-bffb-c434be495a76\") " pod="openshift-cluster-node-tuning-operator/tuned-vlcrr"
Sep 16 18:40:43.124204 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.124107    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1d943a0b-9f9f-4b81-bffb-c434be495a76-etc-sysctl-d\") pod \"tuned-vlcrr\" (UID: \"1d943a0b-9f9f-4b81-bffb-c434be495a76\") " pod="openshift-cluster-node-tuning-operator/tuned-vlcrr"
Sep 16 18:40:43.124204 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.124124    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpfzn\" (UniqueName: \"kubernetes.io/projected/1d943a0b-9f9f-4b81-bffb-c434be495a76-kube-api-access-kpfzn\") pod \"tuned-vlcrr\" (UID: \"1d943a0b-9f9f-4b81-bffb-c434be495a76\") " pod="openshift-cluster-node-tuning-operator/tuned-vlcrr"
Sep 16 18:40:43.124204 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.124188    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d943a0b-9f9f-4b81-bffb-c434be495a76-etc-kubernetes\") pod \"tuned-vlcrr\" (UID: \"1d943a0b-9f9f-4b81-bffb-c434be495a76\") " pod="openshift-cluster-node-tuning-operator/tuned-vlcrr"
Sep 16 18:40:43.124363 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.124221    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba8dcba9-de75-4055-8086-3820f53b55e3-metrics-certs\") pod \"network-metrics-daemon-s229p\" (UID: \"ba8dcba9-de75-4055-8086-3820f53b55e3\") " pod="openshift-multus/network-metrics-daemon-s229p"
Sep 16 18:40:43.124363 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.124246    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1d943a0b-9f9f-4b81-bffb-c434be495a76-etc-modprobe-d\") pod \"tuned-vlcrr\" (UID: \"1d943a0b-9f9f-4b81-bffb-c434be495a76\") " pod="openshift-cluster-node-tuning-operator/tuned-vlcrr"
Sep 16 18:40:43.124363 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.124271    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/590e4879-7945-4583-b644-0caa6b1c6ebd-tmp-dir\") pod \"node-resolver-wmfjw\" (UID: \"590e4879-7945-4583-b644-0caa6b1c6ebd\") " pod="openshift-dns/node-resolver-wmfjw"
Sep 16 18:40:43.124363 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.124269    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1d943a0b-9f9f-4b81-bffb-c434be495a76-etc-sysctl-d\") pod \"tuned-vlcrr\" (UID: \"1d943a0b-9f9f-4b81-bffb-c434be495a76\") " pod="openshift-cluster-node-tuning-operator/tuned-vlcrr"
Sep 16 18:40:43.124363 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.124340    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7ls2\" (UniqueName: \"kubernetes.io/projected/ba8dcba9-de75-4055-8086-3820f53b55e3-kube-api-access-q7ls2\") pod \"network-metrics-daemon-s229p\" (UID: \"ba8dcba9-de75-4055-8086-3820f53b55e3\") " pod="openshift-multus/network-metrics-daemon-s229p"
Sep 16 18:40:43.124592 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.124368    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4870db69-091f-43b3-944c-4fa7d195ca2d-registration-dir\") pod \"aws-ebs-csi-driver-node-hv7z5\" (UID: \"4870db69-091f-43b3-944c-4fa7d195ca2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hv7z5"
Sep 16 18:40:43.124592 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.124375    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1d943a0b-9f9f-4b81-bffb-c434be495a76-etc-modprobe-d\") pod \"tuned-vlcrr\" (UID: \"1d943a0b-9f9f-4b81-bffb-c434be495a76\") " pod="openshift-cluster-node-tuning-operator/tuned-vlcrr"
Sep 16 18:40:43.124592 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.124387    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d943a0b-9f9f-4b81-bffb-c434be495a76-var-lib-kubelet\") pod \"tuned-vlcrr\" (UID: \"1d943a0b-9f9f-4b81-bffb-c434be495a76\") " pod="openshift-cluster-node-tuning-operator/tuned-vlcrr"
Sep 16 18:40:43.124592 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:43.124343    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:43.124592 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.124425    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4870db69-091f-43b3-944c-4fa7d195ca2d-sys-fs\") pod \"aws-ebs-csi-driver-node-hv7z5\" (UID: \"4870db69-091f-43b3-944c-4fa7d195ca2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hv7z5"
Sep 16 18:40:43.124592 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:43.124437    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba8dcba9-de75-4055-8086-3820f53b55e3-metrics-certs podName:ba8dcba9-de75-4055-8086-3820f53b55e3 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:43.624420563 +0000 UTC m=+2.119780323 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba8dcba9-de75-4055-8086-3820f53b55e3-metrics-certs") pod "network-metrics-daemon-s229p" (UID: "ba8dcba9-de75-4055-8086-3820f53b55e3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:43.124592 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.124433    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4870db69-091f-43b3-944c-4fa7d195ca2d-registration-dir\") pod \"aws-ebs-csi-driver-node-hv7z5\" (UID: \"4870db69-091f-43b3-944c-4fa7d195ca2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hv7z5"
Sep 16 18:40:43.124592 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.124441    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d943a0b-9f9f-4b81-bffb-c434be495a76-var-lib-kubelet\") pod \"tuned-vlcrr\" (UID: \"1d943a0b-9f9f-4b81-bffb-c434be495a76\") " pod="openshift-cluster-node-tuning-operator/tuned-vlcrr"
Sep 16 18:40:43.124592 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.124467    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4870db69-091f-43b3-944c-4fa7d195ca2d-sys-fs\") pod \"aws-ebs-csi-driver-node-hv7z5\" (UID: \"4870db69-091f-43b3-944c-4fa7d195ca2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hv7z5"
Sep 16 18:40:43.124592 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.124477    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4870db69-091f-43b3-944c-4fa7d195ca2d-etc-selinux\") pod \"aws-ebs-csi-driver-node-hv7z5\" (UID: \"4870db69-091f-43b3-944c-4fa7d195ca2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hv7z5"
Sep 16 18:40:43.124592 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.124504    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/025a78d4-1397-4057-89d3-56bbf1d96e2c-agent-certs\") pod \"konnectivity-agent-jpq9w\" (UID: \"025a78d4-1397-4057-89d3-56bbf1d96e2c\") " pod="kube-system/konnectivity-agent-jpq9w"
Sep 16 18:40:43.124592 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.124525    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d943a0b-9f9f-4b81-bffb-c434be495a76-tmp\") pod \"tuned-vlcrr\" (UID: \"1d943a0b-9f9f-4b81-bffb-c434be495a76\") " pod="openshift-cluster-node-tuning-operator/tuned-vlcrr"
Sep 16 18:40:43.124592 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.124540    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4870db69-091f-43b3-944c-4fa7d195ca2d-etc-selinux\") pod \"aws-ebs-csi-driver-node-hv7z5\" (UID: \"4870db69-091f-43b3-944c-4fa7d195ca2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hv7z5"
Sep 16 18:40:43.124592 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.124562    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1d943a0b-9f9f-4b81-bffb-c434be495a76-etc-systemd\") pod \"tuned-vlcrr\" (UID: \"1d943a0b-9f9f-4b81-bffb-c434be495a76\") " pod="openshift-cluster-node-tuning-operator/tuned-vlcrr"
Sep 16 18:40:43.124592 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.124561    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/590e4879-7945-4583-b644-0caa6b1c6ebd-tmp-dir\") pod \"node-resolver-wmfjw\" (UID: \"590e4879-7945-4583-b644-0caa6b1c6ebd\") " pod="openshift-dns/node-resolver-wmfjw"
Sep 16 18:40:43.125240 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.124585    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7gt4\" (UniqueName: \"kubernetes.io/projected/590e4879-7945-4583-b644-0caa6b1c6ebd-kube-api-access-m7gt4\") pod \"node-resolver-wmfjw\" (UID: \"590e4879-7945-4583-b644-0caa6b1c6ebd\") " pod="openshift-dns/node-resolver-wmfjw"
Sep 16 18:40:43.125240 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.124660    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4870db69-091f-43b3-944c-4fa7d195ca2d-socket-dir\") pod \"aws-ebs-csi-driver-node-hv7z5\" (UID: \"4870db69-091f-43b3-944c-4fa7d195ca2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hv7z5"
Sep 16 18:40:43.125240 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.124688    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c85w4\" (UniqueName: \"kubernetes.io/projected/4870db69-091f-43b3-944c-4fa7d195ca2d-kube-api-access-c85w4\") pod \"aws-ebs-csi-driver-node-hv7z5\" (UID: \"4870db69-091f-43b3-944c-4fa7d195ca2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hv7z5"
Sep 16 18:40:43.125240 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.124630    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1d943a0b-9f9f-4b81-bffb-c434be495a76-etc-systemd\") pod \"tuned-vlcrr\" (UID: \"1d943a0b-9f9f-4b81-bffb-c434be495a76\") " pod="openshift-cluster-node-tuning-operator/tuned-vlcrr"
Sep 16 18:40:43.125240 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.124717    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1d943a0b-9f9f-4b81-bffb-c434be495a76-etc-sysctl-conf\") pod \"tuned-vlcrr\" (UID: \"1d943a0b-9f9f-4b81-bffb-c434be495a76\") " pod="openshift-cluster-node-tuning-operator/tuned-vlcrr"
Sep 16 18:40:43.125240 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.124752    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/590e4879-7945-4583-b644-0caa6b1c6ebd-hosts-file\") pod \"node-resolver-wmfjw\" (UID: \"590e4879-7945-4583-b644-0caa6b1c6ebd\") " pod="openshift-dns/node-resolver-wmfjw"
Sep 16 18:40:43.125240 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.124790    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1d943a0b-9f9f-4b81-bffb-c434be495a76-lib-modules\") pod \"tuned-vlcrr\" (UID: \"1d943a0b-9f9f-4b81-bffb-c434be495a76\") " pod="openshift-cluster-node-tuning-operator/tuned-vlcrr"
Sep 16 18:40:43.125240 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.124797    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4870db69-091f-43b3-944c-4fa7d195ca2d-socket-dir\") pod \"aws-ebs-csi-driver-node-hv7z5\" (UID: \"4870db69-091f-43b3-944c-4fa7d195ca2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hv7z5"
Sep 16 18:40:43.125240 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.124811    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4870db69-091f-43b3-944c-4fa7d195ca2d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hv7z5\" (UID: \"4870db69-091f-43b3-944c-4fa7d195ca2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hv7z5"
Sep 16 18:40:43.125240 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.124844    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4870db69-091f-43b3-944c-4fa7d195ca2d-device-dir\") pod \"aws-ebs-csi-driver-node-hv7z5\" (UID: \"4870db69-091f-43b3-944c-4fa7d195ca2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hv7z5"
Sep 16 18:40:43.125240 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.125095    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/025a78d4-1397-4057-89d3-56bbf1d96e2c-konnectivity-ca\") pod \"konnectivity-agent-jpq9w\" (UID: \"025a78d4-1397-4057-89d3-56bbf1d96e2c\") " pod="kube-system/konnectivity-agent-jpq9w"
Sep 16 18:40:43.125240 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.124854    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4870db69-091f-43b3-944c-4fa7d195ca2d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hv7z5\" (UID: \"4870db69-091f-43b3-944c-4fa7d195ca2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hv7z5"
Sep 16 18:40:43.125240 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.125167    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1d943a0b-9f9f-4b81-bffb-c434be495a76-etc-sysconfig\") pod \"tuned-vlcrr\" (UID: \"1d943a0b-9f9f-4b81-bffb-c434be495a76\") " pod="openshift-cluster-node-tuning-operator/tuned-vlcrr"
Sep 16 18:40:43.125240 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.124903    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4870db69-091f-43b3-944c-4fa7d195ca2d-device-dir\") pod \"aws-ebs-csi-driver-node-hv7z5\" (UID: \"4870db69-091f-43b3-944c-4fa7d195ca2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hv7z5"
Sep 16 18:40:43.125240 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.124915    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1d943a0b-9f9f-4b81-bffb-c434be495a76-lib-modules\") pod \"tuned-vlcrr\" (UID: \"1d943a0b-9f9f-4b81-bffb-c434be495a76\") " pod="openshift-cluster-node-tuning-operator/tuned-vlcrr"
Sep 16 18:40:43.125240 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.124859    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1d943a0b-9f9f-4b81-bffb-c434be495a76-etc-sysctl-conf\") pod \"tuned-vlcrr\" (UID: \"1d943a0b-9f9f-4b81-bffb-c434be495a76\") " pod="openshift-cluster-node-tuning-operator/tuned-vlcrr"
Sep 16 18:40:43.125780 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.125123    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1d943a0b-9f9f-4b81-bffb-c434be495a76-etc-sysconfig\") pod \"tuned-vlcrr\" (UID: \"1d943a0b-9f9f-4b81-bffb-c434be495a76\") " pod="openshift-cluster-node-tuning-operator/tuned-vlcrr"
Sep 16 18:40:43.125780 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.125301    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d943a0b-9f9f-4b81-bffb-c434be495a76-sys\") pod \"tuned-vlcrr\" (UID: \"1d943a0b-9f9f-4b81-bffb-c434be495a76\") " pod="openshift-cluster-node-tuning-operator/tuned-vlcrr"
Sep 16 18:40:43.125780 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.125333    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d943a0b-9f9f-4b81-bffb-c434be495a76-host\") pod \"tuned-vlcrr\" (UID: \"1d943a0b-9f9f-4b81-bffb-c434be495a76\") " pod="openshift-cluster-node-tuning-operator/tuned-vlcrr"
Sep 16 18:40:43.125780 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.124868    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/590e4879-7945-4583-b644-0caa6b1c6ebd-hosts-file\") pod \"node-resolver-wmfjw\" (UID: \"590e4879-7945-4583-b644-0caa6b1c6ebd\") " pod="openshift-dns/node-resolver-wmfjw"
Sep 16 18:40:43.125780 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.125358    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1d943a0b-9f9f-4b81-bffb-c434be495a76-etc-tuned\") pod \"tuned-vlcrr\" (UID: \"1d943a0b-9f9f-4b81-bffb-c434be495a76\") " pod="openshift-cluster-node-tuning-operator/tuned-vlcrr"
Sep 16 18:40:43.125780 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.125402    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d943a0b-9f9f-4b81-bffb-c434be495a76-sys\") pod \"tuned-vlcrr\" (UID: \"1d943a0b-9f9f-4b81-bffb-c434be495a76\") " pod="openshift-cluster-node-tuning-operator/tuned-vlcrr"
Sep 16 18:40:43.125780 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.125407    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1d943a0b-9f9f-4b81-bffb-c434be495a76-run\") pod \"tuned-vlcrr\" (UID: \"1d943a0b-9f9f-4b81-bffb-c434be495a76\") " pod="openshift-cluster-node-tuning-operator/tuned-vlcrr"
Sep 16 18:40:43.125780 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.125406    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d943a0b-9f9f-4b81-bffb-c434be495a76-host\") pod \"tuned-vlcrr\" (UID: \"1d943a0b-9f9f-4b81-bffb-c434be495a76\") " pod="openshift-cluster-node-tuning-operator/tuned-vlcrr"
Sep 16 18:40:43.125780 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.125509    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1d943a0b-9f9f-4b81-bffb-c434be495a76-run\") pod \"tuned-vlcrr\" (UID: \"1d943a0b-9f9f-4b81-bffb-c434be495a76\") " pod="openshift-cluster-node-tuning-operator/tuned-vlcrr"
Sep 16 18:40:43.125780 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.125525    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/025a78d4-1397-4057-89d3-56bbf1d96e2c-konnectivity-ca\") pod \"konnectivity-agent-jpq9w\" (UID: \"025a78d4-1397-4057-89d3-56bbf1d96e2c\") " pod="kube-system/konnectivity-agent-jpq9w"
Sep 16 18:40:43.126825 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.126802    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d943a0b-9f9f-4b81-bffb-c434be495a76-tmp\") pod \"tuned-vlcrr\" (UID: \"1d943a0b-9f9f-4b81-bffb-c434be495a76\") " pod="openshift-cluster-node-tuning-operator/tuned-vlcrr"
Sep 16 18:40:43.127145 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.127127    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/025a78d4-1397-4057-89d3-56bbf1d96e2c-agent-certs\") pod \"konnectivity-agent-jpq9w\" (UID: \"025a78d4-1397-4057-89d3-56bbf1d96e2c\") " pod="kube-system/konnectivity-agent-jpq9w"
Sep 16 18:40:43.127346 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.127329    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1d943a0b-9f9f-4b81-bffb-c434be495a76-etc-tuned\") pod \"tuned-vlcrr\" (UID: \"1d943a0b-9f9f-4b81-bffb-c434be495a76\") " pod="openshift-cluster-node-tuning-operator/tuned-vlcrr"
Sep 16 18:40:43.140220 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.140194    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7ls2\" (UniqueName: \"kubernetes.io/projected/ba8dcba9-de75-4055-8086-3820f53b55e3-kube-api-access-q7ls2\") pod \"network-metrics-daemon-s229p\" (UID: \"ba8dcba9-de75-4055-8086-3820f53b55e3\") " pod="openshift-multus/network-metrics-daemon-s229p"
Sep 16 18:40:43.141451 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.141431    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c85w4\" (UniqueName: \"kubernetes.io/projected/4870db69-091f-43b3-944c-4fa7d195ca2d-kube-api-access-c85w4\") pod \"aws-ebs-csi-driver-node-hv7z5\" (UID: \"4870db69-091f-43b3-944c-4fa7d195ca2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hv7z5"
Sep 16 18:40:43.142389 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.142351    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7gt4\" (UniqueName: \"kubernetes.io/projected/590e4879-7945-4583-b644-0caa6b1c6ebd-kube-api-access-m7gt4\") pod \"node-resolver-wmfjw\" (UID: \"590e4879-7945-4583-b644-0caa6b1c6ebd\") " pod="openshift-dns/node-resolver-wmfjw"
Sep 16 18:40:43.142457 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.142387    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpfzn\" (UniqueName: \"kubernetes.io/projected/1d943a0b-9f9f-4b81-bffb-c434be495a76-kube-api-access-kpfzn\") pod \"tuned-vlcrr\" (UID: \"1d943a0b-9f9f-4b81-bffb-c434be495a76\") " pod="openshift-cluster-node-tuning-operator/tuned-vlcrr"
Sep 16 18:40:43.233249 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.233198    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8cxsn"
Sep 16 18:40:43.242714 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:43.242696    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf935c176_f27b_4179_bc5e_802ee61499d3.slice/crio-245978a131f9a8a78f8092bcd6e418b4e36d4b1e560ed5420b748e61a2981299 WatchSource:0}: Error finding container 245978a131f9a8a78f8092bcd6e418b4e36d4b1e560ed5420b748e61a2981299: Status 404 returned error can't find the container with id 245978a131f9a8a78f8092bcd6e418b4e36d4b1e560ed5420b748e61a2981299
Sep 16 18:40:43.245668 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.245651    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-298hg"
Sep 16 18:40:43.252835 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:43.252813    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e98550f_c964_434b_995d_60ad18764eff.slice/crio-67aec8a6a511dd5721e7d8bb50d4bc960e3f3e37e751c093685905196ac60f24 WatchSource:0}: Error finding container 67aec8a6a511dd5721e7d8bb50d4bc960e3f3e37e751c093685905196ac60f24: Status 404 returned error can't find the container with id 67aec8a6a511dd5721e7d8bb50d4bc960e3f3e37e751c093685905196ac60f24
Sep 16 18:40:43.273024 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.273004    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:40:43.276780 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.276766    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-btfcs"
Sep 16 18:40:43.279505 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:43.279485    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88b20525_d601_4224_bd43_397308547efd.slice/crio-cfb311409ba5f94efe436b2b7ce8aa6a19616ecab166d0a4420cb4574a8dcf82 WatchSource:0}: Error finding container cfb311409ba5f94efe436b2b7ce8aa6a19616ecab166d0a4420cb4574a8dcf82: Status 404 returned error can't find the container with id cfb311409ba5f94efe436b2b7ce8aa6a19616ecab166d0a4420cb4574a8dcf82
Sep 16 18:40:43.282563 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.282548    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rlvsc"
Sep 16 18:40:43.283689 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:43.283671    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8049c55c_c4ac_44a1_a1bf_5872846445c4.slice/crio-e5f854d46d0b27a02c5a1f6a2a5758a65b519b2f82ac755cffd0a1fb0c7bf873 WatchSource:0}: Error finding container e5f854d46d0b27a02c5a1f6a2a5758a65b519b2f82ac755cffd0a1fb0c7bf873: Status 404 returned error can't find the container with id e5f854d46d0b27a02c5a1f6a2a5758a65b519b2f82ac755cffd0a1fb0c7bf873
Sep 16 18:40:43.287654 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.287635    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jpq9w"
Sep 16 18:40:43.289357 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:43.289342    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8859cf81_dc3a_4a84_a649_efdaee26a8a2.slice/crio-038a22475c8a083575a2cfd5f722a69a2193ec8397c316ea2938a08f57ed2a2f WatchSource:0}: Error finding container 038a22475c8a083575a2cfd5f722a69a2193ec8397c316ea2938a08f57ed2a2f: Status 404 returned error can't find the container with id 038a22475c8a083575a2cfd5f722a69a2193ec8397c316ea2938a08f57ed2a2f
Sep 16 18:40:43.294545 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:43.294528    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod025a78d4_1397_4057_89d3_56bbf1d96e2c.slice/crio-f402db49731bb14894067e55e86d10afa006d5cbf1bce1f9b4931de962e42639 WatchSource:0}: Error finding container f402db49731bb14894067e55e86d10afa006d5cbf1bce1f9b4931de962e42639: Status 404 returned error can't find the container with id f402db49731bb14894067e55e86d10afa006d5cbf1bce1f9b4931de962e42639
Sep 16 18:40:43.295141 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.295125    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hv7z5"
Sep 16 18:40:43.301131 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.301115    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vlcrr"
Sep 16 18:40:43.302197 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:43.302181    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4870db69_091f_43b3_944c_4fa7d195ca2d.slice/crio-de3683c055744ab9aae810ed141e036241298d851f6e4a581e73b452ab0c4707 WatchSource:0}: Error finding container de3683c055744ab9aae810ed141e036241298d851f6e4a581e73b452ab0c4707: Status 404 returned error can't find the container with id de3683c055744ab9aae810ed141e036241298d851f6e4a581e73b452ab0c4707
Sep 16 18:40:43.305132 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.305114    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wmfjw"
Sep 16 18:40:43.307893 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:43.307877    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d943a0b_9f9f_4b81_bffb_c434be495a76.slice/crio-7a057c00ef7e5569ee12a278bfed8bfede97e966957b3e8a23664c847c11542a WatchSource:0}: Error finding container 7a057c00ef7e5569ee12a278bfed8bfede97e966957b3e8a23664c847c11542a: Status 404 returned error can't find the container with id 7a057c00ef7e5569ee12a278bfed8bfede97e966957b3e8a23664c847c11542a
Sep 16 18:40:43.312612 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:40:43.312593    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod590e4879_7945_4583_b644_0caa6b1c6ebd.slice/crio-d2bb8c5601fc6021a6707bad140aa89cabf7148a45d795bde4a9e885507d612c WatchSource:0}: Error finding container d2bb8c5601fc6021a6707bad140aa89cabf7148a45d795bde4a9e885507d612c: Status 404 returned error can't find the container with id d2bb8c5601fc6021a6707bad140aa89cabf7148a45d795bde4a9e885507d612c
Sep 16 18:40:43.507198 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.507125    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:43.528578 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.528537    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b544a31-67f6-4a93-beeb-a21fd49b00b1-original-pull-secret\") pod \"global-pull-secret-syncer-88mzt\" (UID: \"2b544a31-67f6-4a93-beeb-a21fd49b00b1\") " pod="kube-system/global-pull-secret-syncer-88mzt"
Sep 16 18:40:43.528726 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:43.528671    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:43.528798 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:43.528729    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b544a31-67f6-4a93-beeb-a21fd49b00b1-original-pull-secret podName:2b544a31-67f6-4a93-beeb-a21fd49b00b1 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:44.528709808 +0000 UTC m=+3.024069591 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b544a31-67f6-4a93-beeb-a21fd49b00b1-original-pull-secret") pod "global-pull-secret-syncer-88mzt" (UID: "2b544a31-67f6-4a93-beeb-a21fd49b00b1") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:43.629191 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.629159    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba8dcba9-de75-4055-8086-3820f53b55e3-metrics-certs\") pod \"network-metrics-daemon-s229p\" (UID: \"ba8dcba9-de75-4055-8086-3820f53b55e3\") " pod="openshift-multus/network-metrics-daemon-s229p"
Sep 16 18:40:43.629343 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:43.629209    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5ttq\" (UniqueName: \"kubernetes.io/projected/8e33041b-cc50-41a3-bcc4-34942629f4cc-kube-api-access-r5ttq\") pod \"network-check-target-ggvbf\" (UID: \"8e33041b-cc50-41a3-bcc4-34942629f4cc\") " pod="openshift-network-diagnostics/network-check-target-ggvbf"
Sep 16 18:40:43.629405 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:43.629369    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:43.629405 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:43.629389    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:43.629405 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:43.629402    2577 projected.go:196] Error preparing data for projected volume kube-api-access-r5ttq for pod openshift-network-diagnostics/network-check-target-ggvbf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:43.629565 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:43.629457    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e33041b-cc50-41a3-bcc4-34942629f4cc-kube-api-access-r5ttq podName:8e33041b-cc50-41a3-bcc4-34942629f4cc nodeName:}" failed. No retries permitted until 2026-09-16 18:40:44.629438268 +0000 UTC m=+3.124798053 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-r5ttq" (UniqueName: "kubernetes.io/projected/8e33041b-cc50-41a3-bcc4-34942629f4cc-kube-api-access-r5ttq") pod "network-check-target-ggvbf" (UID: "8e33041b-cc50-41a3-bcc4-34942629f4cc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:43.629880 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:43.629862    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:43.629958 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:43.629908    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba8dcba9-de75-4055-8086-3820f53b55e3-metrics-certs podName:ba8dcba9-de75-4055-8086-3820f53b55e3 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:44.629896909 +0000 UTC m=+3.125256683 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba8dcba9-de75-4055-8086-3820f53b55e3-metrics-certs") pod "network-metrics-daemon-s229p" (UID: "ba8dcba9-de75-4055-8086-3820f53b55e3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:45.058231 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:45.057458    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:45.104622 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:45.104555    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:35:44 +0000 UTC" deadline="2028-05-07 17:10:57.594273008 +0000 UTC"
Sep 16 18:40:45.104622 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:45.104589    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14374h30m12.489688279s"
Sep 16 18:40:45.152100 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:45.152069    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggvbf"
Sep 16 18:40:45.152246 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:45.152203    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggvbf" podUID="8e33041b-cc50-41a3-bcc4-34942629f4cc"
Sep 16 18:40:45.176999 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:45.176877    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wcfql" event={"ID":"88b20525-d601-4224-bd43-397308547efd","Type":"ContainerStarted","Data":"cfb311409ba5f94efe436b2b7ce8aa6a19616ecab166d0a4420cb4574a8dcf82"}
Sep 16 18:40:45.182378 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:45.182351    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-298hg" event={"ID":"1e98550f-c964-434b-995d-60ad18764eff","Type":"ContainerStarted","Data":"67aec8a6a511dd5721e7d8bb50d4bc960e3f3e37e751c093685905196ac60f24"}
Sep 16 18:40:45.188342 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:45.188313    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-btfcs" event={"ID":"8049c55c-c4ac-44a1-a1bf-5872846445c4","Type":"ContainerStarted","Data":"e5f854d46d0b27a02c5a1f6a2a5758a65b519b2f82ac755cffd0a1fb0c7bf873"}
Sep 16 18:40:45.193674 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:45.193647    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8cxsn" event={"ID":"f935c176-f27b-4179-bc5e-802ee61499d3","Type":"ContainerStarted","Data":"245978a131f9a8a78f8092bcd6e418b4e36d4b1e560ed5420b748e61a2981299"}
Sep 16 18:40:45.196409 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:45.196385    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wmfjw" event={"ID":"590e4879-7945-4583-b644-0caa6b1c6ebd","Type":"ContainerStarted","Data":"d2bb8c5601fc6021a6707bad140aa89cabf7148a45d795bde4a9e885507d612c"}
Sep 16 18:40:45.200964 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:45.200939    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vlcrr" event={"ID":"1d943a0b-9f9f-4b81-bffb-c434be495a76","Type":"ContainerStarted","Data":"7a057c00ef7e5569ee12a278bfed8bfede97e966957b3e8a23664c847c11542a"}
Sep 16 18:40:45.211730 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:45.209230    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hv7z5" event={"ID":"4870db69-091f-43b3-944c-4fa7d195ca2d","Type":"ContainerStarted","Data":"de3683c055744ab9aae810ed141e036241298d851f6e4a581e73b452ab0c4707"}
Sep 16 18:40:45.214460 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:45.214436    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jpq9w" event={"ID":"025a78d4-1397-4057-89d3-56bbf1d96e2c","Type":"ContainerStarted","Data":"f402db49731bb14894067e55e86d10afa006d5cbf1bce1f9b4931de962e42639"}
Sep 16 18:40:45.220252 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:45.217147    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rlvsc" event={"ID":"8859cf81-dc3a-4a84-a649-efdaee26a8a2","Type":"ContainerStarted","Data":"038a22475c8a083575a2cfd5f722a69a2193ec8397c316ea2938a08f57ed2a2f"}
Sep 16 18:40:45.649213 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:45.648662    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b544a31-67f6-4a93-beeb-a21fd49b00b1-original-pull-secret\") pod \"global-pull-secret-syncer-88mzt\" (UID: \"2b544a31-67f6-4a93-beeb-a21fd49b00b1\") " pod="kube-system/global-pull-secret-syncer-88mzt"
Sep 16 18:40:45.649213 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:45.648813    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:45.649213 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:45.648872    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b544a31-67f6-4a93-beeb-a21fd49b00b1-original-pull-secret podName:2b544a31-67f6-4a93-beeb-a21fd49b00b1 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:47.648855044 +0000 UTC m=+5.031744392 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b544a31-67f6-4a93-beeb-a21fd49b00b1-original-pull-secret") pod "global-pull-secret-syncer-88mzt" (UID: "2b544a31-67f6-4a93-beeb-a21fd49b00b1") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:45.749872 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:45.749803    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba8dcba9-de75-4055-8086-3820f53b55e3-metrics-certs\") pod \"network-metrics-daemon-s229p\" (UID: \"ba8dcba9-de75-4055-8086-3820f53b55e3\") " pod="openshift-multus/network-metrics-daemon-s229p"
Sep 16 18:40:45.750044 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:45.749915    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5ttq\" (UniqueName: \"kubernetes.io/projected/8e33041b-cc50-41a3-bcc4-34942629f4cc-kube-api-access-r5ttq\") pod \"network-check-target-ggvbf\" (UID: \"8e33041b-cc50-41a3-bcc4-34942629f4cc\") " pod="openshift-network-diagnostics/network-check-target-ggvbf"
Sep 16 18:40:45.750101 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:45.750091    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:45.750151 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:45.750113    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:45.750151 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:45.750127    2577 projected.go:196] Error preparing data for projected volume kube-api-access-r5ttq for pod openshift-network-diagnostics/network-check-target-ggvbf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:45.750257 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:45.750180    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e33041b-cc50-41a3-bcc4-34942629f4cc-kube-api-access-r5ttq podName:8e33041b-cc50-41a3-bcc4-34942629f4cc nodeName:}" failed. No retries permitted until 2026-09-16 18:40:47.750162002 +0000 UTC m=+5.133051367 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-r5ttq" (UniqueName: "kubernetes.io/projected/8e33041b-cc50-41a3-bcc4-34942629f4cc-kube-api-access-r5ttq") pod "network-check-target-ggvbf" (UID: "8e33041b-cc50-41a3-bcc4-34942629f4cc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:45.750582 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:45.750563    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:45.750666 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:45.750616    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba8dcba9-de75-4055-8086-3820f53b55e3-metrics-certs podName:ba8dcba9-de75-4055-8086-3820f53b55e3 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:47.750601283 +0000 UTC m=+5.133490631 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba8dcba9-de75-4055-8086-3820f53b55e3-metrics-certs") pod "network-metrics-daemon-s229p" (UID: "ba8dcba9-de75-4055-8086-3820f53b55e3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:46.104957 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:46.104847    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:35:44 +0000 UTC" deadline="2028-06-03 02:07:26.982938748 +0000 UTC"
Sep 16 18:40:46.104957 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:46.104910    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15007h26m40.878032341s"
Sep 16 18:40:46.149618 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:46.149591    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88mzt"
Sep 16 18:40:46.149774 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:46.149721    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88mzt" podUID="2b544a31-67f6-4a93-beeb-a21fd49b00b1"
Sep 16 18:40:46.150146 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:46.150125    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s229p"
Sep 16 18:40:46.150259 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:46.150238    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s229p" podUID="ba8dcba9-de75-4055-8086-3820f53b55e3"
Sep 16 18:40:47.152591 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:47.152089    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggvbf"
Sep 16 18:40:47.152591 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:47.152207    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggvbf" podUID="8e33041b-cc50-41a3-bcc4-34942629f4cc"
Sep 16 18:40:47.666055 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:47.666020    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b544a31-67f6-4a93-beeb-a21fd49b00b1-original-pull-secret\") pod \"global-pull-secret-syncer-88mzt\" (UID: \"2b544a31-67f6-4a93-beeb-a21fd49b00b1\") " pod="kube-system/global-pull-secret-syncer-88mzt"
Sep 16 18:40:47.666234 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:47.666197    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:47.666294 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:47.666255    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b544a31-67f6-4a93-beeb-a21fd49b00b1-original-pull-secret podName:2b544a31-67f6-4a93-beeb-a21fd49b00b1 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:51.666237272 +0000 UTC m=+9.049126620 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b544a31-67f6-4a93-beeb-a21fd49b00b1-original-pull-secret") pod "global-pull-secret-syncer-88mzt" (UID: "2b544a31-67f6-4a93-beeb-a21fd49b00b1") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:47.767430 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:47.767382    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba8dcba9-de75-4055-8086-3820f53b55e3-metrics-certs\") pod \"network-metrics-daemon-s229p\" (UID: \"ba8dcba9-de75-4055-8086-3820f53b55e3\") " pod="openshift-multus/network-metrics-daemon-s229p"
Sep 16 18:40:47.767618 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:47.767450    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5ttq\" (UniqueName: \"kubernetes.io/projected/8e33041b-cc50-41a3-bcc4-34942629f4cc-kube-api-access-r5ttq\") pod \"network-check-target-ggvbf\" (UID: \"8e33041b-cc50-41a3-bcc4-34942629f4cc\") " pod="openshift-network-diagnostics/network-check-target-ggvbf"
Sep 16 18:40:47.767681 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:47.767669    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:47.767737 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:47.767691    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:47.767737 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:47.767704    2577 projected.go:196] Error preparing data for projected volume kube-api-access-r5ttq for pod openshift-network-diagnostics/network-check-target-ggvbf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:47.767844 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:47.767765    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e33041b-cc50-41a3-bcc4-34942629f4cc-kube-api-access-r5ttq podName:8e33041b-cc50-41a3-bcc4-34942629f4cc nodeName:}" failed. No retries permitted until 2026-09-16 18:40:51.767738343 +0000 UTC m=+9.150627693 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-r5ttq" (UniqueName: "kubernetes.io/projected/8e33041b-cc50-41a3-bcc4-34942629f4cc-kube-api-access-r5ttq") pod "network-check-target-ggvbf" (UID: "8e33041b-cc50-41a3-bcc4-34942629f4cc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:47.768189 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:47.768172    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:47.768245 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:47.768225    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba8dcba9-de75-4055-8086-3820f53b55e3-metrics-certs podName:ba8dcba9-de75-4055-8086-3820f53b55e3 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:51.768209457 +0000 UTC m=+9.151098804 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba8dcba9-de75-4055-8086-3820f53b55e3-metrics-certs") pod "network-metrics-daemon-s229p" (UID: "ba8dcba9-de75-4055-8086-3820f53b55e3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:48.149410 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:48.149329    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s229p"
Sep 16 18:40:48.149589 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:48.149448    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s229p" podUID="ba8dcba9-de75-4055-8086-3820f53b55e3"
Sep 16 18:40:48.149666 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:48.149592    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88mzt"
Sep 16 18:40:48.149727 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:48.149693    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88mzt" podUID="2b544a31-67f6-4a93-beeb-a21fd49b00b1"
Sep 16 18:40:49.150005 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:49.149966    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggvbf"
Sep 16 18:40:49.151775 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:49.151728    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggvbf" podUID="8e33041b-cc50-41a3-bcc4-34942629f4cc"
Sep 16 18:40:50.149781 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:50.149745    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s229p"
Sep 16 18:40:50.149975 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:50.149896    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s229p" podUID="ba8dcba9-de75-4055-8086-3820f53b55e3"
Sep 16 18:40:50.150436 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:50.150305    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88mzt"
Sep 16 18:40:50.150436 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:50.150396    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88mzt" podUID="2b544a31-67f6-4a93-beeb-a21fd49b00b1"
Sep 16 18:40:51.150132 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:51.150104    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggvbf"
Sep 16 18:40:51.150355 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:51.150235    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggvbf" podUID="8e33041b-cc50-41a3-bcc4-34942629f4cc"
Sep 16 18:40:51.699950 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:51.699918    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b544a31-67f6-4a93-beeb-a21fd49b00b1-original-pull-secret\") pod \"global-pull-secret-syncer-88mzt\" (UID: \"2b544a31-67f6-4a93-beeb-a21fd49b00b1\") " pod="kube-system/global-pull-secret-syncer-88mzt"
Sep 16 18:40:51.700320 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:51.700076    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:51.700320 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:51.700146    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b544a31-67f6-4a93-beeb-a21fd49b00b1-original-pull-secret podName:2b544a31-67f6-4a93-beeb-a21fd49b00b1 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:59.700127066 +0000 UTC m=+17.083016411 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b544a31-67f6-4a93-beeb-a21fd49b00b1-original-pull-secret") pod "global-pull-secret-syncer-88mzt" (UID: "2b544a31-67f6-4a93-beeb-a21fd49b00b1") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:51.801496 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:51.800752    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba8dcba9-de75-4055-8086-3820f53b55e3-metrics-certs\") pod \"network-metrics-daemon-s229p\" (UID: \"ba8dcba9-de75-4055-8086-3820f53b55e3\") " pod="openshift-multus/network-metrics-daemon-s229p"
Sep 16 18:40:51.801496 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:51.800804    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5ttq\" (UniqueName: \"kubernetes.io/projected/8e33041b-cc50-41a3-bcc4-34942629f4cc-kube-api-access-r5ttq\") pod \"network-check-target-ggvbf\" (UID: \"8e33041b-cc50-41a3-bcc4-34942629f4cc\") " pod="openshift-network-diagnostics/network-check-target-ggvbf"
Sep 16 18:40:51.801496 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:51.800957    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:51.801496 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:51.800977    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:51.801496 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:51.800989    2577 projected.go:196] Error preparing data for projected volume kube-api-access-r5ttq for pod openshift-network-diagnostics/network-check-target-ggvbf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:51.801496 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:51.801046    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e33041b-cc50-41a3-bcc4-34942629f4cc-kube-api-access-r5ttq podName:8e33041b-cc50-41a3-bcc4-34942629f4cc nodeName:}" failed. No retries permitted until 2026-09-16 18:40:59.801029159 +0000 UTC m=+17.183918507 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-r5ttq" (UniqueName: "kubernetes.io/projected/8e33041b-cc50-41a3-bcc4-34942629f4cc-kube-api-access-r5ttq") pod "network-check-target-ggvbf" (UID: "8e33041b-cc50-41a3-bcc4-34942629f4cc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:51.801496 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:51.801409    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:51.801496 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:51.801465    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba8dcba9-de75-4055-8086-3820f53b55e3-metrics-certs podName:ba8dcba9-de75-4055-8086-3820f53b55e3 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:59.801448193 +0000 UTC m=+17.184337550 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba8dcba9-de75-4055-8086-3820f53b55e3-metrics-certs") pod "network-metrics-daemon-s229p" (UID: "ba8dcba9-de75-4055-8086-3820f53b55e3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:52.150358 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:52.150274    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s229p"
Sep 16 18:40:52.150358 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:52.150318    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88mzt"
Sep 16 18:40:52.150597 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:52.150421    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s229p" podUID="ba8dcba9-de75-4055-8086-3820f53b55e3"
Sep 16 18:40:52.150597 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:52.150561    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88mzt" podUID="2b544a31-67f6-4a93-beeb-a21fd49b00b1"
Sep 16 18:40:53.152069 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:53.152030    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggvbf"
Sep 16 18:40:53.152578 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:53.152149    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggvbf" podUID="8e33041b-cc50-41a3-bcc4-34942629f4cc"
Sep 16 18:40:54.149954 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:54.149909    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s229p"
Sep 16 18:40:54.150113 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:54.149957    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88mzt"
Sep 16 18:40:54.150113 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:54.150096    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s229p" podUID="ba8dcba9-de75-4055-8086-3820f53b55e3"
Sep 16 18:40:54.150224 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:54.150195    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88mzt" podUID="2b544a31-67f6-4a93-beeb-a21fd49b00b1"
Sep 16 18:40:55.149969 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:55.149937    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggvbf"
Sep 16 18:40:55.150413 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:55.150070    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggvbf" podUID="8e33041b-cc50-41a3-bcc4-34942629f4cc"
Sep 16 18:40:56.149373 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:56.149339    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s229p"
Sep 16 18:40:56.149567 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:56.149383    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88mzt"
Sep 16 18:40:56.149567 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:56.149476    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s229p" podUID="ba8dcba9-de75-4055-8086-3820f53b55e3"
Sep 16 18:40:56.149674 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:56.149619    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88mzt" podUID="2b544a31-67f6-4a93-beeb-a21fd49b00b1"
Sep 16 18:40:57.150002 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:57.149970    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggvbf"
Sep 16 18:40:57.150379 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:57.150119    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggvbf" podUID="8e33041b-cc50-41a3-bcc4-34942629f4cc"
Sep 16 18:40:58.150033 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:58.149999    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88mzt"
Sep 16 18:40:58.150438 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:58.149996    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s229p"
Sep 16 18:40:58.150438 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:58.150111    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88mzt" podUID="2b544a31-67f6-4a93-beeb-a21fd49b00b1"
Sep 16 18:40:58.150438 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:58.150181    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s229p" podUID="ba8dcba9-de75-4055-8086-3820f53b55e3"
Sep 16 18:40:59.149375 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:59.149348    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggvbf"
Sep 16 18:40:59.149509 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:59.149438    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggvbf" podUID="8e33041b-cc50-41a3-bcc4-34942629f4cc"
Sep 16 18:40:59.759609 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:59.759516    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b544a31-67f6-4a93-beeb-a21fd49b00b1-original-pull-secret\") pod \"global-pull-secret-syncer-88mzt\" (UID: \"2b544a31-67f6-4a93-beeb-a21fd49b00b1\") " pod="kube-system/global-pull-secret-syncer-88mzt"
Sep 16 18:40:59.760055 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:59.759649    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:59.760055 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:59.759720    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b544a31-67f6-4a93-beeb-a21fd49b00b1-original-pull-secret podName:2b544a31-67f6-4a93-beeb-a21fd49b00b1 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:15.759704124 +0000 UTC m=+33.142593482 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b544a31-67f6-4a93-beeb-a21fd49b00b1-original-pull-secret") pod "global-pull-secret-syncer-88mzt" (UID: "2b544a31-67f6-4a93-beeb-a21fd49b00b1") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:59.860663 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:59.860623    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba8dcba9-de75-4055-8086-3820f53b55e3-metrics-certs\") pod \"network-metrics-daemon-s229p\" (UID: \"ba8dcba9-de75-4055-8086-3820f53b55e3\") " pod="openshift-multus/network-metrics-daemon-s229p"
Sep 16 18:40:59.860811 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:40:59.860665    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5ttq\" (UniqueName: \"kubernetes.io/projected/8e33041b-cc50-41a3-bcc4-34942629f4cc-kube-api-access-r5ttq\") pod \"network-check-target-ggvbf\" (UID: \"8e33041b-cc50-41a3-bcc4-34942629f4cc\") " pod="openshift-network-diagnostics/network-check-target-ggvbf"
Sep 16 18:40:59.860811 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:59.860798    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:59.860916 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:59.860822    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:59.860916 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:59.860844    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:59.860916 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:59.860857    2577 projected.go:196] Error preparing data for projected volume kube-api-access-r5ttq for pod openshift-network-diagnostics/network-check-target-ggvbf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:59.860916 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:59.860872    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba8dcba9-de75-4055-8086-3820f53b55e3-metrics-certs podName:ba8dcba9-de75-4055-8086-3820f53b55e3 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:15.860849865 +0000 UTC m=+33.243739221 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba8dcba9-de75-4055-8086-3820f53b55e3-metrics-certs") pod "network-metrics-daemon-s229p" (UID: "ba8dcba9-de75-4055-8086-3820f53b55e3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:59.860916 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:40:59.860905    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e33041b-cc50-41a3-bcc4-34942629f4cc-kube-api-access-r5ttq podName:8e33041b-cc50-41a3-bcc4-34942629f4cc nodeName:}" failed. No retries permitted until 2026-09-16 18:41:15.860889766 +0000 UTC m=+33.243779140 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-r5ttq" (UniqueName: "kubernetes.io/projected/8e33041b-cc50-41a3-bcc4-34942629f4cc-kube-api-access-r5ttq") pod "network-check-target-ggvbf" (UID: "8e33041b-cc50-41a3-bcc4-34942629f4cc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:41:00.149901 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:00.149867    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s229p"
Sep 16 18:41:00.150086 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:00.149909    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88mzt"
Sep 16 18:41:00.150086 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:00.150007    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s229p" podUID="ba8dcba9-de75-4055-8086-3820f53b55e3"
Sep 16 18:41:00.150203 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:00.150139    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88mzt" podUID="2b544a31-67f6-4a93-beeb-a21fd49b00b1"
Sep 16 18:41:01.149928 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:01.149894    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggvbf"
Sep 16 18:41:01.150365 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:01.150029    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggvbf" podUID="8e33041b-cc50-41a3-bcc4-34942629f4cc"
Sep 16 18:41:02.150145 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:02.150114    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88mzt"
Sep 16 18:41:02.150704 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:02.150127    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s229p"
Sep 16 18:41:02.150704 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:02.150254    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88mzt" podUID="2b544a31-67f6-4a93-beeb-a21fd49b00b1"
Sep 16 18:41:02.150704 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:02.150343    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s229p" podUID="ba8dcba9-de75-4055-8086-3820f53b55e3"
Sep 16 18:41:03.150019 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:03.149982    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggvbf"
Sep 16 18:41:03.150197 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:03.150079    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggvbf" podUID="8e33041b-cc50-41a3-bcc4-34942629f4cc"
Sep 16 18:41:04.150120 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:04.149943    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s229p"
Sep 16 18:41:04.150249 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:04.149972    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88mzt"
Sep 16 18:41:04.150249 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:04.150205    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s229p" podUID="ba8dcba9-de75-4055-8086-3820f53b55e3"
Sep 16 18:41:04.150819 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:04.150256    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88mzt" podUID="2b544a31-67f6-4a93-beeb-a21fd49b00b1"
Sep 16 18:41:04.259045 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:04.259009    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rlvsc" event={"ID":"8859cf81-dc3a-4a84-a649-efdaee26a8a2","Type":"ContainerStarted","Data":"7709c1699923f1b1ea4844e0037f6f6eb31ec567b10e542ad4247f9bcddd92b5"}
Sep 16 18:41:04.262520 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:04.262492    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wcfql" event={"ID":"88b20525-d601-4224-bd43-397308547efd","Type":"ContainerStarted","Data":"9b13b4655019e10061c20b6b5746b0dcd5776992189c453b423932bda87588ab"}
Sep 16 18:41:04.262634 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:04.262574    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wcfql" event={"ID":"88b20525-d601-4224-bd43-397308547efd","Type":"ContainerStarted","Data":"2ff078a311bd27bc631f053484bc5085c2e36eb30d4c44f16fe168cbf24405bc"}
Sep 16 18:41:04.262634 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:04.262611    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wcfql" event={"ID":"88b20525-d601-4224-bd43-397308547efd","Type":"ContainerStarted","Data":"a5e6e5f31425048234b0e05eae7ca78ec6561aa9ca35f029e48dfb5951553536"}
Sep 16 18:41:04.262739 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:04.262653    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wcfql" event={"ID":"88b20525-d601-4224-bd43-397308547efd","Type":"ContainerStarted","Data":"6b65de0f1399fcf954f879709cf5d649974c11b209a6a8afbc86065cf361ed98"}
Sep 16 18:41:04.262739 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:04.262691    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wcfql" event={"ID":"88b20525-d601-4224-bd43-397308547efd","Type":"ContainerStarted","Data":"9492d82b47de3c0dfd5aaa39bf8139b566b1dd3da0242d80362e7da19bbda961"}
Sep 16 18:41:04.262739 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:04.262727    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wcfql" event={"ID":"88b20525-d601-4224-bd43-397308547efd","Type":"ContainerStarted","Data":"c728c909371e6d05399f9f1a7c952522da23fc6e4c82ae9131547f2305f40e24"}
Sep 16 18:41:04.264379 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:04.264351    2577 generic.go:356] "Generic (PLEG): container finished" podID="c28a4ec2fbe427bdfc7c14e0446ecb67" containerID="7accc284287e8caafada8bdc313615ec769ff3a860aa173bb60302ee39e6867b" exitCode=0
Sep 16 18:41:04.264499 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:04.264414    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-247.ec2.internal" event={"ID":"c28a4ec2fbe427bdfc7c14e0446ecb67","Type":"ContainerDied","Data":"7accc284287e8caafada8bdc313615ec769ff3a860aa173bb60302ee39e6867b"}
Sep 16 18:41:04.266063 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:04.266031    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-247.ec2.internal" event={"ID":"146a57ab6e613bbac55bc1f7a3764a51","Type":"ContainerStarted","Data":"eaf093ff1a69c193e0eb66bdfb507feaaa6db85abcedad6be79fae0becd59efd"}
Sep 16 18:41:04.267815 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:04.267791    2577 generic.go:356] "Generic (PLEG): container finished" podID="8049c55c-c4ac-44a1-a1bf-5872846445c4" containerID="85dbf0a1c765a1b9365014321ad68f21e2129df73166a25b58a802287c047394" exitCode=0
Sep 16 18:41:04.267911 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:04.267834    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-btfcs" event={"ID":"8049c55c-c4ac-44a1-a1bf-5872846445c4","Type":"ContainerDied","Data":"85dbf0a1c765a1b9365014321ad68f21e2129df73166a25b58a802287c047394"}
Sep 16 18:41:04.271091 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:04.270164    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8cxsn" event={"ID":"f935c176-f27b-4179-bc5e-802ee61499d3","Type":"ContainerStarted","Data":"21c18f26797611dc22a5af8413d6bbfd78e129b4586a39fdb93aca2634290483"}
Sep 16 18:41:04.272365 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:04.272341    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wmfjw" event={"ID":"590e4879-7945-4583-b644-0caa6b1c6ebd","Type":"ContainerStarted","Data":"c0247ac01406081fc87b9ee0096b4bca3665d3cabfd25ec370ed3a16979e29a8"}
Sep 16 18:41:04.275417 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:04.275393    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vlcrr" event={"ID":"1d943a0b-9f9f-4b81-bffb-c434be495a76","Type":"ContainerStarted","Data":"a46bed69db821f98a9cc154ed0b58d5f033a51c99e86a854e9448c3c2dbd9d6b"}
Sep 16 18:41:04.277582 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:04.277558    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hv7z5" event={"ID":"4870db69-091f-43b3-944c-4fa7d195ca2d","Type":"ContainerStarted","Data":"6c5474f80854ed37efc9fb0c08aab7b2f82efe77682aef4047ee2a27d1e0d988"}
Sep 16 18:41:04.279243 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:04.279210    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jpq9w" event={"ID":"025a78d4-1397-4057-89d3-56bbf1d96e2c","Type":"ContainerStarted","Data":"4108cb6e174d84cb8eb0e0dede96d36a20e02e005f36de181b30319e38488f2c"}
Sep 16 18:41:04.281048 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:04.280996    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rlvsc" podStartSLOduration=2.315269506 podStartE2EDuration="21.280979759s" totalImagesPullingTime="18.965710253s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:04.280680243 +0000 UTC m=+21.663569609" watchObservedRunningTime="2026-09-16 18:41:04.280979759 +0000 UTC m=+21.663869126"
Sep 16 18:41:04.299632 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:04.299597    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-247.ec2.internal" podStartSLOduration=20.299585401 podStartE2EDuration="20.299585401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:04.299496829 +0000 UTC m=+21.682386195" watchObservedRunningTime="2026-09-16 18:41:04.299585401 +0000 UTC m=+21.682474767"
Sep 16 18:41:04.319041 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:04.319007    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8cxsn" podStartSLOduration=2.493097927 podStartE2EDuration="21.318994537s" totalImagesPullingTime="18.82589661s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:04.318857461 +0000 UTC m=+21.701746825" watchObservedRunningTime="2026-09-16 18:41:04.318994537 +0000 UTC m=+21.701883901"
Sep 16 18:41:04.393867 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:04.393827    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jpq9w" podStartSLOduration=6.34421317 podStartE2EDuration="21.39381547s" totalImagesPullingTime="15.0496023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:04.371027558 +0000 UTC m=+21.753916923" watchObservedRunningTime="2026-09-16 18:41:04.39381547 +0000 UTC m=+21.776704992"
Sep 16 18:41:04.393975 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:04.393943    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vlcrr" podStartSLOduration=2.617989351 podStartE2EDuration="21.393938876s" totalImagesPullingTime="18.775949525s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:04.393656314 +0000 UTC m=+21.776545670" watchObservedRunningTime="2026-09-16 18:41:04.393938876 +0000 UTC m=+21.776828241"
Sep 16 18:41:04.821931 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:04.821896    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 18:41:05.098193 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:05.098049    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:41:04.821926588Z","UUID":"764c7212-06b8-4e43-b86e-f5efb9d4743e","Handler":null,"Name":"","Endpoint":""}
Sep 16 18:41:05.099685 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:05.099657    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 18:41:05.099685 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:05.099691    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 18:41:05.150103 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:05.150076    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggvbf"
Sep 16 18:41:05.150287 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:05.150236    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggvbf" podUID="8e33041b-cc50-41a3-bcc4-34942629f4cc"
Sep 16 18:41:05.283207 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:05.283168    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-298hg" event={"ID":"1e98550f-c964-434b-995d-60ad18764eff","Type":"ContainerStarted","Data":"7f17482efc74bdc222495c48de88239b8c726a070258d1ad7898eeadcb274e95"}
Sep 16 18:41:05.285765 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:05.285734    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hv7z5" event={"ID":"4870db69-091f-43b3-944c-4fa7d195ca2d","Type":"ContainerStarted","Data":"81e8415bd9ff6d3b9fb3c62b1659e75430426e5bfde0b6717ae3cf0b57499d4f"}
Sep 16 18:41:05.311508 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:05.310177    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-wmfjw" podStartSLOduration=3.552737103 podStartE2EDuration="22.310159139s" totalImagesPullingTime="18.757422036s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:04.414245673 +0000 UTC m=+21.797135038" watchObservedRunningTime="2026-09-16 18:41:05.310159139 +0000 UTC m=+22.693048505"
Sep 16 18:41:05.311508 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:05.310367    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-298hg" podStartSLOduration=3.487185357 podStartE2EDuration="22.310360507s" totalImagesPullingTime="18.82317515s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:05.309783577 +0000 UTC m=+22.692672942" watchObservedRunningTime="2026-09-16 18:41:05.310360507 +0000 UTC m=+22.693249873"
Sep 16 18:41:05.660020 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:05.659987    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jpq9w"
Sep 16 18:41:05.660501 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:05.660480    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jpq9w"
Sep 16 18:41:06.150194 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:06.150174    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s229p"
Sep 16 18:41:06.150329 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:06.150305    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s229p" podUID="ba8dcba9-de75-4055-8086-3820f53b55e3"
Sep 16 18:41:06.150655 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:06.150313    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88mzt"
Sep 16 18:41:06.150655 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:06.150430    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88mzt" podUID="2b544a31-67f6-4a93-beeb-a21fd49b00b1"
Sep 16 18:41:06.291808 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:06.291776    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wcfql" event={"ID":"88b20525-d601-4224-bd43-397308547efd","Type":"ContainerStarted","Data":"5a66cdfde415e0788639ae6f29af7f091fff3690e92a8445919cbc793744fce4"}
Sep 16 18:41:06.293503 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:06.293471    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-247.ec2.internal" event={"ID":"c28a4ec2fbe427bdfc7c14e0446ecb67","Type":"ContainerStarted","Data":"eec72fc7ba163959a8b2b18ebee7508ed66c414980a2880b1a36c5c116cccb99"}
Sep 16 18:41:06.295594 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:06.295565    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hv7z5" event={"ID":"4870db69-091f-43b3-944c-4fa7d195ca2d","Type":"ContainerStarted","Data":"809c8764dd89d28ec55ad50b767f502d04604d2372de8d648f11943a07ba414c"}
Sep 16 18:41:06.314839 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:06.314468    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-247.ec2.internal" podStartSLOduration=22.31444983 podStartE2EDuration="22.31444983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:06.313032965 +0000 UTC m=+23.695922334" watchObservedRunningTime="2026-09-16 18:41:06.31444983 +0000 UTC m=+23.697339196"
Sep 16 18:41:06.335804 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:06.335753    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hv7z5" podStartSLOduration=2.575905727 podStartE2EDuration="23.335725617s" totalImagesPullingTime="20.75981989s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:06.335388489 +0000 UTC m=+23.718277857" watchObservedRunningTime="2026-09-16 18:41:06.335725617 +0000 UTC m=+23.718614984"
Sep 16 18:41:07.149802 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:07.149774    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggvbf"
Sep 16 18:41:07.149976 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:07.149886    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggvbf" podUID="8e33041b-cc50-41a3-bcc4-34942629f4cc"
Sep 16 18:41:07.297444 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:07.297418    2577 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 18:41:08.149744 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:08.149714    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s229p"
Sep 16 18:41:08.149940 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:08.149751    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88mzt"
Sep 16 18:41:08.149940 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:08.149921    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s229p" podUID="ba8dcba9-de75-4055-8086-3820f53b55e3"
Sep 16 18:41:08.150048 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:08.149936    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88mzt" podUID="2b544a31-67f6-4a93-beeb-a21fd49b00b1"
Sep 16 18:41:08.701424 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:08.701296    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jpq9w"
Sep 16 18:41:08.701786 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:08.701505    2577 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 18:41:08.701915 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:08.701899    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jpq9w"
Sep 16 18:41:09.149450 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:09.149430    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggvbf"
Sep 16 18:41:09.149584 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:09.149514    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggvbf" podUID="8e33041b-cc50-41a3-bcc4-34942629f4cc"
Sep 16 18:41:09.302512 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:09.302481    2577 generic.go:356] "Generic (PLEG): container finished" podID="8049c55c-c4ac-44a1-a1bf-5872846445c4" containerID="1266ac31fd84537a551880c4cb27ace19201c10bb7361059e5c3eae541704c33" exitCode=0
Sep 16 18:41:09.302666 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:09.302576    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-btfcs" event={"ID":"8049c55c-c4ac-44a1-a1bf-5872846445c4","Type":"ContainerDied","Data":"1266ac31fd84537a551880c4cb27ace19201c10bb7361059e5c3eae541704c33"}
Sep 16 18:41:09.306727 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:09.306707    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wcfql" event={"ID":"88b20525-d601-4224-bd43-397308547efd","Type":"ContainerStarted","Data":"c753584452a6b24c715eafe3c3fb972e6629a86ee21a2f72735e134912c4f606"}
Sep 16 18:41:09.306854 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:09.306841    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:41:09.322196 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:09.322172    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:41:09.366456 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:09.366416    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wcfql" podStartSLOduration=7.361559858 podStartE2EDuration="26.366404087s" totalImagesPullingTime="19.004844229s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:09.366290766 +0000 UTC m=+26.749180130" watchObservedRunningTime="2026-09-16 18:41:09.366404087 +0000 UTC m=+26.749293451"
Sep 16 18:41:10.149379 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:10.149354    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88mzt"
Sep 16 18:41:10.149764 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:10.149448    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88mzt" podUID="2b544a31-67f6-4a93-beeb-a21fd49b00b1"
Sep 16 18:41:10.149764 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:10.149476    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s229p"
Sep 16 18:41:10.149764 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:10.149546    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s229p" podUID="ba8dcba9-de75-4055-8086-3820f53b55e3"
Sep 16 18:41:10.310546 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:10.310459    2577 generic.go:356] "Generic (PLEG): container finished" podID="8049c55c-c4ac-44a1-a1bf-5872846445c4" containerID="3578cba3a8b740c297613746cc771a44fd383ef7a8daf49b84d4b1073b7c4d95" exitCode=0
Sep 16 18:41:10.310546 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:10.310515    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-btfcs" event={"ID":"8049c55c-c4ac-44a1-a1bf-5872846445c4","Type":"ContainerDied","Data":"3578cba3a8b740c297613746cc771a44fd383ef7a8daf49b84d4b1073b7c4d95"}
Sep 16 18:41:10.311206 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:10.311062    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:41:10.311206 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:10.311154    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:41:10.325712 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:10.325693    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:41:10.712681 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:10.712655    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-88mzt"]
Sep 16 18:41:10.712844 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:10.712759    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88mzt"
Sep 16 18:41:10.712902 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:10.712867    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88mzt" podUID="2b544a31-67f6-4a93-beeb-a21fd49b00b1"
Sep 16 18:41:10.716105 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:10.716085    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s229p"]
Sep 16 18:41:10.716207 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:10.716174    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s229p"
Sep 16 18:41:10.716274 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:10.716258    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s229p" podUID="ba8dcba9-de75-4055-8086-3820f53b55e3"
Sep 16 18:41:10.716695 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:10.716671    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ggvbf"]
Sep 16 18:41:10.716784 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:10.716767    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggvbf"
Sep 16 18:41:10.716873 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:10.716854    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggvbf" podUID="8e33041b-cc50-41a3-bcc4-34942629f4cc"
Sep 16 18:41:12.150235 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:12.150046    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggvbf"
Sep 16 18:41:12.150804 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:12.150055    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s229p"
Sep 16 18:41:12.150804 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:12.150320    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggvbf" podUID="8e33041b-cc50-41a3-bcc4-34942629f4cc"
Sep 16 18:41:12.150804 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:12.150071    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88mzt"
Sep 16 18:41:12.150804 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:12.150394    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s229p" podUID="ba8dcba9-de75-4055-8086-3820f53b55e3"
Sep 16 18:41:12.150804 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:12.150473    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88mzt" podUID="2b544a31-67f6-4a93-beeb-a21fd49b00b1"
Sep 16 18:41:12.317260 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:12.317223    2577 generic.go:356] "Generic (PLEG): container finished" podID="8049c55c-c4ac-44a1-a1bf-5872846445c4" containerID="3490fb53c35e62b3128480d6d52a1ffbfc840edf8a60992cad2ed37335e2db19" exitCode=0
Sep 16 18:41:12.317399 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:12.317264    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-btfcs" event={"ID":"8049c55c-c4ac-44a1-a1bf-5872846445c4","Type":"ContainerDied","Data":"3490fb53c35e62b3128480d6d52a1ffbfc840edf8a60992cad2ed37335e2db19"}
Sep 16 18:41:14.150342 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:14.150297    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggvbf"
Sep 16 18:41:14.150794 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:14.150309    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88mzt"
Sep 16 18:41:14.150794 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:14.150420    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggvbf" podUID="8e33041b-cc50-41a3-bcc4-34942629f4cc"
Sep 16 18:41:14.150794 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:14.150313    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s229p"
Sep 16 18:41:14.150794 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:14.150481    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88mzt" podUID="2b544a31-67f6-4a93-beeb-a21fd49b00b1"
Sep 16 18:41:14.150794 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:14.150578    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s229p" podUID="ba8dcba9-de75-4055-8086-3820f53b55e3"
Sep 16 18:41:15.777875 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:15.777825    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b544a31-67f6-4a93-beeb-a21fd49b00b1-original-pull-secret\") pod \"global-pull-secret-syncer-88mzt\" (UID: \"2b544a31-67f6-4a93-beeb-a21fd49b00b1\") " pod="kube-system/global-pull-secret-syncer-88mzt"
Sep 16 18:41:15.778400 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:15.777960    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:41:15.778400 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:15.778025    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b544a31-67f6-4a93-beeb-a21fd49b00b1-original-pull-secret podName:2b544a31-67f6-4a93-beeb-a21fd49b00b1 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:47.778008186 +0000 UTC m=+65.160897547 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b544a31-67f6-4a93-beeb-a21fd49b00b1-original-pull-secret") pod "global-pull-secret-syncer-88mzt" (UID: "2b544a31-67f6-4a93-beeb-a21fd49b00b1") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:41:15.879103 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:15.879071    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5ttq\" (UniqueName: \"kubernetes.io/projected/8e33041b-cc50-41a3-bcc4-34942629f4cc-kube-api-access-r5ttq\") pod \"network-check-target-ggvbf\" (UID: \"8e33041b-cc50-41a3-bcc4-34942629f4cc\") " pod="openshift-network-diagnostics/network-check-target-ggvbf"
Sep 16 18:41:15.879268 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:15.879153    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba8dcba9-de75-4055-8086-3820f53b55e3-metrics-certs\") pod \"network-metrics-daemon-s229p\" (UID: \"ba8dcba9-de75-4055-8086-3820f53b55e3\") " pod="openshift-multus/network-metrics-daemon-s229p"
Sep 16 18:41:15.879268 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:15.879249    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:41:15.879380 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:15.879265    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:41:15.879380 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:15.879292    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:41:15.879380 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:15.879306    2577 projected.go:196] Error preparing data for projected volume kube-api-access-r5ttq for pod openshift-network-diagnostics/network-check-target-ggvbf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:41:15.879380 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:15.879310    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba8dcba9-de75-4055-8086-3820f53b55e3-metrics-certs podName:ba8dcba9-de75-4055-8086-3820f53b55e3 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:47.879293162 +0000 UTC m=+65.262182510 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba8dcba9-de75-4055-8086-3820f53b55e3-metrics-certs") pod "network-metrics-daemon-s229p" (UID: "ba8dcba9-de75-4055-8086-3820f53b55e3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:41:15.879380 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:15.879357    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e33041b-cc50-41a3-bcc4-34942629f4cc-kube-api-access-r5ttq podName:8e33041b-cc50-41a3-bcc4-34942629f4cc nodeName:}" failed. No retries permitted until 2026-09-16 18:41:47.879339499 +0000 UTC m=+65.262228854 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-r5ttq" (UniqueName: "kubernetes.io/projected/8e33041b-cc50-41a3-bcc4-34942629f4cc-kube-api-access-r5ttq") pod "network-check-target-ggvbf" (UID: "8e33041b-cc50-41a3-bcc4-34942629f4cc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:41:16.149687 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:16.149658    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggvbf"
Sep 16 18:41:16.149821 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:16.149662    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88mzt"
Sep 16 18:41:16.149821 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:16.149754    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggvbf" podUID="8e33041b-cc50-41a3-bcc4-34942629f4cc"
Sep 16 18:41:16.149893 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:16.149662    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s229p"
Sep 16 18:41:16.149893 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:16.149835    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88mzt" podUID="2b544a31-67f6-4a93-beeb-a21fd49b00b1"
Sep 16 18:41:16.149954 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:16.149904    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s229p" podUID="ba8dcba9-de75-4055-8086-3820f53b55e3"
Sep 16 18:41:17.331931 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:17.331887    2577 generic.go:356] "Generic (PLEG): container finished" podID="8049c55c-c4ac-44a1-a1bf-5872846445c4" containerID="1a8ac03bf35641d761f29d95284ec999174a1231e9063f506ce376f034b76681" exitCode=0
Sep 16 18:41:17.332361 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:17.331938    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-btfcs" event={"ID":"8049c55c-c4ac-44a1-a1bf-5872846445c4","Type":"ContainerDied","Data":"1a8ac03bf35641d761f29d95284ec999174a1231e9063f506ce376f034b76681"}
Sep 16 18:41:17.433291 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:17.433265    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-247.ec2.internal" event="NodeReady"
Sep 16 18:41:17.433427 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:17.433394    2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 18:41:17.490987 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:17.490963    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-sxvsf"]
Sep 16 18:41:17.504869 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:17.504847    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ghpxc"]
Sep 16 18:41:17.505034 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:17.505014    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sxvsf"
Sep 16 18:41:17.507436 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:17.507414    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-67vjb\""
Sep 16 18:41:17.507514 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:17.507484    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 18:41:17.507713 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:17.507700    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 18:41:17.516131 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:17.516108    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sxvsf"]
Sep 16 18:41:17.516220 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:17.516204    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ghpxc"
Sep 16 18:41:17.518724 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:17.518707    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 18:41:17.518865 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:17.518840    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 18:41:17.518908 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:17.518874    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 18:41:17.518908 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:17.518896    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-hq9jm\""
Sep 16 18:41:17.531624 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:17.531602    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ghpxc"]
Sep 16 18:41:17.590351 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:17.590297    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a2f00df-e26f-4f86-89ec-eefef7fa3ed0-metrics-tls\") pod \"dns-default-sxvsf\" (UID: \"6a2f00df-e26f-4f86-89ec-eefef7fa3ed0\") " pod="openshift-dns/dns-default-sxvsf"
Sep 16 18:41:17.590351 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:17.590325    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a2f00df-e26f-4f86-89ec-eefef7fa3ed0-tmp-dir\") pod \"dns-default-sxvsf\" (UID: \"6a2f00df-e26f-4f86-89ec-eefef7fa3ed0\") " pod="openshift-dns/dns-default-sxvsf"
Sep 16 18:41:17.590484 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:17.590360    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wls4\" (UniqueName: \"kubernetes.io/projected/6a2f00df-e26f-4f86-89ec-eefef7fa3ed0-kube-api-access-2wls4\") pod \"dns-default-sxvsf\" (UID: \"6a2f00df-e26f-4f86-89ec-eefef7fa3ed0\") " pod="openshift-dns/dns-default-sxvsf"
Sep 16 18:41:17.590484 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:17.590415    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f5bcc8e6-e1b2-4b7d-840f-f4ca1b31b520-cert\") pod \"ingress-canary-ghpxc\" (UID: \"f5bcc8e6-e1b2-4b7d-840f-f4ca1b31b520\") " pod="openshift-ingress-canary/ingress-canary-ghpxc"
Sep 16 18:41:17.590484 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:17.590479    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6a2f00df-e26f-4f86-89ec-eefef7fa3ed0-config-volume\") pod \"dns-default-sxvsf\" (UID: \"6a2f00df-e26f-4f86-89ec-eefef7fa3ed0\") " pod="openshift-dns/dns-default-sxvsf"
Sep 16 18:41:17.590608 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:17.590509    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4jt9\" (UniqueName: \"kubernetes.io/projected/f5bcc8e6-e1b2-4b7d-840f-f4ca1b31b520-kube-api-access-k4jt9\") pod \"ingress-canary-ghpxc\" (UID: \"f5bcc8e6-e1b2-4b7d-840f-f4ca1b31b520\") " pod="openshift-ingress-canary/ingress-canary-ghpxc"
Sep 16 18:41:17.691549 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:17.691506    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a2f00df-e26f-4f86-89ec-eefef7fa3ed0-metrics-tls\") pod \"dns-default-sxvsf\" (UID: \"6a2f00df-e26f-4f86-89ec-eefef7fa3ed0\") " pod="openshift-dns/dns-default-sxvsf"
Sep 16 18:41:17.691661 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:17.691557    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a2f00df-e26f-4f86-89ec-eefef7fa3ed0-tmp-dir\") pod \"dns-default-sxvsf\" (UID: \"6a2f00df-e26f-4f86-89ec-eefef7fa3ed0\") " pod="openshift-dns/dns-default-sxvsf"
Sep 16 18:41:17.691661 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:17.691587    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wls4\" (UniqueName: \"kubernetes.io/projected/6a2f00df-e26f-4f86-89ec-eefef7fa3ed0-kube-api-access-2wls4\") pod \"dns-default-sxvsf\" (UID: \"6a2f00df-e26f-4f86-89ec-eefef7fa3ed0\") " pod="openshift-dns/dns-default-sxvsf"
Sep 16 18:41:17.691661 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:17.691608    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f5bcc8e6-e1b2-4b7d-840f-f4ca1b31b520-cert\") pod \"ingress-canary-ghpxc\" (UID: \"f5bcc8e6-e1b2-4b7d-840f-f4ca1b31b520\") " pod="openshift-ingress-canary/ingress-canary-ghpxc"
Sep 16 18:41:17.691661 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:17.691640    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6a2f00df-e26f-4f86-89ec-eefef7fa3ed0-config-volume\") pod \"dns-default-sxvsf\" (UID: \"6a2f00df-e26f-4f86-89ec-eefef7fa3ed0\") " pod="openshift-dns/dns-default-sxvsf"
Sep 16 18:41:17.691661 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:17.691646    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:17.691895 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:17.691674    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4jt9\" (UniqueName: \"kubernetes.io/projected/f5bcc8e6-e1b2-4b7d-840f-f4ca1b31b520-kube-api-access-k4jt9\") pod \"ingress-canary-ghpxc\" (UID: \"f5bcc8e6-e1b2-4b7d-840f-f4ca1b31b520\") " pod="openshift-ingress-canary/ingress-canary-ghpxc"
Sep 16 18:41:17.691895 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:17.691699    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a2f00df-e26f-4f86-89ec-eefef7fa3ed0-metrics-tls podName:6a2f00df-e26f-4f86-89ec-eefef7fa3ed0 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:18.191678624 +0000 UTC m=+35.574567968 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a2f00df-e26f-4f86-89ec-eefef7fa3ed0-metrics-tls") pod "dns-default-sxvsf" (UID: "6a2f00df-e26f-4f86-89ec-eefef7fa3ed0") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:17.692015 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:17.691997    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a2f00df-e26f-4f86-89ec-eefef7fa3ed0-tmp-dir\") pod \"dns-default-sxvsf\" (UID: \"6a2f00df-e26f-4f86-89ec-eefef7fa3ed0\") " pod="openshift-dns/dns-default-sxvsf"
Sep 16 18:41:17.692255 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:17.692241    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6a2f00df-e26f-4f86-89ec-eefef7fa3ed0-config-volume\") pod \"dns-default-sxvsf\" (UID: \"6a2f00df-e26f-4f86-89ec-eefef7fa3ed0\") " pod="openshift-dns/dns-default-sxvsf"
Sep 16 18:41:17.696032 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:17.695914    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f5bcc8e6-e1b2-4b7d-840f-f4ca1b31b520-cert\") pod \"ingress-canary-ghpxc\" (UID: \"f5bcc8e6-e1b2-4b7d-840f-f4ca1b31b520\") " pod="openshift-ingress-canary/ingress-canary-ghpxc"
Sep 16 18:41:17.710856 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:17.710834    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wls4\" (UniqueName: \"kubernetes.io/projected/6a2f00df-e26f-4f86-89ec-eefef7fa3ed0-kube-api-access-2wls4\") pod \"dns-default-sxvsf\" (UID: \"6a2f00df-e26f-4f86-89ec-eefef7fa3ed0\") " pod="openshift-dns/dns-default-sxvsf"
Sep 16 18:41:17.710945 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:17.710930    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4jt9\" (UniqueName: \"kubernetes.io/projected/f5bcc8e6-e1b2-4b7d-840f-f4ca1b31b520-kube-api-access-k4jt9\") pod \"ingress-canary-ghpxc\" (UID: \"f5bcc8e6-e1b2-4b7d-840f-f4ca1b31b520\") " pod="openshift-ingress-canary/ingress-canary-ghpxc"
Sep 16 18:41:17.824166 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:17.824146    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ghpxc"
Sep 16 18:41:18.140299 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:18.140272    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ghpxc"]
Sep 16 18:41:18.144049 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:41:18.144023    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5bcc8e6_e1b2_4b7d_840f_f4ca1b31b520.slice/crio-1e6c95655d495fba4040db1b03e7161cff111dac0a179524a311c5b59988eabd WatchSource:0}: Error finding container 1e6c95655d495fba4040db1b03e7161cff111dac0a179524a311c5b59988eabd: Status 404 returned error can't find the container with id 1e6c95655d495fba4040db1b03e7161cff111dac0a179524a311c5b59988eabd
Sep 16 18:41:18.149452 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:18.149434    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s229p"
Sep 16 18:41:18.149887 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:18.149867    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88mzt"
Sep 16 18:41:18.149943 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:18.149912    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggvbf"
Sep 16 18:41:18.152618 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:18.152597    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:41:18.153672 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:18.153654    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s9z5g\""
Sep 16 18:41:18.153782 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:18.153685    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:41:18.153782 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:18.153693    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:41:18.153782 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:18.153702    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:41:18.153782 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:18.153760    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-g58c8\""
Sep 16 18:41:18.195749 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:18.195732    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a2f00df-e26f-4f86-89ec-eefef7fa3ed0-metrics-tls\") pod \"dns-default-sxvsf\" (UID: \"6a2f00df-e26f-4f86-89ec-eefef7fa3ed0\") " pod="openshift-dns/dns-default-sxvsf"
Sep 16 18:41:18.195858 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:18.195847    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:18.195900 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:18.195891    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a2f00df-e26f-4f86-89ec-eefef7fa3ed0-metrics-tls podName:6a2f00df-e26f-4f86-89ec-eefef7fa3ed0 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:19.195879833 +0000 UTC m=+36.578769176 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a2f00df-e26f-4f86-89ec-eefef7fa3ed0-metrics-tls") pod "dns-default-sxvsf" (UID: "6a2f00df-e26f-4f86-89ec-eefef7fa3ed0") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:18.337488 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:18.337465    2577 generic.go:356] "Generic (PLEG): container finished" podID="8049c55c-c4ac-44a1-a1bf-5872846445c4" containerID="de71891763c43cf8ebf3b18af928b4cfe9fc11ea796671cc05e3d5af3224e5c0" exitCode=0
Sep 16 18:41:18.337938 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:18.337543    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-btfcs" event={"ID":"8049c55c-c4ac-44a1-a1bf-5872846445c4","Type":"ContainerDied","Data":"de71891763c43cf8ebf3b18af928b4cfe9fc11ea796671cc05e3d5af3224e5c0"}
Sep 16 18:41:18.338759 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:18.338741    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ghpxc" event={"ID":"f5bcc8e6-e1b2-4b7d-840f-f4ca1b31b520","Type":"ContainerStarted","Data":"1e6c95655d495fba4040db1b03e7161cff111dac0a179524a311c5b59988eabd"}
Sep 16 18:41:19.202206 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:19.202170    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a2f00df-e26f-4f86-89ec-eefef7fa3ed0-metrics-tls\") pod \"dns-default-sxvsf\" (UID: \"6a2f00df-e26f-4f86-89ec-eefef7fa3ed0\") " pod="openshift-dns/dns-default-sxvsf"
Sep 16 18:41:19.202365 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:19.202341    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:19.202435 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:19.202415    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a2f00df-e26f-4f86-89ec-eefef7fa3ed0-metrics-tls podName:6a2f00df-e26f-4f86-89ec-eefef7fa3ed0 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:21.20239401 +0000 UTC m=+38.585283371 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a2f00df-e26f-4f86-89ec-eefef7fa3ed0-metrics-tls") pod "dns-default-sxvsf" (UID: "6a2f00df-e26f-4f86-89ec-eefef7fa3ed0") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:19.345791 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:19.345756    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-btfcs" event={"ID":"8049c55c-c4ac-44a1-a1bf-5872846445c4","Type":"ContainerStarted","Data":"887552d60056a7527356f683d7fd20d54b25392bf4b17334e2e16326a62350b9"}
Sep 16 18:41:19.376806 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:19.376755    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-btfcs" podStartSLOduration=7.252196238 podStartE2EDuration="36.3767392s" totalImagesPullingTime="29.124542962s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:19.376325149 +0000 UTC m=+36.759214514" watchObservedRunningTime="2026-09-16 18:41:19.3767392 +0000 UTC m=+36.759628559"
Sep 16 18:41:21.218547 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:21.218362    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a2f00df-e26f-4f86-89ec-eefef7fa3ed0-metrics-tls\") pod \"dns-default-sxvsf\" (UID: \"6a2f00df-e26f-4f86-89ec-eefef7fa3ed0\") " pod="openshift-dns/dns-default-sxvsf"
Sep 16 18:41:21.218898 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:21.218506    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:21.218898 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:21.218627    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a2f00df-e26f-4f86-89ec-eefef7fa3ed0-metrics-tls podName:6a2f00df-e26f-4f86-89ec-eefef7fa3ed0 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:25.218610807 +0000 UTC m=+42.601500150 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a2f00df-e26f-4f86-89ec-eefef7fa3ed0-metrics-tls") pod "dns-default-sxvsf" (UID: "6a2f00df-e26f-4f86-89ec-eefef7fa3ed0") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:21.351963 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:21.351929    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ghpxc" event={"ID":"f5bcc8e6-e1b2-4b7d-840f-f4ca1b31b520","Type":"ContainerStarted","Data":"32d405bf6465a7690f52c5284c45246f287486320c8f5bf7dad87036fa90aad2"}
Sep 16 18:41:21.375866 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:21.375814    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ghpxc" podStartSLOduration=1.769098912 podStartE2EDuration="4.375796349s" totalImagesPullingTime="2.606697437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:21.375444749 +0000 UTC m=+38.758334117" watchObservedRunningTime="2026-09-16 18:41:21.375796349 +0000 UTC m=+38.758685715"
Sep 16 18:41:25.246304 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:25.246269    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a2f00df-e26f-4f86-89ec-eefef7fa3ed0-metrics-tls\") pod \"dns-default-sxvsf\" (UID: \"6a2f00df-e26f-4f86-89ec-eefef7fa3ed0\") " pod="openshift-dns/dns-default-sxvsf"
Sep 16 18:41:25.246715 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:25.246387    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:25.246715 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:25.246437    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a2f00df-e26f-4f86-89ec-eefef7fa3ed0-metrics-tls podName:6a2f00df-e26f-4f86-89ec-eefef7fa3ed0 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:33.246423428 +0000 UTC m=+50.629312770 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a2f00df-e26f-4f86-89ec-eefef7fa3ed0-metrics-tls") pod "dns-default-sxvsf" (UID: "6a2f00df-e26f-4f86-89ec-eefef7fa3ed0") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:29.831647 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:29.831615    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2z2cr"]
Sep 16 18:41:29.861608 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:29.861587    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2z2cr"]
Sep 16 18:41:29.861739 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:29.861670    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2z2cr"
Sep 16 18:41:29.865028 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:29.865001    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 16 18:41:29.865136 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:29.865001    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:41:29.866393 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:29.866378    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-jdtnh\""
Sep 16 18:41:29.955978 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:29.955945    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5bb5799fbb-klzv5"]
Sep 16 18:41:29.976593 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:29.976570    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6bkn\" (UniqueName: \"kubernetes.io/projected/ecc4cc43-8c85-4a54-b0ee-6107f507e1bd-kube-api-access-m6bkn\") pod \"volume-data-source-validator-86695b7cc7-2z2cr\" (UID: \"ecc4cc43-8c85-4a54-b0ee-6107f507e1bd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2z2cr"
Sep 16 18:41:29.980354 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:29.980333    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xvhsz"]
Sep 16 18:41:29.980500 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:29.980482    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5bb5799fbb-klzv5"
Sep 16 18:41:29.983019 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:29.982997    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 16 18:41:29.983275 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:29.983260    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 16 18:41:29.983366 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:29.983348    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-8kzvr\""
Sep 16 18:41:29.983423 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:29.983370    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 18:41:29.983544 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:29.983512    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 16 18:41:29.983759 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:29.983747    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 16 18:41:29.983936 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:29.983922    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 18:41:30.004357 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:30.004336    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xvhsz"]
Sep 16 18:41:30.004424 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:30.004367    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5bb5799fbb-klzv5"]
Sep 16 18:41:30.004457 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:30.004438    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xvhsz"
Sep 16 18:41:30.007372 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:30.007355    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 16 18:41:30.007473 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:30.007410    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:41:30.007473 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:30.007424    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-6mw5c\""
Sep 16 18:41:30.007473 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:30.007416    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 16 18:41:30.076863 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:30.076839    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/75a77f81-d864-4437-b516-a567dfb4a6d8-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xvhsz\" (UID: \"75a77f81-d864-4437-b516-a567dfb4a6d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xvhsz"
Sep 16 18:41:30.076976 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:30.076868    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-default-certificate\") pod \"router-default-5bb5799fbb-klzv5\" (UID: \"cf4b4dde-c989-408c-8a0d-a443fa66e2bf\") " pod="openshift-ingress/router-default-5bb5799fbb-klzv5"
Sep 16 18:41:30.076976 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:30.076885    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfc7g\" (UniqueName: \"kubernetes.io/projected/75a77f81-d864-4437-b516-a567dfb4a6d8-kube-api-access-bfc7g\") pod \"cluster-samples-operator-86499c89c4-xvhsz\" (UID: \"75a77f81-d864-4437-b516-a567dfb4a6d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xvhsz"
Sep 16 18:41:30.076976 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:30.076963    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmbdz\" (UniqueName: \"kubernetes.io/projected/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-kube-api-access-zmbdz\") pod \"router-default-5bb5799fbb-klzv5\" (UID: \"cf4b4dde-c989-408c-8a0d-a443fa66e2bf\") " pod="openshift-ingress/router-default-5bb5799fbb-klzv5"
Sep 16 18:41:30.077091 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:30.076999    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-stats-auth\") pod \"router-default-5bb5799fbb-klzv5\" (UID: \"cf4b4dde-c989-408c-8a0d-a443fa66e2bf\") " pod="openshift-ingress/router-default-5bb5799fbb-klzv5"
Sep 16 18:41:30.077091 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:30.077026    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6bkn\" (UniqueName: \"kubernetes.io/projected/ecc4cc43-8c85-4a54-b0ee-6107f507e1bd-kube-api-access-m6bkn\") pod \"volume-data-source-validator-86695b7cc7-2z2cr\" (UID: \"ecc4cc43-8c85-4a54-b0ee-6107f507e1bd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2z2cr"
Sep 16 18:41:30.077091 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:30.077040    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-service-ca-bundle\") pod \"router-default-5bb5799fbb-klzv5\" (UID: \"cf4b4dde-c989-408c-8a0d-a443fa66e2bf\") " pod="openshift-ingress/router-default-5bb5799fbb-klzv5"
Sep 16 18:41:30.077091 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:30.077054    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-metrics-certs\") pod \"router-default-5bb5799fbb-klzv5\" (UID: \"cf4b4dde-c989-408c-8a0d-a443fa66e2bf\") " pod="openshift-ingress/router-default-5bb5799fbb-klzv5"
Sep 16 18:41:30.100366 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:30.100320    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6bkn\" (UniqueName: \"kubernetes.io/projected/ecc4cc43-8c85-4a54-b0ee-6107f507e1bd-kube-api-access-m6bkn\") pod \"volume-data-source-validator-86695b7cc7-2z2cr\" (UID: \"ecc4cc43-8c85-4a54-b0ee-6107f507e1bd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2z2cr"
Sep 16 18:41:30.169929 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:30.169906    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2z2cr"
Sep 16 18:41:30.178159 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:30.178133    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/75a77f81-d864-4437-b516-a567dfb4a6d8-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xvhsz\" (UID: \"75a77f81-d864-4437-b516-a567dfb4a6d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xvhsz"
Sep 16 18:41:30.178221 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:30.178165    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-default-certificate\") pod \"router-default-5bb5799fbb-klzv5\" (UID: \"cf4b4dde-c989-408c-8a0d-a443fa66e2bf\") " pod="openshift-ingress/router-default-5bb5799fbb-klzv5"
Sep 16 18:41:30.178221 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:30.178182    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfc7g\" (UniqueName: \"kubernetes.io/projected/75a77f81-d864-4437-b516-a567dfb4a6d8-kube-api-access-bfc7g\") pod \"cluster-samples-operator-86499c89c4-xvhsz\" (UID: \"75a77f81-d864-4437-b516-a567dfb4a6d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xvhsz"
Sep 16 18:41:30.178294 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:30.178277    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:41:30.178340 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:30.178330    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75a77f81-d864-4437-b516-a567dfb4a6d8-samples-operator-tls podName:75a77f81-d864-4437-b516-a567dfb4a6d8 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:30.678312033 +0000 UTC m=+48.061201383 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/75a77f81-d864-4437-b516-a567dfb4a6d8-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-xvhsz" (UID: "75a77f81-d864-4437-b516-a567dfb4a6d8") : secret "samples-operator-tls" not found
Sep 16 18:41:30.178421 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:30.178401    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmbdz\" (UniqueName: \"kubernetes.io/projected/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-kube-api-access-zmbdz\") pod \"router-default-5bb5799fbb-klzv5\" (UID: \"cf4b4dde-c989-408c-8a0d-a443fa66e2bf\") " pod="openshift-ingress/router-default-5bb5799fbb-klzv5"
Sep 16 18:41:30.178464 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:30.178432    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-stats-auth\") pod \"router-default-5bb5799fbb-klzv5\" (UID: \"cf4b4dde-c989-408c-8a0d-a443fa66e2bf\") " pod="openshift-ingress/router-default-5bb5799fbb-klzv5"
Sep 16 18:41:30.178464 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:30.178454    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-service-ca-bundle\") pod \"router-default-5bb5799fbb-klzv5\" (UID: \"cf4b4dde-c989-408c-8a0d-a443fa66e2bf\") " pod="openshift-ingress/router-default-5bb5799fbb-klzv5"
Sep 16 18:41:30.178558 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:30.178477    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-metrics-certs\") pod \"router-default-5bb5799fbb-klzv5\" (UID: \"cf4b4dde-c989-408c-8a0d-a443fa66e2bf\") " pod="openshift-ingress/router-default-5bb5799fbb-klzv5"
Sep 16 18:41:30.178607 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:30.178598    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:41:30.178642 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:30.178613    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-service-ca-bundle podName:cf4b4dde-c989-408c-8a0d-a443fa66e2bf nodeName:}" failed. No retries permitted until 2026-09-16 18:41:30.67859038 +0000 UTC m=+48.061479746 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-service-ca-bundle") pod "router-default-5bb5799fbb-klzv5" (UID: "cf4b4dde-c989-408c-8a0d-a443fa66e2bf") : configmap references non-existent config key: service-ca.crt
Sep 16 18:41:30.178687 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:30.178646    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-metrics-certs podName:cf4b4dde-c989-408c-8a0d-a443fa66e2bf nodeName:}" failed. No retries permitted until 2026-09-16 18:41:30.678629505 +0000 UTC m=+48.061518853 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-metrics-certs") pod "router-default-5bb5799fbb-klzv5" (UID: "cf4b4dde-c989-408c-8a0d-a443fa66e2bf") : secret "router-metrics-certs-default" not found
Sep 16 18:41:30.180570 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:30.180552    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-default-certificate\") pod \"router-default-5bb5799fbb-klzv5\" (UID: \"cf4b4dde-c989-408c-8a0d-a443fa66e2bf\") " pod="openshift-ingress/router-default-5bb5799fbb-klzv5"
Sep 16 18:41:30.180820 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:30.180800    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-stats-auth\") pod \"router-default-5bb5799fbb-klzv5\" (UID: \"cf4b4dde-c989-408c-8a0d-a443fa66e2bf\") " pod="openshift-ingress/router-default-5bb5799fbb-klzv5"
Sep 16 18:41:30.206096 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:30.206074    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmbdz\" (UniqueName: \"kubernetes.io/projected/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-kube-api-access-zmbdz\") pod \"router-default-5bb5799fbb-klzv5\" (UID: \"cf4b4dde-c989-408c-8a0d-a443fa66e2bf\") " pod="openshift-ingress/router-default-5bb5799fbb-klzv5"
Sep 16 18:41:30.206236 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:30.206217    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfc7g\" (UniqueName: \"kubernetes.io/projected/75a77f81-d864-4437-b516-a567dfb4a6d8-kube-api-access-bfc7g\") pod \"cluster-samples-operator-86499c89c4-xvhsz\" (UID: \"75a77f81-d864-4437-b516-a567dfb4a6d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xvhsz"
Sep 16 18:41:30.464667 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:30.464638    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2z2cr"]
Sep 16 18:41:30.467477 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:41:30.467453    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecc4cc43_8c85_4a54_b0ee_6107f507e1bd.slice/crio-9574c41e48beca80bc4ffa653db52b25da1a1ae44d6a3056f1eeba94d4163bc1 WatchSource:0}: Error finding container 9574c41e48beca80bc4ffa653db52b25da1a1ae44d6a3056f1eeba94d4163bc1: Status 404 returned error can't find the container with id 9574c41e48beca80bc4ffa653db52b25da1a1ae44d6a3056f1eeba94d4163bc1
Sep 16 18:41:30.682493 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:30.682469    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/75a77f81-d864-4437-b516-a567dfb4a6d8-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xvhsz\" (UID: \"75a77f81-d864-4437-b516-a567dfb4a6d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xvhsz"
Sep 16 18:41:30.682598 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:30.682551    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-service-ca-bundle\") pod \"router-default-5bb5799fbb-klzv5\" (UID: \"cf4b4dde-c989-408c-8a0d-a443fa66e2bf\") " pod="openshift-ingress/router-default-5bb5799fbb-klzv5"
Sep 16 18:41:30.682598 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:30.682569    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-metrics-certs\") pod \"router-default-5bb5799fbb-klzv5\" (UID: \"cf4b4dde-c989-408c-8a0d-a443fa66e2bf\") " pod="openshift-ingress/router-default-5bb5799fbb-klzv5"
Sep 16 18:41:30.682734 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:30.682631    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:41:30.682734 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:30.682665    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:41:30.682734 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:30.682704    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75a77f81-d864-4437-b516-a567dfb4a6d8-samples-operator-tls podName:75a77f81-d864-4437-b516-a567dfb4a6d8 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:31.682674099 +0000 UTC m=+49.065563441 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/75a77f81-d864-4437-b516-a567dfb4a6d8-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-xvhsz" (UID: "75a77f81-d864-4437-b516-a567dfb4a6d8") : secret "samples-operator-tls" not found
Sep 16 18:41:30.682734 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:30.682728    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-service-ca-bundle podName:cf4b4dde-c989-408c-8a0d-a443fa66e2bf nodeName:}" failed. No retries permitted until 2026-09-16 18:41:31.682712366 +0000 UTC m=+49.065601734 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-service-ca-bundle") pod "router-default-5bb5799fbb-klzv5" (UID: "cf4b4dde-c989-408c-8a0d-a443fa66e2bf") : configmap references non-existent config key: service-ca.crt
Sep 16 18:41:30.682866 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:30.682755    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-metrics-certs podName:cf4b4dde-c989-408c-8a0d-a443fa66e2bf nodeName:}" failed. No retries permitted until 2026-09-16 18:41:31.68274532 +0000 UTC m=+49.065634698 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-metrics-certs") pod "router-default-5bb5799fbb-klzv5" (UID: "cf4b4dde-c989-408c-8a0d-a443fa66e2bf") : secret "router-metrics-certs-default" not found
Sep 16 18:41:31.376895 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:31.376864    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2z2cr" event={"ID":"ecc4cc43-8c85-4a54-b0ee-6107f507e1bd","Type":"ContainerStarted","Data":"9574c41e48beca80bc4ffa653db52b25da1a1ae44d6a3056f1eeba94d4163bc1"}
Sep 16 18:41:31.690485 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:31.690452    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-service-ca-bundle\") pod \"router-default-5bb5799fbb-klzv5\" (UID: \"cf4b4dde-c989-408c-8a0d-a443fa66e2bf\") " pod="openshift-ingress/router-default-5bb5799fbb-klzv5"
Sep 16 18:41:31.690485 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:31.690485    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-metrics-certs\") pod \"router-default-5bb5799fbb-klzv5\" (UID: \"cf4b4dde-c989-408c-8a0d-a443fa66e2bf\") " pod="openshift-ingress/router-default-5bb5799fbb-klzv5"
Sep 16 18:41:31.690779 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:31.690521    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/75a77f81-d864-4437-b516-a567dfb4a6d8-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xvhsz\" (UID: \"75a77f81-d864-4437-b516-a567dfb4a6d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xvhsz"
Sep 16 18:41:31.690779 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:31.690639    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:41:31.690779 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:31.690651    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:41:31.690779 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:31.690663    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-service-ca-bundle podName:cf4b4dde-c989-408c-8a0d-a443fa66e2bf nodeName:}" failed. No retries permitted until 2026-09-16 18:41:33.690643344 +0000 UTC m=+51.073532707 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-service-ca-bundle") pod "router-default-5bb5799fbb-klzv5" (UID: "cf4b4dde-c989-408c-8a0d-a443fa66e2bf") : configmap references non-existent config key: service-ca.crt
Sep 16 18:41:31.690779 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:31.690690    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-metrics-certs podName:cf4b4dde-c989-408c-8a0d-a443fa66e2bf nodeName:}" failed. No retries permitted until 2026-09-16 18:41:33.690677124 +0000 UTC m=+51.073566467 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-metrics-certs") pod "router-default-5bb5799fbb-klzv5" (UID: "cf4b4dde-c989-408c-8a0d-a443fa66e2bf") : secret "router-metrics-certs-default" not found
Sep 16 18:41:31.690779 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:31.690706    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75a77f81-d864-4437-b516-a567dfb4a6d8-samples-operator-tls podName:75a77f81-d864-4437-b516-a567dfb4a6d8 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:33.690698095 +0000 UTC m=+51.073587438 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/75a77f81-d864-4437-b516-a567dfb4a6d8-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-xvhsz" (UID: "75a77f81-d864-4437-b516-a567dfb4a6d8") : secret "samples-operator-tls" not found
Sep 16 18:41:32.380550 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:32.380457    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2z2cr" event={"ID":"ecc4cc43-8c85-4a54-b0ee-6107f507e1bd","Type":"ContainerStarted","Data":"ba09bd5676944293b2074ca51c5f1817454148f573bb8a2293618fbca3a0b483"}
Sep 16 18:41:32.408158 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:32.408111    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2z2cr" podStartSLOduration=1.955768928 podStartE2EDuration="3.408097175s" totalImagesPullingTime="1.452328247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:32.407814716 +0000 UTC m=+49.790704081" watchObservedRunningTime="2026-09-16 18:41:32.408097175 +0000 UTC m=+49.790986539"
Sep 16 18:41:33.304731 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:33.304686    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a2f00df-e26f-4f86-89ec-eefef7fa3ed0-metrics-tls\") pod \"dns-default-sxvsf\" (UID: \"6a2f00df-e26f-4f86-89ec-eefef7fa3ed0\") " pod="openshift-dns/dns-default-sxvsf"
Sep 16 18:41:33.304907 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:33.304797    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:33.304907 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:33.304845    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a2f00df-e26f-4f86-89ec-eefef7fa3ed0-metrics-tls podName:6a2f00df-e26f-4f86-89ec-eefef7fa3ed0 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:49.304833505 +0000 UTC m=+66.687722860 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a2f00df-e26f-4f86-89ec-eefef7fa3ed0-metrics-tls") pod "dns-default-sxvsf" (UID: "6a2f00df-e26f-4f86-89ec-eefef7fa3ed0") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:33.707162 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:33.707128    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-service-ca-bundle\") pod \"router-default-5bb5799fbb-klzv5\" (UID: \"cf4b4dde-c989-408c-8a0d-a443fa66e2bf\") " pod="openshift-ingress/router-default-5bb5799fbb-klzv5"
Sep 16 18:41:33.707162 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:33.707167    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-metrics-certs\") pod \"router-default-5bb5799fbb-klzv5\" (UID: \"cf4b4dde-c989-408c-8a0d-a443fa66e2bf\") " pod="openshift-ingress/router-default-5bb5799fbb-klzv5"
Sep 16 18:41:33.707551 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:33.707218    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/75a77f81-d864-4437-b516-a567dfb4a6d8-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xvhsz\" (UID: \"75a77f81-d864-4437-b516-a567dfb4a6d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xvhsz"
Sep 16 18:41:33.707551 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:33.707306    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-service-ca-bundle podName:cf4b4dde-c989-408c-8a0d-a443fa66e2bf nodeName:}" failed. No retries permitted until 2026-09-16 18:41:37.707283993 +0000 UTC m=+55.090173358 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-service-ca-bundle") pod "router-default-5bb5799fbb-klzv5" (UID: "cf4b4dde-c989-408c-8a0d-a443fa66e2bf") : configmap references non-existent config key: service-ca.crt
Sep 16 18:41:33.707551 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:33.707356    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:41:33.707551 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:33.707359    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:41:33.707551 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:33.707404    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75a77f81-d864-4437-b516-a567dfb4a6d8-samples-operator-tls podName:75a77f81-d864-4437-b516-a567dfb4a6d8 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:37.707391546 +0000 UTC m=+55.090280902 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/75a77f81-d864-4437-b516-a567dfb4a6d8-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-xvhsz" (UID: "75a77f81-d864-4437-b516-a567dfb4a6d8") : secret "samples-operator-tls" not found
Sep 16 18:41:33.707551 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:33.707423    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-metrics-certs podName:cf4b4dde-c989-408c-8a0d-a443fa66e2bf nodeName:}" failed. No retries permitted until 2026-09-16 18:41:37.707409846 +0000 UTC m=+55.090299189 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-metrics-certs") pod "router-default-5bb5799fbb-klzv5" (UID: "cf4b4dde-c989-408c-8a0d-a443fa66e2bf") : secret "router-metrics-certs-default" not found
Sep 16 18:41:34.005442 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:34.005372    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vf4vz"]
Sep 16 18:41:34.007213 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:34.007199    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vf4vz"
Sep 16 18:41:34.009950 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:34.009931    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 18:41:34.011042 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:34.011020    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 18:41:34.011042 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:34.011034    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 18:41:34.011241 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:34.011047    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 18:41:34.011241 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:34.011050    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kxlhk\""
Sep 16 18:41:34.022851 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:34.022830    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vf4vz"]
Sep 16 18:41:34.109471 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:34.109449    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce0ea9d9-201e-4df7-aaa4-2549409434a8-data-volume\") pod \"insights-runtime-extractor-vf4vz\" (UID: \"ce0ea9d9-201e-4df7-aaa4-2549409434a8\") " pod="openshift-insights/insights-runtime-extractor-vf4vz"
Sep 16 18:41:34.109598 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:34.109497    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ce0ea9d9-201e-4df7-aaa4-2549409434a8-crio-socket\") pod \"insights-runtime-extractor-vf4vz\" (UID: \"ce0ea9d9-201e-4df7-aaa4-2549409434a8\") " pod="openshift-insights/insights-runtime-extractor-vf4vz"
Sep 16 18:41:34.109598 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:34.109516    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ce0ea9d9-201e-4df7-aaa4-2549409434a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vf4vz\" (UID: \"ce0ea9d9-201e-4df7-aaa4-2549409434a8\") " pod="openshift-insights/insights-runtime-extractor-vf4vz"
Sep 16 18:41:34.109598 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:34.109553    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ce0ea9d9-201e-4df7-aaa4-2549409434a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vf4vz\" (UID: \"ce0ea9d9-201e-4df7-aaa4-2549409434a8\") " pod="openshift-insights/insights-runtime-extractor-vf4vz"
Sep 16 18:41:34.109598 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:34.109571    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f42f8\" (UniqueName: \"kubernetes.io/projected/ce0ea9d9-201e-4df7-aaa4-2549409434a8-kube-api-access-f42f8\") pod \"insights-runtime-extractor-vf4vz\" (UID: \"ce0ea9d9-201e-4df7-aaa4-2549409434a8\") " pod="openshift-insights/insights-runtime-extractor-vf4vz"
Sep 16 18:41:34.210353 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:34.210324    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce0ea9d9-201e-4df7-aaa4-2549409434a8-data-volume\") pod \"insights-runtime-extractor-vf4vz\" (UID: \"ce0ea9d9-201e-4df7-aaa4-2549409434a8\") " pod="openshift-insights/insights-runtime-extractor-vf4vz"
Sep 16 18:41:34.210429 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:34.210384    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ce0ea9d9-201e-4df7-aaa4-2549409434a8-crio-socket\") pod \"insights-runtime-extractor-vf4vz\" (UID: \"ce0ea9d9-201e-4df7-aaa4-2549409434a8\") " pod="openshift-insights/insights-runtime-extractor-vf4vz"
Sep 16 18:41:34.210429 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:34.210414    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ce0ea9d9-201e-4df7-aaa4-2549409434a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vf4vz\" (UID: \"ce0ea9d9-201e-4df7-aaa4-2549409434a8\") " pod="openshift-insights/insights-runtime-extractor-vf4vz"
Sep 16 18:41:34.210506 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:34.210438    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ce0ea9d9-201e-4df7-aaa4-2549409434a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vf4vz\" (UID: \"ce0ea9d9-201e-4df7-aaa4-2549409434a8\") " pod="openshift-insights/insights-runtime-extractor-vf4vz"
Sep 16 18:41:34.210506 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:34.210465    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f42f8\" (UniqueName: \"kubernetes.io/projected/ce0ea9d9-201e-4df7-aaa4-2549409434a8-kube-api-access-f42f8\") pod \"insights-runtime-extractor-vf4vz\" (UID: \"ce0ea9d9-201e-4df7-aaa4-2549409434a8\") " pod="openshift-insights/insights-runtime-extractor-vf4vz"
Sep 16 18:41:34.210625 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:34.210545    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:41:34.210625 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:34.210583    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:34.210625 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:34.210623    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ce0ea9d9-201e-4df7-aaa4-2549409434a8-kube-rbac-proxy-cm podName:ce0ea9d9-201e-4df7-aaa4-2549409434a8 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:34.710603567 +0000 UTC m=+52.093492930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ce0ea9d9-201e-4df7-aaa4-2549409434a8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vf4vz" (UID: "ce0ea9d9-201e-4df7-aaa4-2549409434a8") : configmap "kube-rbac-proxy" not found
Sep 16 18:41:34.210756 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:34.210642    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce0ea9d9-201e-4df7-aaa4-2549409434a8-insights-runtime-extractor-tls podName:ce0ea9d9-201e-4df7-aaa4-2549409434a8 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:34.710633905 +0000 UTC m=+52.093523248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ce0ea9d9-201e-4df7-aaa4-2549409434a8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vf4vz" (UID: "ce0ea9d9-201e-4df7-aaa4-2549409434a8") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:34.210756 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:34.210722    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ce0ea9d9-201e-4df7-aaa4-2549409434a8-crio-socket\") pod \"insights-runtime-extractor-vf4vz\" (UID: \"ce0ea9d9-201e-4df7-aaa4-2549409434a8\") " pod="openshift-insights/insights-runtime-extractor-vf4vz"
Sep 16 18:41:34.210843 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:34.210723    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce0ea9d9-201e-4df7-aaa4-2549409434a8-data-volume\") pod \"insights-runtime-extractor-vf4vz\" (UID: \"ce0ea9d9-201e-4df7-aaa4-2549409434a8\") " pod="openshift-insights/insights-runtime-extractor-vf4vz"
Sep 16 18:41:34.222779 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:34.222754    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f42f8\" (UniqueName: \"kubernetes.io/projected/ce0ea9d9-201e-4df7-aaa4-2549409434a8-kube-api-access-f42f8\") pod \"insights-runtime-extractor-vf4vz\" (UID: \"ce0ea9d9-201e-4df7-aaa4-2549409434a8\") " pod="openshift-insights/insights-runtime-extractor-vf4vz"
Sep 16 18:41:34.635841 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:34.635805    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-rhbfr"]
Sep 16 18:41:34.637778 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:34.637763    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rhbfr"
Sep 16 18:41:34.641323 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:34.641296    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-qtx86\""
Sep 16 18:41:34.641323 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:34.641299    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 16 18:41:34.641487 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:34.641302    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 16 18:41:34.712870 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:34.712838    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-rhbfr"]
Sep 16 18:41:34.714187 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:34.714167    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ce0ea9d9-201e-4df7-aaa4-2549409434a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vf4vz\" (UID: \"ce0ea9d9-201e-4df7-aaa4-2549409434a8\") " pod="openshift-insights/insights-runtime-extractor-vf4vz"
Sep 16 18:41:34.714299 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:34.714193    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ce0ea9d9-201e-4df7-aaa4-2549409434a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vf4vz\" (UID: \"ce0ea9d9-201e-4df7-aaa4-2549409434a8\") " pod="openshift-insights/insights-runtime-extractor-vf4vz"
Sep 16 18:41:34.714363 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:34.714298    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:34.714414 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:34.714362    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce0ea9d9-201e-4df7-aaa4-2549409434a8-insights-runtime-extractor-tls podName:ce0ea9d9-201e-4df7-aaa4-2549409434a8 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:35.714345525 +0000 UTC m=+53.097234869 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ce0ea9d9-201e-4df7-aaa4-2549409434a8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vf4vz" (UID: "ce0ea9d9-201e-4df7-aaa4-2549409434a8") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:34.714414 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:34.714370    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:41:34.714414 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:34.714414    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ce0ea9d9-201e-4df7-aaa4-2549409434a8-kube-rbac-proxy-cm podName:ce0ea9d9-201e-4df7-aaa4-2549409434a8 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:35.714402253 +0000 UTC m=+53.097291611 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ce0ea9d9-201e-4df7-aaa4-2549409434a8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vf4vz" (UID: "ce0ea9d9-201e-4df7-aaa4-2549409434a8") : configmap "kube-rbac-proxy" not found
Sep 16 18:41:34.815077 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:34.815050    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwtvl\" (UniqueName: \"kubernetes.io/projected/0edc11ae-5a70-42d6-81fc-2c7591912cdb-kube-api-access-hwtvl\") pod \"migrator-8b4d697c5-rhbfr\" (UID: \"0edc11ae-5a70-42d6-81fc-2c7591912cdb\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rhbfr"
Sep 16 18:41:34.916087 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:34.916025    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwtvl\" (UniqueName: \"kubernetes.io/projected/0edc11ae-5a70-42d6-81fc-2c7591912cdb-kube-api-access-hwtvl\") pod \"migrator-8b4d697c5-rhbfr\" (UID: \"0edc11ae-5a70-42d6-81fc-2c7591912cdb\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rhbfr"
Sep 16 18:41:34.932132 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:34.932104    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwtvl\" (UniqueName: \"kubernetes.io/projected/0edc11ae-5a70-42d6-81fc-2c7591912cdb-kube-api-access-hwtvl\") pod \"migrator-8b4d697c5-rhbfr\" (UID: \"0edc11ae-5a70-42d6-81fc-2c7591912cdb\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rhbfr"
Sep 16 18:41:34.946114 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:34.946093    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rhbfr"
Sep 16 18:41:35.226126 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:35.226045    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-rhbfr"]
Sep 16 18:41:35.228769 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:41:35.228740    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0edc11ae_5a70_42d6_81fc_2c7591912cdb.slice/crio-2f6aa74a2dbec0fec05954fe01908eaf92b0a757c8db665f7c5a85bedfb9e866 WatchSource:0}: Error finding container 2f6aa74a2dbec0fec05954fe01908eaf92b0a757c8db665f7c5a85bedfb9e866: Status 404 returned error can't find the container with id 2f6aa74a2dbec0fec05954fe01908eaf92b0a757c8db665f7c5a85bedfb9e866
Sep 16 18:41:35.388909 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:35.388882    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rhbfr" event={"ID":"0edc11ae-5a70-42d6-81fc-2c7591912cdb","Type":"ContainerStarted","Data":"2f6aa74a2dbec0fec05954fe01908eaf92b0a757c8db665f7c5a85bedfb9e866"}
Sep 16 18:41:35.722460 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:35.722408    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ce0ea9d9-201e-4df7-aaa4-2549409434a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vf4vz\" (UID: \"ce0ea9d9-201e-4df7-aaa4-2549409434a8\") " pod="openshift-insights/insights-runtime-extractor-vf4vz"
Sep 16 18:41:35.722876 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:35.722470    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ce0ea9d9-201e-4df7-aaa4-2549409434a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vf4vz\" (UID: \"ce0ea9d9-201e-4df7-aaa4-2549409434a8\") " pod="openshift-insights/insights-runtime-extractor-vf4vz"
Sep 16 18:41:35.722876 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:35.722550    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:41:35.722876 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:35.722613    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ce0ea9d9-201e-4df7-aaa4-2549409434a8-kube-rbac-proxy-cm podName:ce0ea9d9-201e-4df7-aaa4-2549409434a8 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:37.722598079 +0000 UTC m=+55.105487423 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ce0ea9d9-201e-4df7-aaa4-2549409434a8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vf4vz" (UID: "ce0ea9d9-201e-4df7-aaa4-2549409434a8") : configmap "kube-rbac-proxy" not found
Sep 16 18:41:35.722876 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:35.722631    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:35.722876 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:35.722685    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce0ea9d9-201e-4df7-aaa4-2549409434a8-insights-runtime-extractor-tls podName:ce0ea9d9-201e-4df7-aaa4-2549409434a8 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:37.722669959 +0000 UTC m=+55.105559302 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ce0ea9d9-201e-4df7-aaa4-2549409434a8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vf4vz" (UID: "ce0ea9d9-201e-4df7-aaa4-2549409434a8") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:35.873310 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:35.873285    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wmfjw_590e4879-7945-4583-b644-0caa6b1c6ebd/dns-node-resolver/0.log"
Sep 16 18:41:36.392971 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:36.392859    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rhbfr" event={"ID":"0edc11ae-5a70-42d6-81fc-2c7591912cdb","Type":"ContainerStarted","Data":"e3ab07dfc9feba5728cbe3997b9505fb6fbcf0fec303bb2b205df12d15f3be26"}
Sep 16 18:41:36.667421 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:36.667396    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8cxsn_f935c176-f27b-4179-bc5e-802ee61499d3/node-ca/0.log"
Sep 16 18:41:37.397355 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:37.397321    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rhbfr" event={"ID":"0edc11ae-5a70-42d6-81fc-2c7591912cdb","Type":"ContainerStarted","Data":"18c6a986677cbcc3dd27f623052c3a6e9a62988420c3a7c8c21d3aac0b741020"}
Sep 16 18:41:37.415973 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:37.415927    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rhbfr" podStartSLOduration=2.333490671 podStartE2EDuration="3.415915504s" totalImagesPullingTime="1.082424833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:37.414939949 +0000 UTC m=+54.797829315" watchObservedRunningTime="2026-09-16 18:41:37.415915504 +0000 UTC m=+54.798804869"
Sep 16 18:41:37.672663 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:37.672598    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ghpxc_f5bcc8e6-e1b2-4b7d-840f-f4ca1b31b520/serve-healthcheck-canary/0.log"
Sep 16 18:41:37.737105 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:37.737069    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-service-ca-bundle\") pod \"router-default-5bb5799fbb-klzv5\" (UID: \"cf4b4dde-c989-408c-8a0d-a443fa66e2bf\") " pod="openshift-ingress/router-default-5bb5799fbb-klzv5"
Sep 16 18:41:37.737105 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:37.737102    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-metrics-certs\") pod \"router-default-5bb5799fbb-klzv5\" (UID: \"cf4b4dde-c989-408c-8a0d-a443fa66e2bf\") " pod="openshift-ingress/router-default-5bb5799fbb-klzv5"
Sep 16 18:41:37.737253 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:37.737132    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ce0ea9d9-201e-4df7-aaa4-2549409434a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vf4vz\" (UID: \"ce0ea9d9-201e-4df7-aaa4-2549409434a8\") " pod="openshift-insights/insights-runtime-extractor-vf4vz"
Sep 16 18:41:37.737253 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:37.737148    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ce0ea9d9-201e-4df7-aaa4-2549409434a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vf4vz\" (UID: \"ce0ea9d9-201e-4df7-aaa4-2549409434a8\") " pod="openshift-insights/insights-runtime-extractor-vf4vz"
Sep 16 18:41:37.737253 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:37.737186    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/75a77f81-d864-4437-b516-a567dfb4a6d8-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xvhsz\" (UID: \"75a77f81-d864-4437-b516-a567dfb4a6d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xvhsz"
Sep 16 18:41:37.737253 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:37.737215    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:41:37.737253 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:37.737224    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:41:37.737417 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:37.737232    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-service-ca-bundle podName:cf4b4dde-c989-408c-8a0d-a443fa66e2bf nodeName:}" failed. No retries permitted until 2026-09-16 18:41:45.737212208 +0000 UTC m=+63.120101566 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-service-ca-bundle") pod "router-default-5bb5799fbb-klzv5" (UID: "cf4b4dde-c989-408c-8a0d-a443fa66e2bf") : configmap references non-existent config key: service-ca.crt
Sep 16 18:41:37.737417 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:37.737282    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:41:37.737417 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:37.737291    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-metrics-certs podName:cf4b4dde-c989-408c-8a0d-a443fa66e2bf nodeName:}" failed. No retries permitted until 2026-09-16 18:41:45.737275623 +0000 UTC m=+63.120164968 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-metrics-certs") pod "router-default-5bb5799fbb-klzv5" (UID: "cf4b4dde-c989-408c-8a0d-a443fa66e2bf") : secret "router-metrics-certs-default" not found
Sep 16 18:41:37.737417 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:37.737308    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:37.737417 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:37.737316    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ce0ea9d9-201e-4df7-aaa4-2549409434a8-kube-rbac-proxy-cm podName:ce0ea9d9-201e-4df7-aaa4-2549409434a8 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:41.737306302 +0000 UTC m=+59.120195649 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ce0ea9d9-201e-4df7-aaa4-2549409434a8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vf4vz" (UID: "ce0ea9d9-201e-4df7-aaa4-2549409434a8") : configmap "kube-rbac-proxy" not found
Sep 16 18:41:37.737417 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:37.737338    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75a77f81-d864-4437-b516-a567dfb4a6d8-samples-operator-tls podName:75a77f81-d864-4437-b516-a567dfb4a6d8 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:45.737329981 +0000 UTC m=+63.120219329 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/75a77f81-d864-4437-b516-a567dfb4a6d8-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-xvhsz" (UID: "75a77f81-d864-4437-b516-a567dfb4a6d8") : secret "samples-operator-tls" not found
Sep 16 18:41:37.737417 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:37.737353    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce0ea9d9-201e-4df7-aaa4-2549409434a8-insights-runtime-extractor-tls podName:ce0ea9d9-201e-4df7-aaa4-2549409434a8 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:41.737346153 +0000 UTC m=+59.120235501 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ce0ea9d9-201e-4df7-aaa4-2549409434a8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vf4vz" (UID: "ce0ea9d9-201e-4df7-aaa4-2549409434a8") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:41.768427 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:41.768394    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ce0ea9d9-201e-4df7-aaa4-2549409434a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vf4vz\" (UID: \"ce0ea9d9-201e-4df7-aaa4-2549409434a8\") " pod="openshift-insights/insights-runtime-extractor-vf4vz"
Sep 16 18:41:41.768427 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:41.768427    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ce0ea9d9-201e-4df7-aaa4-2549409434a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vf4vz\" (UID: \"ce0ea9d9-201e-4df7-aaa4-2549409434a8\") " pod="openshift-insights/insights-runtime-extractor-vf4vz"
Sep 16 18:41:41.768896 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:41.768546    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:41:41.768896 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:41.768563    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:41.768896 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:41.768625    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce0ea9d9-201e-4df7-aaa4-2549409434a8-insights-runtime-extractor-tls podName:ce0ea9d9-201e-4df7-aaa4-2549409434a8 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:49.76860638 +0000 UTC m=+67.151495723 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ce0ea9d9-201e-4df7-aaa4-2549409434a8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vf4vz" (UID: "ce0ea9d9-201e-4df7-aaa4-2549409434a8") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:41.768896 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:41.768649    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ce0ea9d9-201e-4df7-aaa4-2549409434a8-kube-rbac-proxy-cm podName:ce0ea9d9-201e-4df7-aaa4-2549409434a8 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:49.768638871 +0000 UTC m=+67.151528214 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ce0ea9d9-201e-4df7-aaa4-2549409434a8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vf4vz" (UID: "ce0ea9d9-201e-4df7-aaa4-2549409434a8") : configmap "kube-rbac-proxy" not found
Sep 16 18:41:42.328854 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:42.328828    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wcfql"
Sep 16 18:41:45.800996 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:45.800961    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-service-ca-bundle\") pod \"router-default-5bb5799fbb-klzv5\" (UID: \"cf4b4dde-c989-408c-8a0d-a443fa66e2bf\") " pod="openshift-ingress/router-default-5bb5799fbb-klzv5"
Sep 16 18:41:45.800996 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:45.800997    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-metrics-certs\") pod \"router-default-5bb5799fbb-klzv5\" (UID: \"cf4b4dde-c989-408c-8a0d-a443fa66e2bf\") " pod="openshift-ingress/router-default-5bb5799fbb-klzv5"
Sep 16 18:41:45.801453 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:45.801157    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-service-ca-bundle podName:cf4b4dde-c989-408c-8a0d-a443fa66e2bf nodeName:}" failed. No retries permitted until 2026-09-16 18:42:01.801132239 +0000 UTC m=+79.184021622 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-service-ca-bundle") pod "router-default-5bb5799fbb-klzv5" (UID: "cf4b4dde-c989-408c-8a0d-a443fa66e2bf") : configmap references non-existent config key: service-ca.crt
Sep 16 18:41:45.801453 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:45.801188    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/75a77f81-d864-4437-b516-a567dfb4a6d8-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xvhsz\" (UID: \"75a77f81-d864-4437-b516-a567dfb4a6d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xvhsz"
Sep 16 18:41:45.803731 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:45.803699    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-metrics-certs\") pod \"router-default-5bb5799fbb-klzv5\" (UID: \"cf4b4dde-c989-408c-8a0d-a443fa66e2bf\") " pod="openshift-ingress/router-default-5bb5799fbb-klzv5"
Sep 16 18:41:45.803921 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:45.803900    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/75a77f81-d864-4437-b516-a567dfb4a6d8-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xvhsz\" (UID: \"75a77f81-d864-4437-b516-a567dfb4a6d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xvhsz"
Sep 16 18:41:45.914415 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:45.914392    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-6mw5c\""
Sep 16 18:41:45.922563 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:45.922548    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xvhsz"
Sep 16 18:41:46.226145 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:46.226112    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xvhsz"]
Sep 16 18:41:46.420310 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:46.420279    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xvhsz" event={"ID":"75a77f81-d864-4437-b516-a567dfb4a6d8","Type":"ContainerStarted","Data":"5534cbc2c2ca5857a25c9c35b8748394387fa834daa64982bc01d9d4d9a83041"}
Sep 16 18:41:47.814837 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:47.814800    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b544a31-67f6-4a93-beeb-a21fd49b00b1-original-pull-secret\") pod \"global-pull-secret-syncer-88mzt\" (UID: \"2b544a31-67f6-4a93-beeb-a21fd49b00b1\") " pod="kube-system/global-pull-secret-syncer-88mzt"
Sep 16 18:41:47.817819 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:47.817794    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:41:47.827985 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:47.827963    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b544a31-67f6-4a93-beeb-a21fd49b00b1-original-pull-secret\") pod \"global-pull-secret-syncer-88mzt\" (UID: \"2b544a31-67f6-4a93-beeb-a21fd49b00b1\") " pod="kube-system/global-pull-secret-syncer-88mzt"
Sep 16 18:41:47.866166 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:47.866140    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88mzt"
Sep 16 18:41:47.916189 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:47.916154    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba8dcba9-de75-4055-8086-3820f53b55e3-metrics-certs\") pod \"network-metrics-daemon-s229p\" (UID: \"ba8dcba9-de75-4055-8086-3820f53b55e3\") " pod="openshift-multus/network-metrics-daemon-s229p"
Sep 16 18:41:47.916348 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:47.916222    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5ttq\" (UniqueName: \"kubernetes.io/projected/8e33041b-cc50-41a3-bcc4-34942629f4cc-kube-api-access-r5ttq\") pod \"network-check-target-ggvbf\" (UID: \"8e33041b-cc50-41a3-bcc4-34942629f4cc\") " pod="openshift-network-diagnostics/network-check-target-ggvbf"
Sep 16 18:41:47.918796 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:47.918775    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:41:47.918911 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:47.918776    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:41:47.929835 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:47.929813    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba8dcba9-de75-4055-8086-3820f53b55e3-metrics-certs\") pod \"network-metrics-daemon-s229p\" (UID: \"ba8dcba9-de75-4055-8086-3820f53b55e3\") " pod="openshift-multus/network-metrics-daemon-s229p"
Sep 16 18:41:47.930068 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:47.929933    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:41:47.940795 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:47.940768    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5ttq\" (UniqueName: \"kubernetes.io/projected/8e33041b-cc50-41a3-bcc4-34942629f4cc-kube-api-access-r5ttq\") pod \"network-check-target-ggvbf\" (UID: \"8e33041b-cc50-41a3-bcc4-34942629f4cc\") " pod="openshift-network-diagnostics/network-check-target-ggvbf"
Sep 16 18:41:48.161938 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:48.161913    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s9z5g\""
Sep 16 18:41:48.170378 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:48.170346    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s229p"
Sep 16 18:41:48.172799 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:48.172778    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-g58c8\""
Sep 16 18:41:48.181266 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:48.181250    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggvbf"
Sep 16 18:41:48.196471 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:48.196447    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-88mzt"]
Sep 16 18:41:48.199765 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:41:48.199733    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b544a31_67f6_4a93_beeb_a21fd49b00b1.slice/crio-f76432e8a68e0281c9f55c27d0e39b038fecf1261f96ad7d2a7c1b4a048d4f0f WatchSource:0}: Error finding container f76432e8a68e0281c9f55c27d0e39b038fecf1261f96ad7d2a7c1b4a048d4f0f: Status 404 returned error can't find the container with id f76432e8a68e0281c9f55c27d0e39b038fecf1261f96ad7d2a7c1b4a048d4f0f
Sep 16 18:41:48.428319 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:48.428287    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xvhsz" event={"ID":"75a77f81-d864-4437-b516-a567dfb4a6d8","Type":"ContainerStarted","Data":"de8e42da393e0162c273d34782537d1b9bb74f7870dca3bfb443314949cd9355"}
Sep 16 18:41:48.428466 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:48.428349    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xvhsz" event={"ID":"75a77f81-d864-4437-b516-a567dfb4a6d8","Type":"ContainerStarted","Data":"7ef4ca25adbce253c95844a99daa5b3d568db1db97884e1063f9ed758d9ba75a"}
Sep 16 18:41:48.429473 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:48.429446    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-88mzt" event={"ID":"2b544a31-67f6-4a93-beeb-a21fd49b00b1","Type":"ContainerStarted","Data":"f76432e8a68e0281c9f55c27d0e39b038fecf1261f96ad7d2a7c1b4a048d4f0f"}
Sep 16 18:41:48.465607 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:48.465586    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s229p"]
Sep 16 18:41:48.468629 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:48.468579    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xvhsz" podStartSLOduration=17.789794769 podStartE2EDuration="19.468560641s" totalImagesPullingTime="1.678765872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:48.468315689 +0000 UTC m=+65.851205053" watchObservedRunningTime="2026-09-16 18:41:48.468560641 +0000 UTC m=+65.851450009"
Sep 16 18:41:48.468727 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:41:48.468691    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba8dcba9_de75_4055_8086_3820f53b55e3.slice/crio-cc6c80a974f8ead447c883b64fccccd114b8e15ac232e8930ee1e3e3917eb537 WatchSource:0}: Error finding container cc6c80a974f8ead447c883b64fccccd114b8e15ac232e8930ee1e3e3917eb537: Status 404 returned error can't find the container with id cc6c80a974f8ead447c883b64fccccd114b8e15ac232e8930ee1e3e3917eb537
Sep 16 18:41:48.480771 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:48.480751    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ggvbf"]
Sep 16 18:41:48.483314 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:41:48.483295    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e33041b_cc50_41a3_bcc4_34942629f4cc.slice/crio-bb2dfaa0271064432a19167bdbf9cd51eac9e7bd124c10187daedebdae11c5f4 WatchSource:0}: Error finding container bb2dfaa0271064432a19167bdbf9cd51eac9e7bd124c10187daedebdae11c5f4: Status 404 returned error can't find the container with id bb2dfaa0271064432a19167bdbf9cd51eac9e7bd124c10187daedebdae11c5f4
Sep 16 18:41:49.328495 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:49.328458    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a2f00df-e26f-4f86-89ec-eefef7fa3ed0-metrics-tls\") pod \"dns-default-sxvsf\" (UID: \"6a2f00df-e26f-4f86-89ec-eefef7fa3ed0\") " pod="openshift-dns/dns-default-sxvsf"
Sep 16 18:41:49.332153 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:49.332126    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a2f00df-e26f-4f86-89ec-eefef7fa3ed0-metrics-tls\") pod \"dns-default-sxvsf\" (UID: \"6a2f00df-e26f-4f86-89ec-eefef7fa3ed0\") " pod="openshift-dns/dns-default-sxvsf"
Sep 16 18:41:49.437204 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:49.437167    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ggvbf" event={"ID":"8e33041b-cc50-41a3-bcc4-34942629f4cc","Type":"ContainerStarted","Data":"bb2dfaa0271064432a19167bdbf9cd51eac9e7bd124c10187daedebdae11c5f4"}
Sep 16 18:41:49.439304 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:49.439273    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s229p" event={"ID":"ba8dcba9-de75-4055-8086-3820f53b55e3","Type":"ContainerStarted","Data":"cc6c80a974f8ead447c883b64fccccd114b8e15ac232e8930ee1e3e3917eb537"}
Sep 16 18:41:49.617726 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:49.617703    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-67vjb\""
Sep 16 18:41:49.625210 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:49.625025    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sxvsf"
Sep 16 18:41:49.832683 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:49.832648    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ce0ea9d9-201e-4df7-aaa4-2549409434a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vf4vz\" (UID: \"ce0ea9d9-201e-4df7-aaa4-2549409434a8\") " pod="openshift-insights/insights-runtime-extractor-vf4vz"
Sep 16 18:41:49.832842 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:49.832692    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ce0ea9d9-201e-4df7-aaa4-2549409434a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vf4vz\" (UID: \"ce0ea9d9-201e-4df7-aaa4-2549409434a8\") " pod="openshift-insights/insights-runtime-extractor-vf4vz"
Sep 16 18:41:49.832842 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:49.832824    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:49.832956 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:41:49.832868    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce0ea9d9-201e-4df7-aaa4-2549409434a8-insights-runtime-extractor-tls podName:ce0ea9d9-201e-4df7-aaa4-2549409434a8 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:05.832854515 +0000 UTC m=+83.215743857 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ce0ea9d9-201e-4df7-aaa4-2549409434a8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vf4vz" (UID: "ce0ea9d9-201e-4df7-aaa4-2549409434a8") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:49.833912 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:49.833865    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ce0ea9d9-201e-4df7-aaa4-2549409434a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vf4vz\" (UID: \"ce0ea9d9-201e-4df7-aaa4-2549409434a8\") " pod="openshift-insights/insights-runtime-extractor-vf4vz"
Sep 16 18:41:49.924997 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:49.924966    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sxvsf"]
Sep 16 18:41:49.929244 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:41:49.929211    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a2f00df_e26f_4f86_89ec_eefef7fa3ed0.slice/crio-c8b6b9be9606e290ed6f935b1e5f67ea40afb5c2b5461af141c2b0e22dee9455 WatchSource:0}: Error finding container c8b6b9be9606e290ed6f935b1e5f67ea40afb5c2b5461af141c2b0e22dee9455: Status 404 returned error can't find the container with id c8b6b9be9606e290ed6f935b1e5f67ea40afb5c2b5461af141c2b0e22dee9455
Sep 16 18:41:50.445025 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:50.444988    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sxvsf" event={"ID":"6a2f00df-e26f-4f86-89ec-eefef7fa3ed0","Type":"ContainerStarted","Data":"c8b6b9be9606e290ed6f935b1e5f67ea40afb5c2b5461af141c2b0e22dee9455"}
Sep 16 18:41:50.447407 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:50.447369    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s229p" event={"ID":"ba8dcba9-de75-4055-8086-3820f53b55e3","Type":"ContainerStarted","Data":"02b81584b3652618fa31d535a8839704568d3154cf3fccadbf8076ce05738149"}
Sep 16 18:41:50.447521 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:50.447420    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s229p" event={"ID":"ba8dcba9-de75-4055-8086-3820f53b55e3","Type":"ContainerStarted","Data":"784d4fe7d2fdf739833fb504fb413b7da7c393dc06d817b108d498544edd23bf"}
Sep 16 18:41:50.469963 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:50.469913    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-s229p" podStartSLOduration=66.407069189 podStartE2EDuration="1m7.469893972s" totalImagesPullingTime="1.062824783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:50.469342572 +0000 UTC m=+67.852231939" watchObservedRunningTime="2026-09-16 18:41:50.469893972 +0000 UTC m=+67.852783338"
Sep 16 18:41:54.469043 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:54.469005    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-88mzt" event={"ID":"2b544a31-67f6-4a93-beeb-a21fd49b00b1","Type":"ContainerStarted","Data":"c91ef2e8d5874b415c0aa68ece1a1b3bab697d5ef4fd703e3f44ce61f73f0259"}
Sep 16 18:41:54.470907 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:54.470883    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ggvbf" event={"ID":"8e33041b-cc50-41a3-bcc4-34942629f4cc","Type":"ContainerStarted","Data":"63ef4e76c35aabb3a3228afb0bdb2cd5666218ab19085a1aa9418d2f38d82f53"}
Sep 16 18:41:54.471004 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:54.470931    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ggvbf"
Sep 16 18:41:54.473092 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:54.473068    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sxvsf" event={"ID":"6a2f00df-e26f-4f86-89ec-eefef7fa3ed0","Type":"ContainerStarted","Data":"7ef218d9f33fe0a0bb425e248898399d989227e73291b4ce87c6ee96ecb8f726"}
Sep 16 18:41:54.473187 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:54.473104    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sxvsf" event={"ID":"6a2f00df-e26f-4f86-89ec-eefef7fa3ed0","Type":"ContainerStarted","Data":"32085c4b1424d56dc9d49e3fb9926ab5a4ca3b1df549be8f75e7ac4eb91f691c"}
Sep 16 18:41:54.473235 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:54.473203    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-sxvsf"
Sep 16 18:41:54.489204 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:54.489160    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-88mzt" podStartSLOduration=66.212264194 podStartE2EDuration="1m11.489147732s" totalImagesPullingTime="5.276883538s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:54.488549538 +0000 UTC m=+71.871438900" watchObservedRunningTime="2026-09-16 18:41:54.489147732 +0000 UTC m=+71.872037138"
Sep 16 18:41:54.511875 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:54.511834    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-sxvsf" podStartSLOduration=33.970805467 podStartE2EDuration="37.511822773s" totalImagesPullingTime="3.541017306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:54.511589803 +0000 UTC m=+71.894479169" watchObservedRunningTime="2026-09-16 18:41:54.511822773 +0000 UTC m=+71.894712136"
Sep 16 18:41:54.527917 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:41:54.527885    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ggvbf" podStartSLOduration=66.54556207 podStartE2EDuration="1m11.527872449s" totalImagesPullingTime="4.982310379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:54.527707499 +0000 UTC m=+71.910596863" watchObservedRunningTime="2026-09-16 18:41:54.527872449 +0000 UTC m=+71.910761815"
Sep 16 18:42:01.672717 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.672679    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8554b47cf6-vj8t7"]
Sep 16 18:42:01.675863 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.675846    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8554b47cf6-vj8t7"
Sep 16 18:42:01.678582 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.678562    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-fzrdl\""
Sep 16 18:42:01.680408 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.680391    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 18:42:01.680484 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.680414    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 18:42:01.680589 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.680573    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 18:42:01.680688 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.680672    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 16 18:42:01.688563 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.688502    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8554b47cf6-vj8t7"]
Sep 16 18:42:01.710111 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.710087    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/552a0308-3758-4f0b-81cb-092808308ac5-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-8554b47cf6-vj8t7\" (UID: \"552a0308-3758-4f0b-81cb-092808308ac5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8554b47cf6-vj8t7"
Sep 16 18:42:01.710199 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.710136    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvvdx\" (UniqueName: \"kubernetes.io/projected/552a0308-3758-4f0b-81cb-092808308ac5-kube-api-access-bvvdx\") pod \"managed-serviceaccount-addon-agent-8554b47cf6-vj8t7\" (UID: \"552a0308-3758-4f0b-81cb-092808308ac5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8554b47cf6-vj8t7"
Sep 16 18:42:01.790229 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.790206    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-76b8cdc779-5f86t"]
Sep 16 18:42:01.792957 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.792944    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76b8cdc779-5f86t"
Sep 16 18:42:01.796086 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.796065    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 18:42:01.796086 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.796077    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 18:42:01.796307 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.796291    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 18:42:01.796375 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.796329    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xjxjk\""
Sep 16 18:42:01.800555 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.800517    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 18:42:01.811087 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.811071    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ea1706f1-a58e-418b-957b-c7965c54efc9-registry-certificates\") pod \"image-registry-76b8cdc779-5f86t\" (UID: \"ea1706f1-a58e-418b-957b-c7965c54efc9\") " pod="openshift-image-registry/image-registry-76b8cdc779-5f86t"
Sep 16 18:42:01.811148 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.811101    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-service-ca-bundle\") pod \"router-default-5bb5799fbb-klzv5\" (UID: \"cf4b4dde-c989-408c-8a0d-a443fa66e2bf\") " pod="openshift-ingress/router-default-5bb5799fbb-klzv5"
Sep 16 18:42:01.811148 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.811118    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ea1706f1-a58e-418b-957b-c7965c54efc9-ca-trust-extracted\") pod \"image-registry-76b8cdc779-5f86t\" (UID: \"ea1706f1-a58e-418b-957b-c7965c54efc9\") " pod="openshift-image-registry/image-registry-76b8cdc779-5f86t"
Sep 16 18:42:01.811148 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.811134    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4lfz\" (UniqueName: \"kubernetes.io/projected/ea1706f1-a58e-418b-957b-c7965c54efc9-kube-api-access-s4lfz\") pod \"image-registry-76b8cdc779-5f86t\" (UID: \"ea1706f1-a58e-418b-957b-c7965c54efc9\") " pod="openshift-image-registry/image-registry-76b8cdc779-5f86t"
Sep 16 18:42:01.811295 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.811151    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ea1706f1-a58e-418b-957b-c7965c54efc9-image-registry-private-configuration\") pod \"image-registry-76b8cdc779-5f86t\" (UID: \"ea1706f1-a58e-418b-957b-c7965c54efc9\") " pod="openshift-image-registry/image-registry-76b8cdc779-5f86t"
Sep 16 18:42:01.811295 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.811200    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ea1706f1-a58e-418b-957b-c7965c54efc9-bound-sa-token\") pod \"image-registry-76b8cdc779-5f86t\" (UID: \"ea1706f1-a58e-418b-957b-c7965c54efc9\") " pod="openshift-image-registry/image-registry-76b8cdc779-5f86t"
Sep 16 18:42:01.811295 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.811264    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ea1706f1-a58e-418b-957b-c7965c54efc9-registry-tls\") pod \"image-registry-76b8cdc779-5f86t\" (UID: \"ea1706f1-a58e-418b-957b-c7965c54efc9\") " pod="openshift-image-registry/image-registry-76b8cdc779-5f86t"
Sep 16 18:42:01.811441 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.811327    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/552a0308-3758-4f0b-81cb-092808308ac5-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-8554b47cf6-vj8t7\" (UID: \"552a0308-3758-4f0b-81cb-092808308ac5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8554b47cf6-vj8t7"
Sep 16 18:42:01.811441 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.811374    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvvdx\" (UniqueName: \"kubernetes.io/projected/552a0308-3758-4f0b-81cb-092808308ac5-kube-api-access-bvvdx\") pod \"managed-serviceaccount-addon-agent-8554b47cf6-vj8t7\" (UID: \"552a0308-3758-4f0b-81cb-092808308ac5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8554b47cf6-vj8t7"
Sep 16 18:42:01.811441 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.811399    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ea1706f1-a58e-418b-957b-c7965c54efc9-installation-pull-secrets\") pod \"image-registry-76b8cdc779-5f86t\" (UID: \"ea1706f1-a58e-418b-957b-c7965c54efc9\") " pod="openshift-image-registry/image-registry-76b8cdc779-5f86t"
Sep 16 18:42:01.811546 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.811460    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ea1706f1-a58e-418b-957b-c7965c54efc9-trusted-ca\") pod \"image-registry-76b8cdc779-5f86t\" (UID: \"ea1706f1-a58e-418b-957b-c7965c54efc9\") " pod="openshift-image-registry/image-registry-76b8cdc779-5f86t"
Sep 16 18:42:01.811799 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.811775    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf4b4dde-c989-408c-8a0d-a443fa66e2bf-service-ca-bundle\") pod \"router-default-5bb5799fbb-klzv5\" (UID: \"cf4b4dde-c989-408c-8a0d-a443fa66e2bf\") " pod="openshift-ingress/router-default-5bb5799fbb-klzv5"
Sep 16 18:42:01.813931 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.813902    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/552a0308-3758-4f0b-81cb-092808308ac5-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-8554b47cf6-vj8t7\" (UID: \"552a0308-3758-4f0b-81cb-092808308ac5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8554b47cf6-vj8t7"
Sep 16 18:42:01.815599 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.815581    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76b8cdc779-5f86t"]
Sep 16 18:42:01.831182 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.831155    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvvdx\" (UniqueName: \"kubernetes.io/projected/552a0308-3758-4f0b-81cb-092808308ac5-kube-api-access-bvvdx\") pod \"managed-serviceaccount-addon-agent-8554b47cf6-vj8t7\" (UID: \"552a0308-3758-4f0b-81cb-092808308ac5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8554b47cf6-vj8t7"
Sep 16 18:42:01.911943 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.911924    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ea1706f1-a58e-418b-957b-c7965c54efc9-trusted-ca\") pod \"image-registry-76b8cdc779-5f86t\" (UID: \"ea1706f1-a58e-418b-957b-c7965c54efc9\") " pod="openshift-image-registry/image-registry-76b8cdc779-5f86t"
Sep 16 18:42:01.912024 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.911955    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ea1706f1-a58e-418b-957b-c7965c54efc9-registry-certificates\") pod \"image-registry-76b8cdc779-5f86t\" (UID: \"ea1706f1-a58e-418b-957b-c7965c54efc9\") " pod="openshift-image-registry/image-registry-76b8cdc779-5f86t"
Sep 16 18:42:01.912024 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.911976    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ea1706f1-a58e-418b-957b-c7965c54efc9-ca-trust-extracted\") pod \"image-registry-76b8cdc779-5f86t\" (UID: \"ea1706f1-a58e-418b-957b-c7965c54efc9\") " pod="openshift-image-registry/image-registry-76b8cdc779-5f86t"
Sep 16 18:42:01.912024 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.911993    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4lfz\" (UniqueName: \"kubernetes.io/projected/ea1706f1-a58e-418b-957b-c7965c54efc9-kube-api-access-s4lfz\") pod \"image-registry-76b8cdc779-5f86t\" (UID: \"ea1706f1-a58e-418b-957b-c7965c54efc9\") " pod="openshift-image-registry/image-registry-76b8cdc779-5f86t"
Sep 16 18:42:01.912024 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.912009    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ea1706f1-a58e-418b-957b-c7965c54efc9-image-registry-private-configuration\") pod \"image-registry-76b8cdc779-5f86t\" (UID: \"ea1706f1-a58e-418b-957b-c7965c54efc9\") " pod="openshift-image-registry/image-registry-76b8cdc779-5f86t"
Sep 16 18:42:01.912211 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.912042    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ea1706f1-a58e-418b-957b-c7965c54efc9-bound-sa-token\") pod \"image-registry-76b8cdc779-5f86t\" (UID: \"ea1706f1-a58e-418b-957b-c7965c54efc9\") " pod="openshift-image-registry/image-registry-76b8cdc779-5f86t"
Sep 16 18:42:01.912211 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.912066    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ea1706f1-a58e-418b-957b-c7965c54efc9-registry-tls\") pod \"image-registry-76b8cdc779-5f86t\" (UID: \"ea1706f1-a58e-418b-957b-c7965c54efc9\") " pod="openshift-image-registry/image-registry-76b8cdc779-5f86t"
Sep 16 18:42:01.912211 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.912124    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ea1706f1-a58e-418b-957b-c7965c54efc9-installation-pull-secrets\") pod \"image-registry-76b8cdc779-5f86t\" (UID: \"ea1706f1-a58e-418b-957b-c7965c54efc9\") " pod="openshift-image-registry/image-registry-76b8cdc779-5f86t"
Sep 16 18:42:01.912440 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.912417    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ea1706f1-a58e-418b-957b-c7965c54efc9-ca-trust-extracted\") pod \"image-registry-76b8cdc779-5f86t\" (UID: \"ea1706f1-a58e-418b-957b-c7965c54efc9\") " pod="openshift-image-registry/image-registry-76b8cdc779-5f86t"
Sep 16 18:42:01.912836 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.912816    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ea1706f1-a58e-418b-957b-c7965c54efc9-registry-certificates\") pod \"image-registry-76b8cdc779-5f86t\" (UID: \"ea1706f1-a58e-418b-957b-c7965c54efc9\") " pod="openshift-image-registry/image-registry-76b8cdc779-5f86t"
Sep 16 18:42:01.913123 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.913103    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ea1706f1-a58e-418b-957b-c7965c54efc9-trusted-ca\") pod \"image-registry-76b8cdc779-5f86t\" (UID: \"ea1706f1-a58e-418b-957b-c7965c54efc9\") " pod="openshift-image-registry/image-registry-76b8cdc779-5f86t"
Sep 16 18:42:01.914640 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.914615    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ea1706f1-a58e-418b-957b-c7965c54efc9-registry-tls\") pod \"image-registry-76b8cdc779-5f86t\" (UID: \"ea1706f1-a58e-418b-957b-c7965c54efc9\") " pod="openshift-image-registry/image-registry-76b8cdc779-5f86t"
Sep 16 18:42:01.915005 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.914982    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ea1706f1-a58e-418b-957b-c7965c54efc9-installation-pull-secrets\") pod \"image-registry-76b8cdc779-5f86t\" (UID: \"ea1706f1-a58e-418b-957b-c7965c54efc9\") " pod="openshift-image-registry/image-registry-76b8cdc779-5f86t"
Sep 16 18:42:01.915079 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.915049    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ea1706f1-a58e-418b-957b-c7965c54efc9-image-registry-private-configuration\") pod \"image-registry-76b8cdc779-5f86t\" (UID: \"ea1706f1-a58e-418b-957b-c7965c54efc9\") " pod="openshift-image-registry/image-registry-76b8cdc779-5f86t"
Sep 16 18:42:01.927715 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.927673    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4lfz\" (UniqueName: \"kubernetes.io/projected/ea1706f1-a58e-418b-957b-c7965c54efc9-kube-api-access-s4lfz\") pod \"image-registry-76b8cdc779-5f86t\" (UID: \"ea1706f1-a58e-418b-957b-c7965c54efc9\") " pod="openshift-image-registry/image-registry-76b8cdc779-5f86t"
Sep 16 18:42:01.927715 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.927697    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ea1706f1-a58e-418b-957b-c7965c54efc9-bound-sa-token\") pod \"image-registry-76b8cdc779-5f86t\" (UID: \"ea1706f1-a58e-418b-957b-c7965c54efc9\") " pod="openshift-image-registry/image-registry-76b8cdc779-5f86t"
Sep 16 18:42:01.996406 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:01.996384    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8554b47cf6-vj8t7"
Sep 16 18:42:02.092109 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:02.092086    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-8kzvr\""
Sep 16 18:42:02.099275 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:02.099255    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5bb5799fbb-klzv5"
Sep 16 18:42:02.101560 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:02.101522    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76b8cdc779-5f86t"
Sep 16 18:42:02.265568 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:02.265434    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8554b47cf6-vj8t7"]
Sep 16 18:42:02.267889 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:42:02.267866    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod552a0308_3758_4f0b_81cb_092808308ac5.slice/crio-758aeddf03551ea9f71da77e35edae17e6a0bbe36ca0c4f3a390a54d54a2d164 WatchSource:0}: Error finding container 758aeddf03551ea9f71da77e35edae17e6a0bbe36ca0c4f3a390a54d54a2d164: Status 404 returned error can't find the container with id 758aeddf03551ea9f71da77e35edae17e6a0bbe36ca0c4f3a390a54d54a2d164
Sep 16 18:42:02.380381 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:02.380357    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76b8cdc779-5f86t"]
Sep 16 18:42:02.382153 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:02.382131    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5bb5799fbb-klzv5"]
Sep 16 18:42:02.383514 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:42:02.383483    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea1706f1_a58e_418b_957b_c7965c54efc9.slice/crio-fbef23c66ca0252014fdfa30e9b6275e4fb589d0100e7ff64ef89b2ae9f2c11b WatchSource:0}: Error finding container fbef23c66ca0252014fdfa30e9b6275e4fb589d0100e7ff64ef89b2ae9f2c11b: Status 404 returned error can't find the container with id fbef23c66ca0252014fdfa30e9b6275e4fb589d0100e7ff64ef89b2ae9f2c11b
Sep 16 18:42:02.386064 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:42:02.386046    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf4b4dde_c989_408c_8a0d_a443fa66e2bf.slice/crio-4918d188800ad701e106d4e3640572ccff011d8683cf8f43f29844a90f29a5cd WatchSource:0}: Error finding container 4918d188800ad701e106d4e3640572ccff011d8683cf8f43f29844a90f29a5cd: Status 404 returned error can't find the container with id 4918d188800ad701e106d4e3640572ccff011d8683cf8f43f29844a90f29a5cd
Sep 16 18:42:02.503022 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:02.502978    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76b8cdc779-5f86t" event={"ID":"ea1706f1-a58e-418b-957b-c7965c54efc9","Type":"ContainerStarted","Data":"3477cd973008dd0c57110f902823619eb2c82c152c3700d993039db5922681a6"}
Sep 16 18:42:02.503180 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:02.503075    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76b8cdc779-5f86t" event={"ID":"ea1706f1-a58e-418b-957b-c7965c54efc9","Type":"ContainerStarted","Data":"fbef23c66ca0252014fdfa30e9b6275e4fb589d0100e7ff64ef89b2ae9f2c11b"}
Sep 16 18:42:02.503180 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:02.503105    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-76b8cdc779-5f86t"
Sep 16 18:42:02.504912 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:02.504883    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5bb5799fbb-klzv5" event={"ID":"cf4b4dde-c989-408c-8a0d-a443fa66e2bf","Type":"ContainerStarted","Data":"8aa8f247f690594cc031deaa5497f29dfda7dfe1e0b4360be0497a061d382d92"}
Sep 16 18:42:02.505030 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:02.504929    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5bb5799fbb-klzv5" event={"ID":"cf4b4dde-c989-408c-8a0d-a443fa66e2bf","Type":"ContainerStarted","Data":"4918d188800ad701e106d4e3640572ccff011d8683cf8f43f29844a90f29a5cd"}
Sep 16 18:42:02.506058 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:02.506022    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8554b47cf6-vj8t7" event={"ID":"552a0308-3758-4f0b-81cb-092808308ac5","Type":"ContainerStarted","Data":"758aeddf03551ea9f71da77e35edae17e6a0bbe36ca0c4f3a390a54d54a2d164"}
Sep 16 18:42:02.526965 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:02.526879    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-76b8cdc779-5f86t" podStartSLOduration=1.526862791 podStartE2EDuration="1.526862791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:02.526428504 +0000 UTC m=+79.909317869" watchObservedRunningTime="2026-09-16 18:42:02.526862791 +0000 UTC m=+79.909752205"
Sep 16 18:42:02.550621 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:02.550585    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5bb5799fbb-klzv5" podStartSLOduration=33.550573385 podStartE2EDuration="33.550573385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:02.550475204 +0000 UTC m=+79.933364570" watchObservedRunningTime="2026-09-16 18:42:02.550573385 +0000 UTC m=+79.933462749"
Sep 16 18:42:03.099852 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:03.099814    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5bb5799fbb-klzv5"
Sep 16 18:42:03.102573 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:03.102551    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5bb5799fbb-klzv5"
Sep 16 18:42:03.509436 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:03.509400    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5bb5799fbb-klzv5"
Sep 16 18:42:03.510779 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:03.510759    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5bb5799fbb-klzv5"
Sep 16 18:42:04.479888 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:04.479862    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sxvsf"
Sep 16 18:42:05.519008 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:05.518966    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8554b47cf6-vj8t7" event={"ID":"552a0308-3758-4f0b-81cb-092808308ac5","Type":"ContainerStarted","Data":"231b90d85069c8cc47fd65dbd36dd43e7d13dddfe939fe92d9bc3e6d803883f2"}
Sep 16 18:42:05.540678 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:05.540618    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8554b47cf6-vj8t7" podStartSLOduration=2.316060614 podStartE2EDuration="4.540597835s" totalImagesPullingTime="2.224537221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:05.539448607 +0000 UTC m=+82.922337971" watchObservedRunningTime="2026-09-16 18:42:05.540597835 +0000 UTC m=+82.923487201"
Sep 16 18:42:05.848226 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:05.848151    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ce0ea9d9-201e-4df7-aaa4-2549409434a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vf4vz\" (UID: \"ce0ea9d9-201e-4df7-aaa4-2549409434a8\") " pod="openshift-insights/insights-runtime-extractor-vf4vz"
Sep 16 18:42:05.850625 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:05.850604    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ce0ea9d9-201e-4df7-aaa4-2549409434a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vf4vz\" (UID: \"ce0ea9d9-201e-4df7-aaa4-2549409434a8\") " pod="openshift-insights/insights-runtime-extractor-vf4vz"
Sep 16 18:42:06.118451 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:06.118389    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kxlhk\""
Sep 16 18:42:06.125724 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:06.125707    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vf4vz"
Sep 16 18:42:06.405566 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:06.405498    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vf4vz"]
Sep 16 18:42:06.408136 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:42:06.408104    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce0ea9d9_201e_4df7_aaa4_2549409434a8.slice/crio-4ba165b9fbcd60b57220d7ae304954bfc983dade63087c4cf12ce25d5086c96d WatchSource:0}: Error finding container 4ba165b9fbcd60b57220d7ae304954bfc983dade63087c4cf12ce25d5086c96d: Status 404 returned error can't find the container with id 4ba165b9fbcd60b57220d7ae304954bfc983dade63087c4cf12ce25d5086c96d
Sep 16 18:42:06.523132 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:06.523098    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vf4vz" event={"ID":"ce0ea9d9-201e-4df7-aaa4-2549409434a8","Type":"ContainerStarted","Data":"7de99bc26d21c1d3db8043709a9f82699b4fe027842018f9f1241907fa5ab845"}
Sep 16 18:42:06.523422 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:06.523163    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vf4vz" event={"ID":"ce0ea9d9-201e-4df7-aaa4-2549409434a8","Type":"ContainerStarted","Data":"4ba165b9fbcd60b57220d7ae304954bfc983dade63087c4cf12ce25d5086c96d"}
Sep 16 18:42:07.528827 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:07.528754    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vf4vz" event={"ID":"ce0ea9d9-201e-4df7-aaa4-2549409434a8","Type":"ContainerStarted","Data":"fd82b1d75989a87ececce3f16232c2a9fc54b35f6873e5492853b80fde2b8111"}
Sep 16 18:42:09.538139 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:09.538105    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vf4vz" event={"ID":"ce0ea9d9-201e-4df7-aaa4-2549409434a8","Type":"ContainerStarted","Data":"849ca9bd1020ea59fa34be55ca9250cf314bda64c46a103f5192be56374f8618"}
Sep 16 18:42:09.567135 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:09.567085    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vf4vz" podStartSLOduration=34.411216984 podStartE2EDuration="36.567070139s" totalImagesPullingTime="2.155853155s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:41:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:09.56551997 +0000 UTC m=+86.948409340" watchObservedRunningTime="2026-09-16 18:42:09.567070139 +0000 UTC m=+86.949959504"
Sep 16 18:42:14.915320 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:14.915283    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-tf57h"]
Sep 16 18:42:14.920603 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:14.920586    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-tf57h"
Sep 16 18:42:14.923936 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:14.923912    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 16 18:42:14.925924 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:14.925897    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 16 18:42:14.925924 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:14.925911    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 18:42:14.926089 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:14.925934    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-bqn6t\""
Sep 16 18:42:14.926089 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:14.925963    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 18:42:14.926089 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:14.925963    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 18:42:14.926283 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:14.926268    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 16 18:42:14.933684 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:14.933665    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-tf57h"]
Sep 16 18:42:14.956295 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:14.956271    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tqgtg"]
Sep 16 18:42:14.961101 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:14.961082    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tqgtg"
Sep 16 18:42:14.963560 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:14.963521    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-t5v59\""
Sep 16 18:42:14.963646 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:14.963623    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 18:42:14.964035 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:14.964019    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 18:42:14.964400 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:14.964386    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 18:42:15.007625 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.007594    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp6sc\" (UniqueName: \"kubernetes.io/projected/1ec56458-30a8-48bb-bb63-4ac4d6262080-kube-api-access-tp6sc\") pod \"kube-state-metrics-59fbdd77d-tf57h\" (UID: \"1ec56458-30a8-48bb-bb63-4ac4d6262080\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-tf57h"
Sep 16 18:42:15.007625 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.007625    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3e478983-4465-4a94-b2ed-fd0032d4dc38-node-exporter-textfile\") pod \"node-exporter-tqgtg\" (UID: \"3e478983-4465-4a94-b2ed-fd0032d4dc38\") " pod="openshift-monitoring/node-exporter-tqgtg"
Sep 16 18:42:15.007822 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.007648    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ec56458-30a8-48bb-bb63-4ac4d6262080-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-tf57h\" (UID: \"1ec56458-30a8-48bb-bb63-4ac4d6262080\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-tf57h"
Sep 16 18:42:15.007822 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.007709    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e478983-4465-4a94-b2ed-fd0032d4dc38-metrics-client-ca\") pod \"node-exporter-tqgtg\" (UID: \"3e478983-4465-4a94-b2ed-fd0032d4dc38\") " pod="openshift-monitoring/node-exporter-tqgtg"
Sep 16 18:42:15.007822 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.007737    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3e478983-4465-4a94-b2ed-fd0032d4dc38-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tqgtg\" (UID: \"3e478983-4465-4a94-b2ed-fd0032d4dc38\") " pod="openshift-monitoring/node-exporter-tqgtg"
Sep 16 18:42:15.007822 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.007780    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ec56458-30a8-48bb-bb63-4ac4d6262080-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-tf57h\" (UID: \"1ec56458-30a8-48bb-bb63-4ac4d6262080\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-tf57h"
Sep 16 18:42:15.007822 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.007799    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3e478983-4465-4a94-b2ed-fd0032d4dc38-node-exporter-accelerators-collector-config\") pod \"node-exporter-tqgtg\" (UID: \"3e478983-4465-4a94-b2ed-fd0032d4dc38\") " pod="openshift-monitoring/node-exporter-tqgtg"
Sep 16 18:42:15.008056 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.007826    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9txd8\" (UniqueName: \"kubernetes.io/projected/3e478983-4465-4a94-b2ed-fd0032d4dc38-kube-api-access-9txd8\") pod \"node-exporter-tqgtg\" (UID: \"3e478983-4465-4a94-b2ed-fd0032d4dc38\") " pod="openshift-monitoring/node-exporter-tqgtg"
Sep 16 18:42:15.008056 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.007872    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3e478983-4465-4a94-b2ed-fd0032d4dc38-node-exporter-tls\") pod \"node-exporter-tqgtg\" (UID: \"3e478983-4465-4a94-b2ed-fd0032d4dc38\") " pod="openshift-monitoring/node-exporter-tqgtg"
Sep 16 18:42:15.008056 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.007902    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1ec56458-30a8-48bb-bb63-4ac4d6262080-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-tf57h\" (UID: \"1ec56458-30a8-48bb-bb63-4ac4d6262080\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-tf57h"
Sep 16 18:42:15.008056 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.007947    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1ec56458-30a8-48bb-bb63-4ac4d6262080-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-tf57h\" (UID: \"1ec56458-30a8-48bb-bb63-4ac4d6262080\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-tf57h"
Sep 16 18:42:15.008056 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.007967    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3e478983-4465-4a94-b2ed-fd0032d4dc38-root\") pod \"node-exporter-tqgtg\" (UID: \"3e478983-4465-4a94-b2ed-fd0032d4dc38\") " pod="openshift-monitoring/node-exporter-tqgtg"
Sep 16 18:42:15.008056 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.008005    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ec56458-30a8-48bb-bb63-4ac4d6262080-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-tf57h\" (UID: \"1ec56458-30a8-48bb-bb63-4ac4d6262080\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-tf57h"
Sep 16 18:42:15.008056 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.008024    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3e478983-4465-4a94-b2ed-fd0032d4dc38-node-exporter-wtmp\") pod \"node-exporter-tqgtg\" (UID: \"3e478983-4465-4a94-b2ed-fd0032d4dc38\") " pod="openshift-monitoring/node-exporter-tqgtg"
Sep 16 18:42:15.008056 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.008040    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e478983-4465-4a94-b2ed-fd0032d4dc38-sys\") pod \"node-exporter-tqgtg\" (UID: \"3e478983-4465-4a94-b2ed-fd0032d4dc38\") " pod="openshift-monitoring/node-exporter-tqgtg"
Sep 16 18:42:15.108658 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.108627    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3e478983-4465-4a94-b2ed-fd0032d4dc38-node-exporter-tls\") pod \"node-exporter-tqgtg\" (UID: \"3e478983-4465-4a94-b2ed-fd0032d4dc38\") " pod="openshift-monitoring/node-exporter-tqgtg"
Sep 16 18:42:15.108833 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.108666    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1ec56458-30a8-48bb-bb63-4ac4d6262080-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-tf57h\" (UID: \"1ec56458-30a8-48bb-bb63-4ac4d6262080\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-tf57h"
Sep 16 18:42:15.108833 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.108689    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1ec56458-30a8-48bb-bb63-4ac4d6262080-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-tf57h\" (UID: \"1ec56458-30a8-48bb-bb63-4ac4d6262080\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-tf57h"
Sep 16 18:42:15.108833 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.108707    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3e478983-4465-4a94-b2ed-fd0032d4dc38-root\") pod \"node-exporter-tqgtg\" (UID: \"3e478983-4465-4a94-b2ed-fd0032d4dc38\") " pod="openshift-monitoring/node-exporter-tqgtg"
Sep 16 18:42:15.108833 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.108739    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ec56458-30a8-48bb-bb63-4ac4d6262080-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-tf57h\" (UID: \"1ec56458-30a8-48bb-bb63-4ac4d6262080\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-tf57h"
Sep 16 18:42:15.109065 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.108827    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3e478983-4465-4a94-b2ed-fd0032d4dc38-root\") pod \"node-exporter-tqgtg\" (UID: \"3e478983-4465-4a94-b2ed-fd0032d4dc38\") " pod="openshift-monitoring/node-exporter-tqgtg"
Sep 16 18:42:15.109065 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.108843    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3e478983-4465-4a94-b2ed-fd0032d4dc38-node-exporter-wtmp\") pod \"node-exporter-tqgtg\" (UID: \"3e478983-4465-4a94-b2ed-fd0032d4dc38\") " pod="openshift-monitoring/node-exporter-tqgtg"
Sep 16 18:42:15.109065 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.108881    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e478983-4465-4a94-b2ed-fd0032d4dc38-sys\") pod \"node-exporter-tqgtg\" (UID: \"3e478983-4465-4a94-b2ed-fd0032d4dc38\") " pod="openshift-monitoring/node-exporter-tqgtg"
Sep 16 18:42:15.109065 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.108927    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp6sc\" (UniqueName: \"kubernetes.io/projected/1ec56458-30a8-48bb-bb63-4ac4d6262080-kube-api-access-tp6sc\") pod \"kube-state-metrics-59fbdd77d-tf57h\" (UID: \"1ec56458-30a8-48bb-bb63-4ac4d6262080\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-tf57h"
Sep 16 18:42:15.109065 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.108955    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3e478983-4465-4a94-b2ed-fd0032d4dc38-node-exporter-textfile\") pod \"node-exporter-tqgtg\" (UID: \"3e478983-4465-4a94-b2ed-fd0032d4dc38\") " pod="openshift-monitoring/node-exporter-tqgtg"
Sep 16 18:42:15.109065 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.108969    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3e478983-4465-4a94-b2ed-fd0032d4dc38-node-exporter-wtmp\") pod \"node-exporter-tqgtg\" (UID: \"3e478983-4465-4a94-b2ed-fd0032d4dc38\") " pod="openshift-monitoring/node-exporter-tqgtg"
Sep 16 18:42:15.109065 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.108987    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ec56458-30a8-48bb-bb63-4ac4d6262080-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-tf57h\" (UID: \"1ec56458-30a8-48bb-bb63-4ac4d6262080\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-tf57h"
Sep 16 18:42:15.109065 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.109028    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e478983-4465-4a94-b2ed-fd0032d4dc38-metrics-client-ca\") pod \"node-exporter-tqgtg\" (UID: \"3e478983-4465-4a94-b2ed-fd0032d4dc38\") " pod="openshift-monitoring/node-exporter-tqgtg"
Sep 16 18:42:15.109065 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.109067    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3e478983-4465-4a94-b2ed-fd0032d4dc38-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tqgtg\" (UID: \"3e478983-4465-4a94-b2ed-fd0032d4dc38\") " pod="openshift-monitoring/node-exporter-tqgtg"
Sep 16 18:42:15.109518 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.109097    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ec56458-30a8-48bb-bb63-4ac4d6262080-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-tf57h\" (UID: \"1ec56458-30a8-48bb-bb63-4ac4d6262080\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-tf57h"
Sep 16 18:42:15.109518 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.109111    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1ec56458-30a8-48bb-bb63-4ac4d6262080-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-tf57h\" (UID: \"1ec56458-30a8-48bb-bb63-4ac4d6262080\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-tf57h"
Sep 16 18:42:15.109518 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.109146    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3e478983-4465-4a94-b2ed-fd0032d4dc38-node-exporter-accelerators-collector-config\") pod \"node-exporter-tqgtg\" (UID: \"3e478983-4465-4a94-b2ed-fd0032d4dc38\") " pod="openshift-monitoring/node-exporter-tqgtg"
Sep 16 18:42:15.109518 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.109168    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9txd8\" (UniqueName: \"kubernetes.io/projected/3e478983-4465-4a94-b2ed-fd0032d4dc38-kube-api-access-9txd8\") pod \"node-exporter-tqgtg\" (UID: \"3e478983-4465-4a94-b2ed-fd0032d4dc38\") " pod="openshift-monitoring/node-exporter-tqgtg"
Sep 16 18:42:15.109518 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.109178    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e478983-4465-4a94-b2ed-fd0032d4dc38-sys\") pod \"node-exporter-tqgtg\" (UID: \"3e478983-4465-4a94-b2ed-fd0032d4dc38\") " pod="openshift-monitoring/node-exporter-tqgtg"
Sep 16 18:42:15.109821 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.109793    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1ec56458-30a8-48bb-bb63-4ac4d6262080-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-tf57h\" (UID: \"1ec56458-30a8-48bb-bb63-4ac4d6262080\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-tf57h"
Sep 16 18:42:15.109971 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.109940    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e478983-4465-4a94-b2ed-fd0032d4dc38-metrics-client-ca\") pod \"node-exporter-tqgtg\" (UID: \"3e478983-4465-4a94-b2ed-fd0032d4dc38\") " pod="openshift-monitoring/node-exporter-tqgtg"
Sep 16 18:42:15.110485 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.110461    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ec56458-30a8-48bb-bb63-4ac4d6262080-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-tf57h\" (UID: \"1ec56458-30a8-48bb-bb63-4ac4d6262080\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-tf57h"
Sep 16 18:42:15.110585 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.110460    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3e478983-4465-4a94-b2ed-fd0032d4dc38-node-exporter-textfile\") pod \"node-exporter-tqgtg\" (UID: \"3e478983-4465-4a94-b2ed-fd0032d4dc38\") " pod="openshift-monitoring/node-exporter-tqgtg"
Sep 16 18:42:15.110721 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.110700    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3e478983-4465-4a94-b2ed-fd0032d4dc38-node-exporter-accelerators-collector-config\") pod \"node-exporter-tqgtg\" (UID: \"3e478983-4465-4a94-b2ed-fd0032d4dc38\") " pod="openshift-monitoring/node-exporter-tqgtg"
Sep 16 18:42:15.111523 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.111505    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3e478983-4465-4a94-b2ed-fd0032d4dc38-node-exporter-tls\") pod \"node-exporter-tqgtg\" (UID: \"3e478983-4465-4a94-b2ed-fd0032d4dc38\") " pod="openshift-monitoring/node-exporter-tqgtg"
Sep 16 18:42:15.111830 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.111805    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ec56458-30a8-48bb-bb63-4ac4d6262080-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-tf57h\" (UID: \"1ec56458-30a8-48bb-bb63-4ac4d6262080\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-tf57h"
Sep 16 18:42:15.111919 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.111904    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ec56458-30a8-48bb-bb63-4ac4d6262080-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-tf57h\" (UID: \"1ec56458-30a8-48bb-bb63-4ac4d6262080\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-tf57h"
Sep 16 18:42:15.112394 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.112377    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3e478983-4465-4a94-b2ed-fd0032d4dc38-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tqgtg\" (UID: \"3e478983-4465-4a94-b2ed-fd0032d4dc38\") " pod="openshift-monitoring/node-exporter-tqgtg"
Sep 16 18:42:15.123545 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.123506    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp6sc\" (UniqueName: \"kubernetes.io/projected/1ec56458-30a8-48bb-bb63-4ac4d6262080-kube-api-access-tp6sc\") pod \"kube-state-metrics-59fbdd77d-tf57h\" (UID: \"1ec56458-30a8-48bb-bb63-4ac4d6262080\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-tf57h"
Sep 16 18:42:15.123958 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.123941    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9txd8\" (UniqueName: \"kubernetes.io/projected/3e478983-4465-4a94-b2ed-fd0032d4dc38-kube-api-access-9txd8\") pod \"node-exporter-tqgtg\" (UID: \"3e478983-4465-4a94-b2ed-fd0032d4dc38\") " pod="openshift-monitoring/node-exporter-tqgtg"
Sep 16 18:42:15.230328 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.230265    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-tf57h"
Sep 16 18:42:15.270351 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.270317    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tqgtg"
Sep 16 18:42:15.284582 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:42:15.283946    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e478983_4465_4a94_b2ed_fd0032d4dc38.slice/crio-2b2a44aa348ca7edf6bef7e79f96eaab4ec483501b709fcab085cad15bded5c5 WatchSource:0}: Error finding container 2b2a44aa348ca7edf6bef7e79f96eaab4ec483501b709fcab085cad15bded5c5: Status 404 returned error can't find the container with id 2b2a44aa348ca7edf6bef7e79f96eaab4ec483501b709fcab085cad15bded5c5
Sep 16 18:42:15.529276 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.529237    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-tf57h"]
Sep 16 18:42:15.530308 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:42:15.530281    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ec56458_30a8_48bb_bb63_4ac4d6262080.slice/crio-f23df825890f73ed84035ebb288dcabb7b9bfc72ce2672187d12dbf0f1198db6 WatchSource:0}: Error finding container f23df825890f73ed84035ebb288dcabb7b9bfc72ce2672187d12dbf0f1198db6: Status 404 returned error can't find the container with id f23df825890f73ed84035ebb288dcabb7b9bfc72ce2672187d12dbf0f1198db6
Sep 16 18:42:15.559058 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.559034    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tqgtg" event={"ID":"3e478983-4465-4a94-b2ed-fd0032d4dc38","Type":"ContainerStarted","Data":"2b2a44aa348ca7edf6bef7e79f96eaab4ec483501b709fcab085cad15bded5c5"}
Sep 16 18:42:15.560234 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:15.560212    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-tf57h" event={"ID":"1ec56458-30a8-48bb-bb63-4ac4d6262080","Type":"ContainerStarted","Data":"f23df825890f73ed84035ebb288dcabb7b9bfc72ce2672187d12dbf0f1198db6"}
Sep 16 18:42:16.167310 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.167267    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:42:16.171547 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.171512    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:16.177980 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.177957    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 18:42:16.178245 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.178224    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 18:42:16.178472 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.178457    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 18:42:16.178562 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.178473    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 18:42:16.178562 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.178458    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8krsz\""
Sep 16 18:42:16.178756 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.178735    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 18:42:16.178821 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.178763    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 18:42:16.178821 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.178782    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 18:42:16.178972 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.178946    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 18:42:16.191282 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.191258    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 18:42:16.202146 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.202124    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:42:16.217438 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.217413    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c3deca89-6dc0-462a-a538-f3e7446d6a60-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:16.217565 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.217456    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3deca89-6dc0-462a-a538-f3e7446d6a60-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:16.217638 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.217606    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c3deca89-6dc0-462a-a538-f3e7446d6a60-config-volume\") pod \"alertmanager-main-0\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:16.217701 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.217661    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c3deca89-6dc0-462a-a538-f3e7446d6a60-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:16.217701 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.217689    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvkrs\" (UniqueName: \"kubernetes.io/projected/c3deca89-6dc0-462a-a538-f3e7446d6a60-kube-api-access-fvkrs\") pod \"alertmanager-main-0\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:16.217806 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.217726    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3deca89-6dc0-462a-a538-f3e7446d6a60-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:16.217806 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.217758    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c3deca89-6dc0-462a-a538-f3e7446d6a60-config-out\") pod \"alertmanager-main-0\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:16.217806 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.217783    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c3deca89-6dc0-462a-a538-f3e7446d6a60-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:16.217916 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.217814    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c3deca89-6dc0-462a-a538-f3e7446d6a60-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:16.217916 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.217872    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c3deca89-6dc0-462a-a538-f3e7446d6a60-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:16.218008 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.217928    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c3deca89-6dc0-462a-a538-f3e7446d6a60-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:16.218008 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.217957    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c3deca89-6dc0-462a-a538-f3e7446d6a60-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:16.218008 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.217978    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c3deca89-6dc0-462a-a538-f3e7446d6a60-web-config\") pod \"alertmanager-main-0\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:16.318389 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.318358    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c3deca89-6dc0-462a-a538-f3e7446d6a60-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:16.318585 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.318414    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c3deca89-6dc0-462a-a538-f3e7446d6a60-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:16.318585 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.318443    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c3deca89-6dc0-462a-a538-f3e7446d6a60-web-config\") pod \"alertmanager-main-0\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:16.318585 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.318470    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c3deca89-6dc0-462a-a538-f3e7446d6a60-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:16.318585 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.318499    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3deca89-6dc0-462a-a538-f3e7446d6a60-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:16.318585 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.318546    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c3deca89-6dc0-462a-a538-f3e7446d6a60-config-volume\") pod \"alertmanager-main-0\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:16.318585 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.318571    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c3deca89-6dc0-462a-a538-f3e7446d6a60-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:16.318897 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.318597    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvkrs\" (UniqueName: \"kubernetes.io/projected/c3deca89-6dc0-462a-a538-f3e7446d6a60-kube-api-access-fvkrs\") pod \"alertmanager-main-0\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:16.318897 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.318641    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3deca89-6dc0-462a-a538-f3e7446d6a60-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:16.318897 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.318675    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c3deca89-6dc0-462a-a538-f3e7446d6a60-config-out\") pod \"alertmanager-main-0\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:16.318897 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.318699    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c3deca89-6dc0-462a-a538-f3e7446d6a60-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:16.318897 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.318723    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c3deca89-6dc0-462a-a538-f3e7446d6a60-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:16.318897 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.318772    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c3deca89-6dc0-462a-a538-f3e7446d6a60-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:16.319177 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.319149    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c3deca89-6dc0-462a-a538-f3e7446d6a60-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:16.320395 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.320367    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3deca89-6dc0-462a-a538-f3e7446d6a60-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:16.321000 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.320978    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3deca89-6dc0-462a-a538-f3e7446d6a60-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:16.322010 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.321946    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c3deca89-6dc0-462a-a538-f3e7446d6a60-config-out\") pod \"alertmanager-main-0\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:16.322351 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.322329    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c3deca89-6dc0-462a-a538-f3e7446d6a60-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:16.322656 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.322633    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c3deca89-6dc0-462a-a538-f3e7446d6a60-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:16.323432 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.323411    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c3deca89-6dc0-462a-a538-f3e7446d6a60-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:16.323608 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.323585    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c3deca89-6dc0-462a-a538-f3e7446d6a60-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:16.324063 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.324045    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c3deca89-6dc0-462a-a538-f3e7446d6a60-config-volume\") pod \"alertmanager-main-0\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:16.324133 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.324096    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c3deca89-6dc0-462a-a538-f3e7446d6a60-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:16.324749 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.324728    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c3deca89-6dc0-462a-a538-f3e7446d6a60-web-config\") pod \"alertmanager-main-0\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:16.326220 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.326197    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c3deca89-6dc0-462a-a538-f3e7446d6a60-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:16.331208 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.331183    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvkrs\" (UniqueName: \"kubernetes.io/projected/c3deca89-6dc0-462a-a538-f3e7446d6a60-kube-api-access-fvkrs\") pod \"alertmanager-main-0\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:16.482633 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.482602    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:16.569400 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.568481    2577 generic.go:356] "Generic (PLEG): container finished" podID="3e478983-4465-4a94-b2ed-fd0032d4dc38" containerID="f74b5615275b82689c2bf63367468393f46dbe20d58466f392164c4ac3e75e62" exitCode=0
Sep 16 18:42:16.569400 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.568582    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tqgtg" event={"ID":"3e478983-4465-4a94-b2ed-fd0032d4dc38","Type":"ContainerDied","Data":"f74b5615275b82689c2bf63367468393f46dbe20d58466f392164c4ac3e75e62"}
Sep 16 18:42:16.807454 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:16.807386    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:42:16.924039 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:42:16.924008    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3deca89_6dc0_462a_a538_f3e7446d6a60.slice/crio-bb40c2f173e44efde681085770375fe5c9af34a74c8f13a018e21c88f5fb909c WatchSource:0}: Error finding container bb40c2f173e44efde681085770375fe5c9af34a74c8f13a018e21c88f5fb909c: Status 404 returned error can't find the container with id bb40c2f173e44efde681085770375fe5c9af34a74c8f13a018e21c88f5fb909c
Sep 16 18:42:17.576240 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:17.576202    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tqgtg" event={"ID":"3e478983-4465-4a94-b2ed-fd0032d4dc38","Type":"ContainerStarted","Data":"cfbbdbf5098f60d86ca933027deaf64a8ea1886c0026278bd38dbcc2e9cb81fd"}
Sep 16 18:42:17.576699 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:17.576270    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tqgtg" event={"ID":"3e478983-4465-4a94-b2ed-fd0032d4dc38","Type":"ContainerStarted","Data":"8e1700374c29cb267488a72ca137c463f4fb89222313bad92f858ce98d64a74d"}
Sep 16 18:42:17.579037 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:17.579005    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-tf57h" event={"ID":"1ec56458-30a8-48bb-bb63-4ac4d6262080","Type":"ContainerStarted","Data":"1bf54fb8f4bb706eaf5d766df497a3cbeacd29ecf6ad36eaa3e9a55a010ab239"}
Sep 16 18:42:17.579164 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:17.579076    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-tf57h" event={"ID":"1ec56458-30a8-48bb-bb63-4ac4d6262080","Type":"ContainerStarted","Data":"fb10569f73ba903f29273948d2e52900b8aff31fa1c2b98bdb6364efde20a044"}
Sep 16 18:42:17.579164 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:17.579107    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-tf57h" event={"ID":"1ec56458-30a8-48bb-bb63-4ac4d6262080","Type":"ContainerStarted","Data":"95a5b8c53f80c5667c2c8aa0e4ad29fde1dfd7e292b31e944eaf26936f34e7d9"}
Sep 16 18:42:17.580440 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:17.580408    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3deca89-6dc0-462a-a538-f3e7446d6a60","Type":"ContainerStarted","Data":"bb40c2f173e44efde681085770375fe5c9af34a74c8f13a018e21c88f5fb909c"}
Sep 16 18:42:17.604011 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:17.603958    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tqgtg" podStartSLOduration=2.431143797 podStartE2EDuration="3.603940344s" totalImagesPullingTime="1.090838617s" totalInitContainerRuntime="81.95793ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:42:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:17.603424151 +0000 UTC m=+94.986313517" watchObservedRunningTime="2026-09-16 18:42:17.603940344 +0000 UTC m=+94.986829710"
Sep 16 18:42:17.633576 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:17.633513    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-tf57h" podStartSLOduration=2.194428635 podStartE2EDuration="3.633497159s" totalImagesPullingTime="1.439068524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:17.63256771 +0000 UTC m=+95.015457096" watchObservedRunningTime="2026-09-16 18:42:17.633497159 +0000 UTC m=+95.016386524"
Sep 16 18:42:17.932840 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:17.932817    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-779cc55674-bq9jx"]
Sep 16 18:42:17.937552 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:17.937517    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-779cc55674-bq9jx"
Sep 16 18:42:17.940705 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:17.940684    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 16 18:42:17.941573 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:17.941454    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 16 18:42:17.941573 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:17.941479    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 16 18:42:17.941573 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:17.941486    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 16 18:42:17.941776 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:17.941644    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 16 18:42:17.942378 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:17.942355    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-1ns13ibtnofkt\""
Sep 16 18:42:17.942659 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:17.942645    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-zrrz7\""
Sep 16 18:42:17.959000 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:17.958937    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-779cc55674-bq9jx"]
Sep 16 18:42:18.032752 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:18.032723    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-779cc55674-bq9jx\" (UID: \"1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1\") " pod="openshift-monitoring/thanos-querier-779cc55674-bq9jx"
Sep 16 18:42:18.032869 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:18.032761    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-779cc55674-bq9jx\" (UID: \"1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1\") " pod="openshift-monitoring/thanos-querier-779cc55674-bq9jx"
Sep 16 18:42:18.032869 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:18.032792    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1-metrics-client-ca\") pod \"thanos-querier-779cc55674-bq9jx\" (UID: \"1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1\") " pod="openshift-monitoring/thanos-querier-779cc55674-bq9jx"
Sep 16 18:42:18.032869 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:18.032851    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-779cc55674-bq9jx\" (UID: \"1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1\") " pod="openshift-monitoring/thanos-querier-779cc55674-bq9jx"
Sep 16 18:42:18.032978 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:18.032889    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-779cc55674-bq9jx\" (UID: \"1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1\") " pod="openshift-monitoring/thanos-querier-779cc55674-bq9jx"
Sep 16 18:42:18.032978 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:18.032917    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1-secret-grpc-tls\") pod \"thanos-querier-779cc55674-bq9jx\" (UID: \"1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1\") " pod="openshift-monitoring/thanos-querier-779cc55674-bq9jx"
Sep 16 18:42:18.032978 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:18.032936    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1-secret-thanos-querier-tls\") pod \"thanos-querier-779cc55674-bq9jx\" (UID: \"1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1\") " pod="openshift-monitoring/thanos-querier-779cc55674-bq9jx"
Sep 16 18:42:18.033061 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:18.032978    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9wvq\" (UniqueName: \"kubernetes.io/projected/1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1-kube-api-access-f9wvq\") pod \"thanos-querier-779cc55674-bq9jx\" (UID: \"1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1\") " pod="openshift-monitoring/thanos-querier-779cc55674-bq9jx"
Sep 16 18:42:18.133502 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:18.133429    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-779cc55674-bq9jx\" (UID: \"1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1\") " pod="openshift-monitoring/thanos-querier-779cc55674-bq9jx"
Sep 16 18:42:18.133502 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:18.133465    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1-metrics-client-ca\") pod \"thanos-querier-779cc55674-bq9jx\" (UID: \"1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1\") " pod="openshift-monitoring/thanos-querier-779cc55674-bq9jx"
Sep 16 18:42:18.133726 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:18.133516    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-779cc55674-bq9jx\" (UID: \"1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1\") " pod="openshift-monitoring/thanos-querier-779cc55674-bq9jx"
Sep 16 18:42:18.133726 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:18.133560    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-779cc55674-bq9jx\" (UID: \"1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1\") " pod="openshift-monitoring/thanos-querier-779cc55674-bq9jx"
Sep 16 18:42:18.133726 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:18.133598    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1-secret-grpc-tls\") pod \"thanos-querier-779cc55674-bq9jx\" (UID: \"1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1\") " pod="openshift-monitoring/thanos-querier-779cc55674-bq9jx"
Sep 16 18:42:18.133726 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:18.133620    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1-secret-thanos-querier-tls\") pod \"thanos-querier-779cc55674-bq9jx\" (UID: \"1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1\") " pod="openshift-monitoring/thanos-querier-779cc55674-bq9jx"
Sep 16 18:42:18.133726 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:18.133644    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9wvq\" (UniqueName: \"kubernetes.io/projected/1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1-kube-api-access-f9wvq\") pod \"thanos-querier-779cc55674-bq9jx\" (UID: \"1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1\") " pod="openshift-monitoring/thanos-querier-779cc55674-bq9jx"
Sep 16 18:42:18.133930 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:18.133817    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-779cc55674-bq9jx\" (UID: \"1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1\") " pod="openshift-monitoring/thanos-querier-779cc55674-bq9jx"
Sep 16 18:42:18.134346 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:18.134324    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1-metrics-client-ca\") pod \"thanos-querier-779cc55674-bq9jx\" (UID: \"1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1\") " pod="openshift-monitoring/thanos-querier-779cc55674-bq9jx"
Sep 16 18:42:18.136113 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:18.136084    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-779cc55674-bq9jx\" (UID: \"1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1\") " pod="openshift-monitoring/thanos-querier-779cc55674-bq9jx"
Sep 16 18:42:18.136565 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:18.136520    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1-secret-thanos-querier-tls\") pod \"thanos-querier-779cc55674-bq9jx\" (UID: \"1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1\") " pod="openshift-monitoring/thanos-querier-779cc55674-bq9jx"
Sep 16 18:42:18.136682 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:18.136663    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-779cc55674-bq9jx\" (UID: \"1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1\") " pod="openshift-monitoring/thanos-querier-779cc55674-bq9jx"
Sep 16 18:42:18.136913 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:18.136894    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-779cc55674-bq9jx\" (UID: \"1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1\") " pod="openshift-monitoring/thanos-querier-779cc55674-bq9jx"
Sep 16 18:42:18.136955 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:18.136894    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-779cc55674-bq9jx\" (UID: \"1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1\") " pod="openshift-monitoring/thanos-querier-779cc55674-bq9jx"
Sep 16 18:42:18.136955 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:18.136941    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1-secret-grpc-tls\") pod \"thanos-querier-779cc55674-bq9jx\" (UID: \"1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1\") " pod="openshift-monitoring/thanos-querier-779cc55674-bq9jx"
Sep 16 18:42:18.151264 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:18.151245    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9wvq\" (UniqueName: \"kubernetes.io/projected/1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1-kube-api-access-f9wvq\") pod \"thanos-querier-779cc55674-bq9jx\" (UID: \"1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1\") " pod="openshift-monitoring/thanos-querier-779cc55674-bq9jx"
Sep 16 18:42:18.247158 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:18.247132    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-779cc55674-bq9jx"
Sep 16 18:42:18.555369 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:18.555347    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-779cc55674-bq9jx"]
Sep 16 18:42:18.557482 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:42:18.557456    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e0b0b6b_b2c8_4e3a_92a0_0c67d14794f1.slice/crio-c52ad0be8f0043aba31b56f74c33c6c02630e838198686f90776a51e841f6dac WatchSource:0}: Error finding container c52ad0be8f0043aba31b56f74c33c6c02630e838198686f90776a51e841f6dac: Status 404 returned error can't find the container with id c52ad0be8f0043aba31b56f74c33c6c02630e838198686f90776a51e841f6dac
Sep 16 18:42:18.585987 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:18.585959    2577 generic.go:356] "Generic (PLEG): container finished" podID="c3deca89-6dc0-462a-a538-f3e7446d6a60" containerID="427e34f72c9197ac3fa9e9d741e44635304886654567df2f0ae9eee9ace9ecfc" exitCode=0
Sep 16 18:42:18.586315 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:18.586041    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3deca89-6dc0-462a-a538-f3e7446d6a60","Type":"ContainerDied","Data":"427e34f72c9197ac3fa9e9d741e44635304886654567df2f0ae9eee9ace9ecfc"}
Sep 16 18:42:18.587484 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:18.587448    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-779cc55674-bq9jx" event={"ID":"1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1","Type":"ContainerStarted","Data":"c52ad0be8f0043aba31b56f74c33c6c02630e838198686f90776a51e841f6dac"}
Sep 16 18:42:19.552811 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:19.552779    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-l6f5z"]
Sep 16 18:42:19.558068 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:19.558046    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-l6f5z"
Sep 16 18:42:19.560634 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:19.560608    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 16 18:42:19.561397 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:19.561372    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-whdpp\""
Sep 16 18:42:19.571431 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:19.571408    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-l6f5z"]
Sep 16 18:42:19.645836 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:19.645807    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgkz8\" (UniqueName: \"kubernetes.io/projected/c67af528-b4c8-4dc4-97e2-2e91547e6020-kube-api-access-pgkz8\") pod \"monitoring-plugin-5d85778b9d-l6f5z\" (UID: \"c67af528-b4c8-4dc4-97e2-2e91547e6020\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-l6f5z"
Sep 16 18:42:19.646258 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:19.645876    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c67af528-b4c8-4dc4-97e2-2e91547e6020-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-l6f5z\" (UID: \"c67af528-b4c8-4dc4-97e2-2e91547e6020\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-l6f5z"
Sep 16 18:42:19.747447 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:19.747401    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgkz8\" (UniqueName: \"kubernetes.io/projected/c67af528-b4c8-4dc4-97e2-2e91547e6020-kube-api-access-pgkz8\") pod \"monitoring-plugin-5d85778b9d-l6f5z\" (UID: \"c67af528-b4c8-4dc4-97e2-2e91547e6020\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-l6f5z"
Sep 16 18:42:19.747633 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:19.747480    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c67af528-b4c8-4dc4-97e2-2e91547e6020-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-l6f5z\" (UID: \"c67af528-b4c8-4dc4-97e2-2e91547e6020\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-l6f5z"
Sep 16 18:42:19.750813 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:19.750789    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c67af528-b4c8-4dc4-97e2-2e91547e6020-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-l6f5z\" (UID: \"c67af528-b4c8-4dc4-97e2-2e91547e6020\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-l6f5z"
Sep 16 18:42:19.768496 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:19.768470    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgkz8\" (UniqueName: \"kubernetes.io/projected/c67af528-b4c8-4dc4-97e2-2e91547e6020-kube-api-access-pgkz8\") pod \"monitoring-plugin-5d85778b9d-l6f5z\" (UID: \"c67af528-b4c8-4dc4-97e2-2e91547e6020\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-l6f5z"
Sep 16 18:42:19.870481 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:19.870403    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-l6f5z"
Sep 16 18:42:20.213371 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:20.213345    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-l6f5z"]
Sep 16 18:42:20.593841 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:42:20.593776    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc67af528_b4c8_4dc4_97e2_2e91547e6020.slice/crio-12f70d121a5e4ce4dd3b56f5869cdff16c4440583f5d3bb3541665b5e992afca WatchSource:0}: Error finding container 12f70d121a5e4ce4dd3b56f5869cdff16c4440583f5d3bb3541665b5e992afca: Status 404 returned error can't find the container with id 12f70d121a5e4ce4dd3b56f5869cdff16c4440583f5d3bb3541665b5e992afca
Sep 16 18:42:20.600435 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:20.600409    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3deca89-6dc0-462a-a538-f3e7446d6a60","Type":"ContainerStarted","Data":"25c71b6a2f66bfaf8d97dca768c506f0e469d96848ea570363f91ba44082e1fa"}
Sep 16 18:42:21.608809 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:21.608775    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-779cc55674-bq9jx" event={"ID":"1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1","Type":"ContainerStarted","Data":"1f9fa0f3d61f59675cd6e2a5c6aeb0cc25317ca0065c4b9a349c83297ccc814a"}
Sep 16 18:42:21.609143 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:21.608871    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-779cc55674-bq9jx" event={"ID":"1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1","Type":"ContainerStarted","Data":"280a6dd45c5ea2fbbf2022ea620214558b4a6796d71282336f514a4dadfe4231"}
Sep 16 18:42:21.609143 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:21.608918    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-779cc55674-bq9jx" event={"ID":"1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1","Type":"ContainerStarted","Data":"7aabd47926d9a498414eccbaa96885a0bde18076725d6eb8eb3762562fc033bc"}
Sep 16 18:42:21.610263 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:21.610201    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-l6f5z" event={"ID":"c67af528-b4c8-4dc4-97e2-2e91547e6020","Type":"ContainerStarted","Data":"12f70d121a5e4ce4dd3b56f5869cdff16c4440583f5d3bb3541665b5e992afca"}
Sep 16 18:42:21.615044 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:21.615014    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3deca89-6dc0-462a-a538-f3e7446d6a60","Type":"ContainerStarted","Data":"ea02c865fb9fd0e74de5a8b52bb3cecd51f301eb005079f13da1806dc17fcd55"}
Sep 16 18:42:21.615125 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:21.615101    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3deca89-6dc0-462a-a538-f3e7446d6a60","Type":"ContainerStarted","Data":"279a6e0ff7f821e508801c992ab47337373bffecd0d5aa65e4fb92b8d95cc228"}
Sep 16 18:42:21.615185 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:21.615143    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3deca89-6dc0-462a-a538-f3e7446d6a60","Type":"ContainerStarted","Data":"a1b6aa4fb8625aa9db441ea01ecf2e24ed8ee9c8c8f26c20f50de979adb7e99a"}
Sep 16 18:42:21.615239 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:21.615189    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3deca89-6dc0-462a-a538-f3e7446d6a60","Type":"ContainerStarted","Data":"ab4a095c1ec922b35af235bb9ea64a7ef30e0163263e4e1a6bfbcc04a8580142"}
Sep 16 18:42:22.620834 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:22.620794    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-l6f5z" event={"ID":"c67af528-b4c8-4dc4-97e2-2e91547e6020","Type":"ContainerStarted","Data":"df09ae1159f368d794ad0efde49931de38a56d197c1ef74dda14bf24301872ea"}
Sep 16 18:42:22.621252 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:22.620854    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-l6f5z"
Sep 16 18:42:22.624684 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:22.624655    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3deca89-6dc0-462a-a538-f3e7446d6a60","Type":"ContainerStarted","Data":"a06e3cf95b8dfaa0d8de4f89af42800fc8aeca0c97c7900bee63856433a3b214"}
Sep 16 18:42:22.626152 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:22.626131    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-l6f5z"
Sep 16 18:42:22.627690 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:22.627666    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-779cc55674-bq9jx" event={"ID":"1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1","Type":"ContainerStarted","Data":"05cd1303ea205f6b3f8ef45bdcaf20a8faabb8ac3bdefe2ca9ee4731a4979463"}
Sep 16 18:42:22.627781 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:22.627719    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-779cc55674-bq9jx" event={"ID":"1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1","Type":"ContainerStarted","Data":"447ed9bc39975f10eef2c78e3a0644321a32ce0865a078deca56799cd55653e0"}
Sep 16 18:42:22.627781 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:22.627744    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-779cc55674-bq9jx" event={"ID":"1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1","Type":"ContainerStarted","Data":"0f48fff8908e9bb20632a0aecb5e44496387cc1eaddc0bbae02e95e74eea2e23"}
Sep 16 18:42:22.627885 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:22.627786    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-779cc55674-bq9jx"
Sep 16 18:42:22.646322 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:22.646281    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-l6f5z" podStartSLOduration=2.196269011 podStartE2EDuration="3.646269399s" totalImagesPullingTime="1.450000388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:22.646079782 +0000 UTC m=+100.028969146" watchObservedRunningTime="2026-09-16 18:42:22.646269399 +0000 UTC m=+100.029158763"
Sep 16 18:42:22.686734 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:22.686695    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.624547981 podStartE2EDuration="6.686681382s" totalImagesPullingTime="3.044057607s" totalInitContainerRuntime="18.075794ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:22.685580102 +0000 UTC m=+100.068469474" watchObservedRunningTime="2026-09-16 18:42:22.686681382 +0000 UTC m=+100.069570746"
Sep 16 18:42:22.717292 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:22.717245    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-779cc55674-bq9jx" podStartSLOduration=2.894504149 podStartE2EDuration="5.717226274s" totalImagesPullingTime="2.822722125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:22.716486032 +0000 UTC m=+100.099375431" watchObservedRunningTime="2026-09-16 18:42:22.717226274 +0000 UTC m=+100.100115665"
Sep 16 18:42:23.513854 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:23.513825    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-76b8cdc779-5f86t"
Sep 16 18:42:25.479963 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:25.479933    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ggvbf"
Sep 16 18:42:28.638521 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:42:28.638488    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-779cc55674-bq9jx"
Sep 16 18:43:35.922874 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:35.922799    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:43:35.923328 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:35.923069    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c3deca89-6dc0-462a-a538-f3e7446d6a60" containerName="alertmanager" containerID="cri-o://25c71b6a2f66bfaf8d97dca768c506f0e469d96848ea570363f91ba44082e1fa" gracePeriod=120
Sep 16 18:43:35.923328 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:35.923145    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c3deca89-6dc0-462a-a538-f3e7446d6a60" containerName="kube-rbac-proxy-metric" containerID="cri-o://ea02c865fb9fd0e74de5a8b52bb3cecd51f301eb005079f13da1806dc17fcd55" gracePeriod=120
Sep 16 18:43:35.923328 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:35.923172    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c3deca89-6dc0-462a-a538-f3e7446d6a60" containerName="config-reloader" containerID="cri-o://ab4a095c1ec922b35af235bb9ea64a7ef30e0163263e4e1a6bfbcc04a8580142" gracePeriod=120
Sep 16 18:43:35.923328 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:35.923183    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c3deca89-6dc0-462a-a538-f3e7446d6a60" containerName="prom-label-proxy" containerID="cri-o://a06e3cf95b8dfaa0d8de4f89af42800fc8aeca0c97c7900bee63856433a3b214" gracePeriod=120
Sep 16 18:43:35.923328 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:35.923204    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c3deca89-6dc0-462a-a538-f3e7446d6a60" containerName="kube-rbac-proxy" containerID="cri-o://279a6e0ff7f821e508801c992ab47337373bffecd0d5aa65e4fb92b8d95cc228" gracePeriod=120
Sep 16 18:43:35.923328 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:35.923150    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c3deca89-6dc0-462a-a538-f3e7446d6a60" containerName="kube-rbac-proxy-web" containerID="cri-o://a1b6aa4fb8625aa9db441ea01ecf2e24ed8ee9c8c8f26c20f50de979adb7e99a" gracePeriod=120
Sep 16 18:43:36.923696 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:36.923666    2577 generic.go:356] "Generic (PLEG): container finished" podID="c3deca89-6dc0-462a-a538-f3e7446d6a60" containerID="a06e3cf95b8dfaa0d8de4f89af42800fc8aeca0c97c7900bee63856433a3b214" exitCode=0
Sep 16 18:43:36.923696 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:36.923691    2577 generic.go:356] "Generic (PLEG): container finished" podID="c3deca89-6dc0-462a-a538-f3e7446d6a60" containerID="279a6e0ff7f821e508801c992ab47337373bffecd0d5aa65e4fb92b8d95cc228" exitCode=0
Sep 16 18:43:36.923696 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:36.923698    2577 generic.go:356] "Generic (PLEG): container finished" podID="c3deca89-6dc0-462a-a538-f3e7446d6a60" containerID="ab4a095c1ec922b35af235bb9ea64a7ef30e0163263e4e1a6bfbcc04a8580142" exitCode=0
Sep 16 18:43:36.923696 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:36.923703    2577 generic.go:356] "Generic (PLEG): container finished" podID="c3deca89-6dc0-462a-a538-f3e7446d6a60" containerID="25c71b6a2f66bfaf8d97dca768c506f0e469d96848ea570363f91ba44082e1fa" exitCode=0
Sep 16 18:43:36.924175 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:36.923739    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3deca89-6dc0-462a-a538-f3e7446d6a60","Type":"ContainerDied","Data":"a06e3cf95b8dfaa0d8de4f89af42800fc8aeca0c97c7900bee63856433a3b214"}
Sep 16 18:43:36.924175 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:36.923810    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3deca89-6dc0-462a-a538-f3e7446d6a60","Type":"ContainerDied","Data":"279a6e0ff7f821e508801c992ab47337373bffecd0d5aa65e4fb92b8d95cc228"}
Sep 16 18:43:36.924175 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:36.923847    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3deca89-6dc0-462a-a538-f3e7446d6a60","Type":"ContainerDied","Data":"ab4a095c1ec922b35af235bb9ea64a7ef30e0163263e4e1a6bfbcc04a8580142"}
Sep 16 18:43:36.924175 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:36.923873    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3deca89-6dc0-462a-a538-f3e7446d6a60","Type":"ContainerDied","Data":"25c71b6a2f66bfaf8d97dca768c506f0e469d96848ea570363f91ba44082e1fa"}
Sep 16 18:43:37.156570 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.156545    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:43:37.202415 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.202358    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3deca89-6dc0-462a-a538-f3e7446d6a60-metrics-client-ca\") pod \"c3deca89-6dc0-462a-a538-f3e7446d6a60\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") "
Sep 16 18:43:37.202415 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.202401    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c3deca89-6dc0-462a-a538-f3e7446d6a60-tls-assets\") pod \"c3deca89-6dc0-462a-a538-f3e7446d6a60\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") "
Sep 16 18:43:37.202600 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.202430    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c3deca89-6dc0-462a-a538-f3e7446d6a60-config-volume\") pod \"c3deca89-6dc0-462a-a538-f3e7446d6a60\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") "
Sep 16 18:43:37.202600 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.202521    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c3deca89-6dc0-462a-a538-f3e7446d6a60-secret-alertmanager-kube-rbac-proxy\") pod \"c3deca89-6dc0-462a-a538-f3e7446d6a60\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") "
Sep 16 18:43:37.202600 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.202577    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c3deca89-6dc0-462a-a538-f3e7446d6a60-web-config\") pod \"c3deca89-6dc0-462a-a538-f3e7446d6a60\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") "
Sep 16 18:43:37.202755 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.202606    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3deca89-6dc0-462a-a538-f3e7446d6a60-alertmanager-trusted-ca-bundle\") pod \"c3deca89-6dc0-462a-a538-f3e7446d6a60\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") "
Sep 16 18:43:37.202755 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.202629    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c3deca89-6dc0-462a-a538-f3e7446d6a60-secret-alertmanager-kube-rbac-proxy-metric\") pod \"c3deca89-6dc0-462a-a538-f3e7446d6a60\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") "
Sep 16 18:43:37.202755 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.202657    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c3deca89-6dc0-462a-a538-f3e7446d6a60-cluster-tls-config\") pod \"c3deca89-6dc0-462a-a538-f3e7446d6a60\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") "
Sep 16 18:43:37.202755 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.202694    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvkrs\" (UniqueName: \"kubernetes.io/projected/c3deca89-6dc0-462a-a538-f3e7446d6a60-kube-api-access-fvkrs\") pod \"c3deca89-6dc0-462a-a538-f3e7446d6a60\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") "
Sep 16 18:43:37.202755 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.202718    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c3deca89-6dc0-462a-a538-f3e7446d6a60-config-out\") pod \"c3deca89-6dc0-462a-a538-f3e7446d6a60\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") "
Sep 16 18:43:37.202755 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.202730    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3deca89-6dc0-462a-a538-f3e7446d6a60-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "c3deca89-6dc0-462a-a538-f3e7446d6a60" (UID: "c3deca89-6dc0-462a-a538-f3e7446d6a60"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:37.202755 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.202749    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c3deca89-6dc0-462a-a538-f3e7446d6a60-alertmanager-main-db\") pod \"c3deca89-6dc0-462a-a538-f3e7446d6a60\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") "
Sep 16 18:43:37.203095 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.202797    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c3deca89-6dc0-462a-a538-f3e7446d6a60-secret-alertmanager-main-tls\") pod \"c3deca89-6dc0-462a-a538-f3e7446d6a60\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") "
Sep 16 18:43:37.203095 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.202826    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c3deca89-6dc0-462a-a538-f3e7446d6a60-secret-alertmanager-kube-rbac-proxy-web\") pod \"c3deca89-6dc0-462a-a538-f3e7446d6a60\" (UID: \"c3deca89-6dc0-462a-a538-f3e7446d6a60\") "
Sep 16 18:43:37.203095 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.203077    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3deca89-6dc0-462a-a538-f3e7446d6a60-metrics-client-ca\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:43:37.203281 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.203223    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3deca89-6dc0-462a-a538-f3e7446d6a60-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "c3deca89-6dc0-462a-a538-f3e7446d6a60" (UID: "c3deca89-6dc0-462a-a538-f3e7446d6a60"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:37.203926 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.203850    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3deca89-6dc0-462a-a538-f3e7446d6a60-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "c3deca89-6dc0-462a-a538-f3e7446d6a60" (UID: "c3deca89-6dc0-462a-a538-f3e7446d6a60"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:43:37.205612 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.205586    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3deca89-6dc0-462a-a538-f3e7446d6a60-config-volume" (OuterVolumeSpecName: "config-volume") pod "c3deca89-6dc0-462a-a538-f3e7446d6a60" (UID: "c3deca89-6dc0-462a-a538-f3e7446d6a60"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:43:37.206121 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.206091    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3deca89-6dc0-462a-a538-f3e7446d6a60-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "c3deca89-6dc0-462a-a538-f3e7446d6a60" (UID: "c3deca89-6dc0-462a-a538-f3e7446d6a60"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:43:37.206217 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.206127    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3deca89-6dc0-462a-a538-f3e7446d6a60-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "c3deca89-6dc0-462a-a538-f3e7446d6a60" (UID: "c3deca89-6dc0-462a-a538-f3e7446d6a60"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:43:37.206268 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.206218    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3deca89-6dc0-462a-a538-f3e7446d6a60-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "c3deca89-6dc0-462a-a538-f3e7446d6a60" (UID: "c3deca89-6dc0-462a-a538-f3e7446d6a60"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:43:37.206268 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.206250    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3deca89-6dc0-462a-a538-f3e7446d6a60-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "c3deca89-6dc0-462a-a538-f3e7446d6a60" (UID: "c3deca89-6dc0-462a-a538-f3e7446d6a60"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:43:37.206583 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.206564    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3deca89-6dc0-462a-a538-f3e7446d6a60-config-out" (OuterVolumeSpecName: "config-out") pod "c3deca89-6dc0-462a-a538-f3e7446d6a60" (UID: "c3deca89-6dc0-462a-a538-f3e7446d6a60"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:43:37.207498 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.207474    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3deca89-6dc0-462a-a538-f3e7446d6a60-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "c3deca89-6dc0-462a-a538-f3e7446d6a60" (UID: "c3deca89-6dc0-462a-a538-f3e7446d6a60"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:43:37.207603 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.207508    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3deca89-6dc0-462a-a538-f3e7446d6a60-kube-api-access-fvkrs" (OuterVolumeSpecName: "kube-api-access-fvkrs") pod "c3deca89-6dc0-462a-a538-f3e7446d6a60" (UID: "c3deca89-6dc0-462a-a538-f3e7446d6a60"). InnerVolumeSpecName "kube-api-access-fvkrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:43:37.211244 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.211222    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3deca89-6dc0-462a-a538-f3e7446d6a60-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "c3deca89-6dc0-462a-a538-f3e7446d6a60" (UID: "c3deca89-6dc0-462a-a538-f3e7446d6a60"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:43:37.218819 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.218794    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3deca89-6dc0-462a-a538-f3e7446d6a60-web-config" (OuterVolumeSpecName: "web-config") pod "c3deca89-6dc0-462a-a538-f3e7446d6a60" (UID: "c3deca89-6dc0-462a-a538-f3e7446d6a60"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:43:37.304365 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.304326    2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3deca89-6dc0-462a-a538-f3e7446d6a60-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:43:37.304365 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.304357    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c3deca89-6dc0-462a-a538-f3e7446d6a60-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:43:37.304365 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.304372    2577 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c3deca89-6dc0-462a-a538-f3e7446d6a60-cluster-tls-config\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:43:37.304653 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.304386    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvkrs\" (UniqueName: \"kubernetes.io/projected/c3deca89-6dc0-462a-a538-f3e7446d6a60-kube-api-access-fvkrs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:43:37.304653 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.304398    2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c3deca89-6dc0-462a-a538-f3e7446d6a60-config-out\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:43:37.304653 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.304409    2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c3deca89-6dc0-462a-a538-f3e7446d6a60-alertmanager-main-db\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:43:37.304653 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.304421    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c3deca89-6dc0-462a-a538-f3e7446d6a60-secret-alertmanager-main-tls\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:43:37.304653 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.304432    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c3deca89-6dc0-462a-a538-f3e7446d6a60-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:43:37.304653 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.304446    2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c3deca89-6dc0-462a-a538-f3e7446d6a60-tls-assets\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:43:37.304653 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.304458    2577 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c3deca89-6dc0-462a-a538-f3e7446d6a60-config-volume\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:43:37.304653 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.304469    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c3deca89-6dc0-462a-a538-f3e7446d6a60-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:43:37.304653 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.304482    2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c3deca89-6dc0-462a-a538-f3e7446d6a60-web-config\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:43:37.931347 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.931310    2577 generic.go:356] "Generic (PLEG): container finished" podID="c3deca89-6dc0-462a-a538-f3e7446d6a60" containerID="ea02c865fb9fd0e74de5a8b52bb3cecd51f301eb005079f13da1806dc17fcd55" exitCode=0
Sep 16 18:43:37.931347 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.931341    2577 generic.go:356] "Generic (PLEG): container finished" podID="c3deca89-6dc0-462a-a538-f3e7446d6a60" containerID="a1b6aa4fb8625aa9db441ea01ecf2e24ed8ee9c8c8f26c20f50de979adb7e99a" exitCode=0
Sep 16 18:43:37.931894 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.931393    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3deca89-6dc0-462a-a538-f3e7446d6a60","Type":"ContainerDied","Data":"ea02c865fb9fd0e74de5a8b52bb3cecd51f301eb005079f13da1806dc17fcd55"}
Sep 16 18:43:37.931894 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.931412    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:43:37.931894 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.931500    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3deca89-6dc0-462a-a538-f3e7446d6a60","Type":"ContainerDied","Data":"a1b6aa4fb8625aa9db441ea01ecf2e24ed8ee9c8c8f26c20f50de979adb7e99a"}
Sep 16 18:43:37.931894 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.931552    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3deca89-6dc0-462a-a538-f3e7446d6a60","Type":"ContainerDied","Data":"bb40c2f173e44efde681085770375fe5c9af34a74c8f13a018e21c88f5fb909c"}
Sep 16 18:43:37.931894 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.931559    2577 scope.go:117] "RemoveContainer" containerID="a06e3cf95b8dfaa0d8de4f89af42800fc8aeca0c97c7900bee63856433a3b214"
Sep 16 18:43:37.941113 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.941095    2577 scope.go:117] "RemoveContainer" containerID="ea02c865fb9fd0e74de5a8b52bb3cecd51f301eb005079f13da1806dc17fcd55"
Sep 16 18:43:37.948392 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.948374    2577 scope.go:117] "RemoveContainer" containerID="279a6e0ff7f821e508801c992ab47337373bffecd0d5aa65e4fb92b8d95cc228"
Sep 16 18:43:37.955760 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.955737    2577 scope.go:117] "RemoveContainer" containerID="a1b6aa4fb8625aa9db441ea01ecf2e24ed8ee9c8c8f26c20f50de979adb7e99a"
Sep 16 18:43:37.957096 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.957079    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:43:37.962464 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.962442    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:43:37.963701 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.963686    2577 scope.go:117] "RemoveContainer" containerID="ab4a095c1ec922b35af235bb9ea64a7ef30e0163263e4e1a6bfbcc04a8580142"
Sep 16 18:43:37.970776 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.970759    2577 scope.go:117] "RemoveContainer" containerID="25c71b6a2f66bfaf8d97dca768c506f0e469d96848ea570363f91ba44082e1fa"
Sep 16 18:43:37.977561 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.977541    2577 scope.go:117] "RemoveContainer" containerID="427e34f72c9197ac3fa9e9d741e44635304886654567df2f0ae9eee9ace9ecfc"
Sep 16 18:43:37.984086 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.984067    2577 scope.go:117] "RemoveContainer" containerID="a06e3cf95b8dfaa0d8de4f89af42800fc8aeca0c97c7900bee63856433a3b214"
Sep 16 18:43:37.984362 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:43:37.984345    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a06e3cf95b8dfaa0d8de4f89af42800fc8aeca0c97c7900bee63856433a3b214\": container with ID starting with a06e3cf95b8dfaa0d8de4f89af42800fc8aeca0c97c7900bee63856433a3b214 not found: ID does not exist" containerID="a06e3cf95b8dfaa0d8de4f89af42800fc8aeca0c97c7900bee63856433a3b214"
Sep 16 18:43:37.984427 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.984371    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a06e3cf95b8dfaa0d8de4f89af42800fc8aeca0c97c7900bee63856433a3b214"} err="failed to get container status \"a06e3cf95b8dfaa0d8de4f89af42800fc8aeca0c97c7900bee63856433a3b214\": rpc error: code = NotFound desc = could not find container \"a06e3cf95b8dfaa0d8de4f89af42800fc8aeca0c97c7900bee63856433a3b214\": container with ID starting with a06e3cf95b8dfaa0d8de4f89af42800fc8aeca0c97c7900bee63856433a3b214 not found: ID does not exist"
Sep 16 18:43:37.984427 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.984403    2577 scope.go:117] "RemoveContainer" containerID="ea02c865fb9fd0e74de5a8b52bb3cecd51f301eb005079f13da1806dc17fcd55"
Sep 16 18:43:37.984684 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:43:37.984666    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea02c865fb9fd0e74de5a8b52bb3cecd51f301eb005079f13da1806dc17fcd55\": container with ID starting with ea02c865fb9fd0e74de5a8b52bb3cecd51f301eb005079f13da1806dc17fcd55 not found: ID does not exist" containerID="ea02c865fb9fd0e74de5a8b52bb3cecd51f301eb005079f13da1806dc17fcd55"
Sep 16 18:43:37.984744 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.984689    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea02c865fb9fd0e74de5a8b52bb3cecd51f301eb005079f13da1806dc17fcd55"} err="failed to get container status \"ea02c865fb9fd0e74de5a8b52bb3cecd51f301eb005079f13da1806dc17fcd55\": rpc error: code = NotFound desc = could not find container \"ea02c865fb9fd0e74de5a8b52bb3cecd51f301eb005079f13da1806dc17fcd55\": container with ID starting with ea02c865fb9fd0e74de5a8b52bb3cecd51f301eb005079f13da1806dc17fcd55 not found: ID does not exist"
Sep 16 18:43:37.984744 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.984707    2577 scope.go:117] "RemoveContainer" containerID="279a6e0ff7f821e508801c992ab47337373bffecd0d5aa65e4fb92b8d95cc228"
Sep 16 18:43:37.984927 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:43:37.984914    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"279a6e0ff7f821e508801c992ab47337373bffecd0d5aa65e4fb92b8d95cc228\": container with ID starting with 279a6e0ff7f821e508801c992ab47337373bffecd0d5aa65e4fb92b8d95cc228 not found: ID does not exist" containerID="279a6e0ff7f821e508801c992ab47337373bffecd0d5aa65e4fb92b8d95cc228"
Sep 16 18:43:37.984969 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.984930    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"279a6e0ff7f821e508801c992ab47337373bffecd0d5aa65e4fb92b8d95cc228"} err="failed to get container status \"279a6e0ff7f821e508801c992ab47337373bffecd0d5aa65e4fb92b8d95cc228\": rpc error: code = NotFound desc = could not find container \"279a6e0ff7f821e508801c992ab47337373bffecd0d5aa65e4fb92b8d95cc228\": container with ID starting with 279a6e0ff7f821e508801c992ab47337373bffecd0d5aa65e4fb92b8d95cc228 not found: ID does not exist"
Sep 16 18:43:37.984969 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.984941    2577 scope.go:117] "RemoveContainer" containerID="a1b6aa4fb8625aa9db441ea01ecf2e24ed8ee9c8c8f26c20f50de979adb7e99a"
Sep 16 18:43:37.985156 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:43:37.985138    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1b6aa4fb8625aa9db441ea01ecf2e24ed8ee9c8c8f26c20f50de979adb7e99a\": container with ID starting with a1b6aa4fb8625aa9db441ea01ecf2e24ed8ee9c8c8f26c20f50de979adb7e99a not found: ID does not exist" containerID="a1b6aa4fb8625aa9db441ea01ecf2e24ed8ee9c8c8f26c20f50de979adb7e99a"
Sep 16 18:43:37.985219 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.985165    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1b6aa4fb8625aa9db441ea01ecf2e24ed8ee9c8c8f26c20f50de979adb7e99a"} err="failed to get container status \"a1b6aa4fb8625aa9db441ea01ecf2e24ed8ee9c8c8f26c20f50de979adb7e99a\": rpc error: code = NotFound desc = could not find container \"a1b6aa4fb8625aa9db441ea01ecf2e24ed8ee9c8c8f26c20f50de979adb7e99a\": container with ID starting with a1b6aa4fb8625aa9db441ea01ecf2e24ed8ee9c8c8f26c20f50de979adb7e99a not found: ID does not exist"
Sep 16 18:43:37.985219 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.985186    2577 scope.go:117] "RemoveContainer" containerID="ab4a095c1ec922b35af235bb9ea64a7ef30e0163263e4e1a6bfbcc04a8580142"
Sep 16 18:43:37.985418 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:43:37.985402    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab4a095c1ec922b35af235bb9ea64a7ef30e0163263e4e1a6bfbcc04a8580142\": container with ID starting with ab4a095c1ec922b35af235bb9ea64a7ef30e0163263e4e1a6bfbcc04a8580142 not found: ID does not exist" containerID="ab4a095c1ec922b35af235bb9ea64a7ef30e0163263e4e1a6bfbcc04a8580142"
Sep 16 18:43:37.985472 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.985420    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab4a095c1ec922b35af235bb9ea64a7ef30e0163263e4e1a6bfbcc04a8580142"} err="failed to get container status \"ab4a095c1ec922b35af235bb9ea64a7ef30e0163263e4e1a6bfbcc04a8580142\": rpc error: code = NotFound desc = could not find container \"ab4a095c1ec922b35af235bb9ea64a7ef30e0163263e4e1a6bfbcc04a8580142\": container with ID starting with ab4a095c1ec922b35af235bb9ea64a7ef30e0163263e4e1a6bfbcc04a8580142 not found: ID does not exist"
Sep 16 18:43:37.985472 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.985432    2577 scope.go:117] "RemoveContainer" containerID="25c71b6a2f66bfaf8d97dca768c506f0e469d96848ea570363f91ba44082e1fa"
Sep 16 18:43:37.985737 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:43:37.985718    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25c71b6a2f66bfaf8d97dca768c506f0e469d96848ea570363f91ba44082e1fa\": container with ID starting with 25c71b6a2f66bfaf8d97dca768c506f0e469d96848ea570363f91ba44082e1fa not found: ID does not exist" containerID="25c71b6a2f66bfaf8d97dca768c506f0e469d96848ea570363f91ba44082e1fa"
Sep 16 18:43:37.985783 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.985742    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25c71b6a2f66bfaf8d97dca768c506f0e469d96848ea570363f91ba44082e1fa"} err="failed to get container status \"25c71b6a2f66bfaf8d97dca768c506f0e469d96848ea570363f91ba44082e1fa\": rpc error: code = NotFound desc = could not find container \"25c71b6a2f66bfaf8d97dca768c506f0e469d96848ea570363f91ba44082e1fa\": container with ID starting with 25c71b6a2f66bfaf8d97dca768c506f0e469d96848ea570363f91ba44082e1fa not found: ID does not exist"
Sep 16 18:43:37.985783 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.985757    2577 scope.go:117] "RemoveContainer" containerID="427e34f72c9197ac3fa9e9d741e44635304886654567df2f0ae9eee9ace9ecfc"
Sep 16 18:43:37.986012 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:43:37.985994    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"427e34f72c9197ac3fa9e9d741e44635304886654567df2f0ae9eee9ace9ecfc\": container with ID starting with 427e34f72c9197ac3fa9e9d741e44635304886654567df2f0ae9eee9ace9ecfc not found: ID does not exist" containerID="427e34f72c9197ac3fa9e9d741e44635304886654567df2f0ae9eee9ace9ecfc"
Sep 16 18:43:37.986067 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.986016    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"427e34f72c9197ac3fa9e9d741e44635304886654567df2f0ae9eee9ace9ecfc"} err="failed to get container status \"427e34f72c9197ac3fa9e9d741e44635304886654567df2f0ae9eee9ace9ecfc\": rpc error: code = NotFound desc = could not find container \"427e34f72c9197ac3fa9e9d741e44635304886654567df2f0ae9eee9ace9ecfc\": container with ID starting with 427e34f72c9197ac3fa9e9d741e44635304886654567df2f0ae9eee9ace9ecfc not found: ID does not exist"
Sep 16 18:43:37.986067 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.986028    2577 scope.go:117] "RemoveContainer" containerID="a06e3cf95b8dfaa0d8de4f89af42800fc8aeca0c97c7900bee63856433a3b214"
Sep 16 18:43:37.986307 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.986272    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a06e3cf95b8dfaa0d8de4f89af42800fc8aeca0c97c7900bee63856433a3b214"} err="failed to get container status \"a06e3cf95b8dfaa0d8de4f89af42800fc8aeca0c97c7900bee63856433a3b214\": rpc error: code = NotFound desc = could not find container \"a06e3cf95b8dfaa0d8de4f89af42800fc8aeca0c97c7900bee63856433a3b214\": container with ID starting with a06e3cf95b8dfaa0d8de4f89af42800fc8aeca0c97c7900bee63856433a3b214 not found: ID does not exist"
Sep 16 18:43:37.986390 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.986308    2577 scope.go:117] "RemoveContainer" containerID="ea02c865fb9fd0e74de5a8b52bb3cecd51f301eb005079f13da1806dc17fcd55"
Sep 16 18:43:37.986592 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.986572    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea02c865fb9fd0e74de5a8b52bb3cecd51f301eb005079f13da1806dc17fcd55"} err="failed to get container status \"ea02c865fb9fd0e74de5a8b52bb3cecd51f301eb005079f13da1806dc17fcd55\": rpc error: code = NotFound desc = could not find container \"ea02c865fb9fd0e74de5a8b52bb3cecd51f301eb005079f13da1806dc17fcd55\": container with ID starting with ea02c865fb9fd0e74de5a8b52bb3cecd51f301eb005079f13da1806dc17fcd55 not found: ID does not exist"
Sep 16 18:43:37.986646 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.986594    2577 scope.go:117] "RemoveContainer" containerID="279a6e0ff7f821e508801c992ab47337373bffecd0d5aa65e4fb92b8d95cc228"
Sep 16 18:43:37.986839 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.986823    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"279a6e0ff7f821e508801c992ab47337373bffecd0d5aa65e4fb92b8d95cc228"} err="failed to get container status \"279a6e0ff7f821e508801c992ab47337373bffecd0d5aa65e4fb92b8d95cc228\": rpc error: code = NotFound desc = could not find container \"279a6e0ff7f821e508801c992ab47337373bffecd0d5aa65e4fb92b8d95cc228\": container with ID starting with 279a6e0ff7f821e508801c992ab47337373bffecd0d5aa65e4fb92b8d95cc228 not found: ID does not exist"
Sep 16 18:43:37.986839 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.986838    2577 scope.go:117] "RemoveContainer" containerID="a1b6aa4fb8625aa9db441ea01ecf2e24ed8ee9c8c8f26c20f50de979adb7e99a"
Sep 16 18:43:37.987069 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.987054    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1b6aa4fb8625aa9db441ea01ecf2e24ed8ee9c8c8f26c20f50de979adb7e99a"} err="failed to get container status \"a1b6aa4fb8625aa9db441ea01ecf2e24ed8ee9c8c8f26c20f50de979adb7e99a\": rpc error: code = NotFound desc = could not find container \"a1b6aa4fb8625aa9db441ea01ecf2e24ed8ee9c8c8f26c20f50de979adb7e99a\": container with ID starting with a1b6aa4fb8625aa9db441ea01ecf2e24ed8ee9c8c8f26c20f50de979adb7e99a not found: ID does not exist"
Sep 16 18:43:37.987069 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.987069    2577 scope.go:117] "RemoveContainer" containerID="ab4a095c1ec922b35af235bb9ea64a7ef30e0163263e4e1a6bfbcc04a8580142"
Sep 16 18:43:37.987314 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.987295    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab4a095c1ec922b35af235bb9ea64a7ef30e0163263e4e1a6bfbcc04a8580142"} err="failed to get container status \"ab4a095c1ec922b35af235bb9ea64a7ef30e0163263e4e1a6bfbcc04a8580142\": rpc error: code = NotFound desc = could not find container \"ab4a095c1ec922b35af235bb9ea64a7ef30e0163263e4e1a6bfbcc04a8580142\": container with ID starting with ab4a095c1ec922b35af235bb9ea64a7ef30e0163263e4e1a6bfbcc04a8580142 not found: ID does not exist"
Sep 16 18:43:37.987314 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.987314    2577 scope.go:117] "RemoveContainer" containerID="25c71b6a2f66bfaf8d97dca768c506f0e469d96848ea570363f91ba44082e1fa"
Sep 16 18:43:37.987651 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.987607    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25c71b6a2f66bfaf8d97dca768c506f0e469d96848ea570363f91ba44082e1fa"} err="failed to get container status \"25c71b6a2f66bfaf8d97dca768c506f0e469d96848ea570363f91ba44082e1fa\": rpc error: code = NotFound desc = could not find container \"25c71b6a2f66bfaf8d97dca768c506f0e469d96848ea570363f91ba44082e1fa\": container with ID starting with 25c71b6a2f66bfaf8d97dca768c506f0e469d96848ea570363f91ba44082e1fa not found: ID does not exist"
Sep 16 18:43:37.987651 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.987647    2577 scope.go:117] "RemoveContainer" containerID="427e34f72c9197ac3fa9e9d741e44635304886654567df2f0ae9eee9ace9ecfc"
Sep 16 18:43:37.988000 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.987969    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"427e34f72c9197ac3fa9e9d741e44635304886654567df2f0ae9eee9ace9ecfc"} err="failed to get container status \"427e34f72c9197ac3fa9e9d741e44635304886654567df2f0ae9eee9ace9ecfc\": rpc error: code = NotFound desc = could not find container \"427e34f72c9197ac3fa9e9d741e44635304886654567df2f0ae9eee9ace9ecfc\": container with ID starting with 427e34f72c9197ac3fa9e9d741e44635304886654567df2f0ae9eee9ace9ecfc not found: ID does not exist"
Sep 16 18:43:37.989972 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.989952    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:43:37.990346 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.990329    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3deca89-6dc0-462a-a538-f3e7446d6a60" containerName="alertmanager"
Sep 16 18:43:37.990382 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.990345    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3deca89-6dc0-462a-a538-f3e7446d6a60" containerName="alertmanager"
Sep 16 18:43:37.990382 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.990359    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3deca89-6dc0-462a-a538-f3e7446d6a60" containerName="config-reloader"
Sep 16 18:43:37.990382 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.990368    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3deca89-6dc0-462a-a538-f3e7446d6a60" containerName="config-reloader"
Sep 16 18:43:37.990469 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.990381    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3deca89-6dc0-462a-a538-f3e7446d6a60" containerName="kube-rbac-proxy-metric"
Sep 16 18:43:37.990469 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.990390    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3deca89-6dc0-462a-a538-f3e7446d6a60" containerName="kube-rbac-proxy-metric"
Sep 16 18:43:37.990469 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.990404    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3deca89-6dc0-462a-a538-f3e7446d6a60" containerName="kube-rbac-proxy-web"
Sep 16 18:43:37.990469 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.990413    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3deca89-6dc0-462a-a538-f3e7446d6a60" containerName="kube-rbac-proxy-web"
Sep 16 18:43:37.990469 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.990423    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3deca89-6dc0-462a-a538-f3e7446d6a60" containerName="prom-label-proxy"
Sep 16 18:43:37.990469 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.990432    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3deca89-6dc0-462a-a538-f3e7446d6a60" containerName="prom-label-proxy"
Sep 16 18:43:37.990469 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.990443    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3deca89-6dc0-462a-a538-f3e7446d6a60" containerName="init-config-reloader"
Sep 16 18:43:37.990469 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.990451    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3deca89-6dc0-462a-a538-f3e7446d6a60" containerName="init-config-reloader"
Sep 16 18:43:37.990469 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.990463    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3deca89-6dc0-462a-a538-f3e7446d6a60" containerName="kube-rbac-proxy"
Sep 16 18:43:37.990469 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.990471    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3deca89-6dc0-462a-a538-f3e7446d6a60" containerName="kube-rbac-proxy"
Sep 16 18:43:37.990816 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.990577    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3deca89-6dc0-462a-a538-f3e7446d6a60" containerName="kube-rbac-proxy"
Sep 16 18:43:37.990816 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.990589    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3deca89-6dc0-462a-a538-f3e7446d6a60" containerName="kube-rbac-proxy-metric"
Sep 16 18:43:37.990816 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.990600    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3deca89-6dc0-462a-a538-f3e7446d6a60" containerName="kube-rbac-proxy-web"
Sep 16 18:43:37.990816 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.990612    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3deca89-6dc0-462a-a538-f3e7446d6a60" containerName="alertmanager"
Sep 16 18:43:37.990816 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.990620    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3deca89-6dc0-462a-a538-f3e7446d6a60" containerName="config-reloader"
Sep 16 18:43:37.990816 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.990630    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3deca89-6dc0-462a-a538-f3e7446d6a60" containerName="prom-label-proxy"
Sep 16 18:43:37.996400 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.996384    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:43:37.999092 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.999073    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 18:43:37.999193 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.999135    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 18:43:37.999193 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.999135    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 18:43:37.999307 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.999268    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 18:43:37.999475 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.999452    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 18:43:37.999565 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.999490    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 18:43:37.999631 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.999602    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8krsz\""
Sep 16 18:43:37.999784 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.999763    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 18:43:37.999892 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:37.999876    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 18:43:38.005095 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.005075    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 18:43:38.010845 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.010799    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:43:38.010949 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.010847    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef-tls-assets\") pod \"alertmanager-main-0\" (UID: \"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:43:38.010949 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.010879    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:43:38.010949 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.010909    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4q88\" (UniqueName: \"kubernetes.io/projected/17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef-kube-api-access-n4q88\") pod \"alertmanager-main-0\" (UID: \"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:43:38.010949 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.010936    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:43:38.011144 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.010965    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:43:38.011144 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.010990    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:43:38.011144 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.011021    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef-web-config\") pod \"alertmanager-main-0\" (UID: \"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:43:38.011144 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.011043    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef-config-out\") pod \"alertmanager-main-0\" (UID: \"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:43:38.011144 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.011066    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:43:38.011144 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.011087    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:43:38.011144 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.011119    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef-config-volume\") pod \"alertmanager-main-0\" (UID: \"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:43:38.011457 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.011154    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:43:38.012355 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.012337    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:43:38.112443 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.112420    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef-web-config\") pod \"alertmanager-main-0\" (UID: \"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:43:38.112561 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.112447    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef-config-out\") pod \"alertmanager-main-0\" (UID: \"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:43:38.112561 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.112464    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:43:38.112561 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.112487    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:43:38.112561 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.112517    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef-config-volume\") pod \"alertmanager-main-0\" (UID: \"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:43:38.112749 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.112571    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:43:38.112749 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.112630    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:43:38.112749 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.112657    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef-tls-assets\") pod \"alertmanager-main-0\" (UID: \"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:43:38.112749 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.112683    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:43:38.112749 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.112712    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4q88\" (UniqueName: \"kubernetes.io/projected/17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef-kube-api-access-n4q88\") pod \"alertmanager-main-0\" (UID: \"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:43:38.112749 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.112737    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:43:38.113030 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.112766    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:43:38.113030 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.112797    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:43:38.115131 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.113775    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:43:38.115131 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.114515    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:43:38.115286 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.115263    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:43:38.115764 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.115739    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:43:38.115874 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.115818    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef-web-config\") pod \"alertmanager-main-0\" (UID: \"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:43:38.116015 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.115996    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef-config-volume\") pod \"alertmanager-main-0\" (UID: \"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:43:38.116084 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.116044    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef-config-out\") pod \"alertmanager-main-0\" (UID: \"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:43:38.116084 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.116072    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:43:38.116468 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.116449    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:43:38.116595 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.116578    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef-tls-assets\") pod \"alertmanager-main-0\" (UID: \"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:43:38.117363 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.117346    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:43:38.117556 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.117540    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:43:38.123856 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.123829    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4q88\" (UniqueName: \"kubernetes.io/projected/17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef-kube-api-access-n4q88\") pod \"alertmanager-main-0\" (UID: \"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:43:38.305564 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.305469    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:43:38.587618 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.587587    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:43:38.589256 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:43:38.589226    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17b4a9e5_49dd_4f0e_b209_c7d7bf1c3bef.slice/crio-335368629dc3a9860a42ae6b3c7b544fc188ee007d19d5e72e7d1a5de6869bb5 WatchSource:0}: Error finding container 335368629dc3a9860a42ae6b3c7b544fc188ee007d19d5e72e7d1a5de6869bb5: Status 404 returned error can't find the container with id 335368629dc3a9860a42ae6b3c7b544fc188ee007d19d5e72e7d1a5de6869bb5
Sep 16 18:43:38.938612 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.938581    2577 generic.go:356] "Generic (PLEG): container finished" podID="17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef" containerID="072aca02b6d7b50aa2a8f5b3b3ad4fc697a8c3ad8bd6835e913cc3209dbb015c" exitCode=0
Sep 16 18:43:38.938952 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.938621    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef","Type":"ContainerDied","Data":"072aca02b6d7b50aa2a8f5b3b3ad4fc697a8c3ad8bd6835e913cc3209dbb015c"}
Sep 16 18:43:38.938952 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:38.938682    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef","Type":"ContainerStarted","Data":"335368629dc3a9860a42ae6b3c7b544fc188ee007d19d5e72e7d1a5de6869bb5"}
Sep 16 18:43:39.155947 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:39.155922    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3deca89-6dc0-462a-a538-f3e7446d6a60" path="/var/lib/kubelet/pods/c3deca89-6dc0-462a-a538-f3e7446d6a60/volumes"
Sep 16 18:43:39.947372 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:39.947330    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef","Type":"ContainerStarted","Data":"7d2439ac19b444852e5fdc3bb7469f2330eeb2f68aa86f358d24ee299b0a33f8"}
Sep 16 18:43:39.947876 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:39.947431    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef","Type":"ContainerStarted","Data":"da8e8403385805cb3d6663286f196ce6cb5961d53090186b85e80dafd50fe758"}
Sep 16 18:43:39.947876 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:39.947467    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef","Type":"ContainerStarted","Data":"26ba8b9f924adb4bc2fad6067beb3e8bc8fc2fc141b187fc852f6f25da392a43"}
Sep 16 18:43:39.947876 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:39.947494    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef","Type":"ContainerStarted","Data":"6e7adc328db6b3c6def8433181b0cff59f9fde77d7b27e21b963bcb2527fc679"}
Sep 16 18:43:39.947876 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:39.947518    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef","Type":"ContainerStarted","Data":"4cda905c8f33801f4d9085e86ad96735f7d381cdc5534992841eebab8abbcca7"}
Sep 16 18:43:39.947876 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:39.947574    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef","Type":"ContainerStarted","Data":"03de6454a1c8b7211483b9321f0bda093ff672235994e0315b283e34ddd13136"}
Sep 16 18:43:39.978055 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:43:39.978002    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.963284309 podStartE2EDuration="2.97798237s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.698061ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:39.975862973 +0000 UTC m=+177.358752335" watchObservedRunningTime="2026-09-16 18:43:39.97798237 +0000 UTC m=+177.360871734"
Sep 16 18:45:20.908919 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:20.908883    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-66m9f"]
Sep 16 18:45:20.912163 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:20.912144    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-66m9f"
Sep 16 18:45:20.914793 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:20.914758    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 16 18:45:20.914904 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:20.914811    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 16 18:45:20.914904 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:20.914825    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-c9wjp\""
Sep 16 18:45:20.914904 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:20.914820    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 16 18:45:20.915940 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:20.915921    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 16 18:45:20.916030 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:20.915971    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 16 18:45:20.921991 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:20.921972    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-66m9f"]
Sep 16 18:45:20.945786 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:20.945760    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdvmp\" (UniqueName: \"kubernetes.io/projected/3c982479-8b6e-48a4-9243-aa1399541afb-kube-api-access-qdvmp\") pod \"keda-metrics-apiserver-77857b7b98-66m9f\" (UID: \"3c982479-8b6e-48a4-9243-aa1399541afb\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-66m9f"
Sep 16 18:45:20.945872 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:20.945790    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/3c982479-8b6e-48a4-9243-aa1399541afb-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-66m9f\" (UID: \"3c982479-8b6e-48a4-9243-aa1399541afb\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-66m9f"
Sep 16 18:45:20.945872 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:20.945808    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3c982479-8b6e-48a4-9243-aa1399541afb-certificates\") pod \"keda-metrics-apiserver-77857b7b98-66m9f\" (UID: \"3c982479-8b6e-48a4-9243-aa1399541afb\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-66m9f"
Sep 16 18:45:21.046344 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:21.046321    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdvmp\" (UniqueName: \"kubernetes.io/projected/3c982479-8b6e-48a4-9243-aa1399541afb-kube-api-access-qdvmp\") pod \"keda-metrics-apiserver-77857b7b98-66m9f\" (UID: \"3c982479-8b6e-48a4-9243-aa1399541afb\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-66m9f"
Sep 16 18:45:21.046423 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:21.046350    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/3c982479-8b6e-48a4-9243-aa1399541afb-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-66m9f\" (UID: \"3c982479-8b6e-48a4-9243-aa1399541afb\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-66m9f"
Sep 16 18:45:21.046423 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:21.046368    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3c982479-8b6e-48a4-9243-aa1399541afb-certificates\") pod \"keda-metrics-apiserver-77857b7b98-66m9f\" (UID: \"3c982479-8b6e-48a4-9243-aa1399541afb\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-66m9f"
Sep 16 18:45:21.046491 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:45:21.046460    2577 secret.go:281] references non-existent secret key: tls.crt
Sep 16 18:45:21.046491 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:45:21.046476    2577 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Sep 16 18:45:21.046572 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:45:21.046494    2577 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Sep 16 18:45:21.046572 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:45:21.046509    2577 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-66m9f: [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Sep 16 18:45:21.046635 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:45:21.046589    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c982479-8b6e-48a4-9243-aa1399541afb-certificates podName:3c982479-8b6e-48a4-9243-aa1399541afb nodeName:}" failed. No retries permitted until 2026-09-16 18:45:21.546573025 +0000 UTC m=+278.929462372 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/3c982479-8b6e-48a4-9243-aa1399541afb-certificates") pod "keda-metrics-apiserver-77857b7b98-66m9f" (UID: "3c982479-8b6e-48a4-9243-aa1399541afb") : [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Sep 16 18:45:21.046716 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:21.046700    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/3c982479-8b6e-48a4-9243-aa1399541afb-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-66m9f\" (UID: \"3c982479-8b6e-48a4-9243-aa1399541afb\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-66m9f"
Sep 16 18:45:21.058574 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:21.058547    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdvmp\" (UniqueName: \"kubernetes.io/projected/3c982479-8b6e-48a4-9243-aa1399541afb-kube-api-access-qdvmp\") pod \"keda-metrics-apiserver-77857b7b98-66m9f\" (UID: \"3c982479-8b6e-48a4-9243-aa1399541afb\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-66m9f"
Sep 16 18:45:21.334030 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:21.333949    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-zx56v"]
Sep 16 18:45:21.337256 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:21.337238    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-zx56v"
Sep 16 18:45:21.339825 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:21.339804    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 16 18:45:21.347728 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:21.347705    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-zx56v"]
Sep 16 18:45:21.349394 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:21.349376    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/70778805-479a-448a-8ccb-1b77306e4272-certificates\") pod \"keda-admission-6c4d864b86-zx56v\" (UID: \"70778805-479a-448a-8ccb-1b77306e4272\") " pod="openshift-keda/keda-admission-6c4d864b86-zx56v"
Sep 16 18:45:21.349495 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:21.349414    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbqtc\" (UniqueName: \"kubernetes.io/projected/70778805-479a-448a-8ccb-1b77306e4272-kube-api-access-kbqtc\") pod \"keda-admission-6c4d864b86-zx56v\" (UID: \"70778805-479a-448a-8ccb-1b77306e4272\") " pod="openshift-keda/keda-admission-6c4d864b86-zx56v"
Sep 16 18:45:21.450603 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:21.450579    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/70778805-479a-448a-8ccb-1b77306e4272-certificates\") pod \"keda-admission-6c4d864b86-zx56v\" (UID: \"70778805-479a-448a-8ccb-1b77306e4272\") " pod="openshift-keda/keda-admission-6c4d864b86-zx56v"
Sep 16 18:45:21.450704 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:21.450616    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbqtc\" (UniqueName: \"kubernetes.io/projected/70778805-479a-448a-8ccb-1b77306e4272-kube-api-access-kbqtc\") pod \"keda-admission-6c4d864b86-zx56v\" (UID: \"70778805-479a-448a-8ccb-1b77306e4272\") " pod="openshift-keda/keda-admission-6c4d864b86-zx56v"
Sep 16 18:45:21.450745 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:45:21.450715    2577 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Sep 16 18:45:21.450745 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:45:21.450739    2577 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-6c4d864b86-zx56v: secret "keda-admission-webhooks-certs" not found
Sep 16 18:45:21.450807 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:45:21.450791    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/70778805-479a-448a-8ccb-1b77306e4272-certificates podName:70778805-479a-448a-8ccb-1b77306e4272 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:21.950777456 +0000 UTC m=+279.333666846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/70778805-479a-448a-8ccb-1b77306e4272-certificates") pod "keda-admission-6c4d864b86-zx56v" (UID: "70778805-479a-448a-8ccb-1b77306e4272") : secret "keda-admission-webhooks-certs" not found
Sep 16 18:45:21.463150 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:21.463131    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbqtc\" (UniqueName: \"kubernetes.io/projected/70778805-479a-448a-8ccb-1b77306e4272-kube-api-access-kbqtc\") pod \"keda-admission-6c4d864b86-zx56v\" (UID: \"70778805-479a-448a-8ccb-1b77306e4272\") " pod="openshift-keda/keda-admission-6c4d864b86-zx56v"
Sep 16 18:45:21.551608 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:21.551591    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3c982479-8b6e-48a4-9243-aa1399541afb-certificates\") pod \"keda-metrics-apiserver-77857b7b98-66m9f\" (UID: \"3c982479-8b6e-48a4-9243-aa1399541afb\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-66m9f"
Sep 16 18:45:21.554067 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:21.554043    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3c982479-8b6e-48a4-9243-aa1399541afb-certificates\") pod \"keda-metrics-apiserver-77857b7b98-66m9f\" (UID: \"3c982479-8b6e-48a4-9243-aa1399541afb\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-66m9f"
Sep 16 18:45:21.822316 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:21.822295    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-66m9f"
Sep 16 18:45:21.953904 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:21.953872    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/70778805-479a-448a-8ccb-1b77306e4272-certificates\") pod \"keda-admission-6c4d864b86-zx56v\" (UID: \"70778805-479a-448a-8ccb-1b77306e4272\") " pod="openshift-keda/keda-admission-6c4d864b86-zx56v"
Sep 16 18:45:21.956271 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:21.956253    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/70778805-479a-448a-8ccb-1b77306e4272-certificates\") pod \"keda-admission-6c4d864b86-zx56v\" (UID: \"70778805-479a-448a-8ccb-1b77306e4272\") " pod="openshift-keda/keda-admission-6c4d864b86-zx56v"
Sep 16 18:45:22.093216 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:22.093193    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-66m9f"]
Sep 16 18:45:22.095718 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:45:22.095693    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c982479_8b6e_48a4_9243_aa1399541afb.slice/crio-0e00f215e1c941b298f004158085b53c4fde2b7173c0d0babe0c45120c191044 WatchSource:0}: Error finding container 0e00f215e1c941b298f004158085b53c4fde2b7173c0d0babe0c45120c191044: Status 404 returned error can't find the container with id 0e00f215e1c941b298f004158085b53c4fde2b7173c0d0babe0c45120c191044
Sep 16 18:45:22.247081 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:22.247060    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-zx56v"
Sep 16 18:45:22.341123 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:22.341092    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-66m9f" event={"ID":"3c982479-8b6e-48a4-9243-aa1399541afb","Type":"ContainerStarted","Data":"0e00f215e1c941b298f004158085b53c4fde2b7173c0d0babe0c45120c191044"}
Sep 16 18:45:22.519610 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:22.519487    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-zx56v"]
Sep 16 18:45:22.521699 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:45:22.521674    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70778805_479a_448a_8ccb_1b77306e4272.slice/crio-2a661e95e7174cb1a711276a46260493c5ff016000cf19b637bba0f6886d17e2 WatchSource:0}: Error finding container 2a661e95e7174cb1a711276a46260493c5ff016000cf19b637bba0f6886d17e2: Status 404 returned error can't find the container with id 2a661e95e7174cb1a711276a46260493c5ff016000cf19b637bba0f6886d17e2
Sep 16 18:45:23.368548 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:23.360852    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-zx56v" event={"ID":"70778805-479a-448a-8ccb-1b77306e4272","Type":"ContainerStarted","Data":"2a661e95e7174cb1a711276a46260493c5ff016000cf19b637bba0f6886d17e2"}
Sep 16 18:45:26.374894 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:26.374858    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-66m9f" event={"ID":"3c982479-8b6e-48a4-9243-aa1399541afb","Type":"ContainerStarted","Data":"d42f616320d53415feada7edbe509d500f8fa21dd8f2eb43e96f50ab8f1dae3c"}
Sep 16 18:45:26.375327 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:26.375091    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-66m9f"
Sep 16 18:45:26.376715 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:26.376690    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-zx56v" event={"ID":"70778805-479a-448a-8ccb-1b77306e4272","Type":"ContainerStarted","Data":"32cf650fb63d9b977f28e2ad47e35b96b2b612158864f4c25393525497287bee"}
Sep 16 18:45:26.376852 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:26.376735    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-zx56v"
Sep 16 18:45:26.393042 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:26.392847    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-66m9f" podStartSLOduration=2.71639626 podStartE2EDuration="6.39283164s" totalImagesPullingTime="3.67643538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:26.391947268 +0000 UTC m=+283.774836634" watchObservedRunningTime="2026-09-16 18:45:26.39283164 +0000 UTC m=+283.775721006"
Sep 16 18:45:26.410047 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:26.410004    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-zx56v" podStartSLOduration=2.160785657 podStartE2EDuration="5.409991002s" totalImagesPullingTime="3.249205345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:26.407991476 +0000 UTC m=+283.790880842" watchObservedRunningTime="2026-09-16 18:45:26.409991002 +0000 UTC m=+283.792880367"
Sep 16 18:45:37.386041 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:37.386015    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-66m9f"
Sep 16 18:45:43.063444 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:43.063415    2577 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 18:45:47.383838 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:45:47.383809    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-zx56v"
Sep 16 18:46:50.603589 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:46:50.603492    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-8pq4w"]
Sep 16 18:46:50.606724 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:46:50.606707    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-8pq4w"
Sep 16 18:46:50.609390 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:46:50.609356    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-rwgsd\""
Sep 16 18:46:50.609510 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:46:50.609429    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 16 18:46:50.610735 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:46:50.610715    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 16 18:46:50.617953 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:46:50.617934    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-8pq4w"]
Sep 16 18:46:50.716593 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:46:50.716563    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdpn2\" (UniqueName: \"kubernetes.io/projected/8dc7e5df-f9c8-401e-8cb6-a4a4bea77156-kube-api-access-cdpn2\") pod \"cert-manager-55b4d954d5-8pq4w\" (UID: \"8dc7e5df-f9c8-401e-8cb6-a4a4bea77156\") " pod="cert-manager/cert-manager-55b4d954d5-8pq4w"
Sep 16 18:46:50.716720 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:46:50.716655    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8dc7e5df-f9c8-401e-8cb6-a4a4bea77156-bound-sa-token\") pod \"cert-manager-55b4d954d5-8pq4w\" (UID: \"8dc7e5df-f9c8-401e-8cb6-a4a4bea77156\") " pod="cert-manager/cert-manager-55b4d954d5-8pq4w"
Sep 16 18:46:50.817352 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:46:50.817315    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdpn2\" (UniqueName: \"kubernetes.io/projected/8dc7e5df-f9c8-401e-8cb6-a4a4bea77156-kube-api-access-cdpn2\") pod \"cert-manager-55b4d954d5-8pq4w\" (UID: \"8dc7e5df-f9c8-401e-8cb6-a4a4bea77156\") " pod="cert-manager/cert-manager-55b4d954d5-8pq4w"
Sep 16 18:46:50.817475 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:46:50.817388    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8dc7e5df-f9c8-401e-8cb6-a4a4bea77156-bound-sa-token\") pod \"cert-manager-55b4d954d5-8pq4w\" (UID: \"8dc7e5df-f9c8-401e-8cb6-a4a4bea77156\") " pod="cert-manager/cert-manager-55b4d954d5-8pq4w"
Sep 16 18:46:50.827617 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:46:50.827586    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8dc7e5df-f9c8-401e-8cb6-a4a4bea77156-bound-sa-token\") pod \"cert-manager-55b4d954d5-8pq4w\" (UID: \"8dc7e5df-f9c8-401e-8cb6-a4a4bea77156\") " pod="cert-manager/cert-manager-55b4d954d5-8pq4w"
Sep 16 18:46:50.827617 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:46:50.827599    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdpn2\" (UniqueName: \"kubernetes.io/projected/8dc7e5df-f9c8-401e-8cb6-a4a4bea77156-kube-api-access-cdpn2\") pod \"cert-manager-55b4d954d5-8pq4w\" (UID: \"8dc7e5df-f9c8-401e-8cb6-a4a4bea77156\") " pod="cert-manager/cert-manager-55b4d954d5-8pq4w"
Sep 16 18:46:50.916299 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:46:50.916268    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-8pq4w"
Sep 16 18:46:51.194869 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:46:51.194697    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-8pq4w"]
Sep 16 18:46:51.197333 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:46:51.197308    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8dc7e5df_f9c8_401e_8cb6_a4a4bea77156.slice/crio-86f3f825475e5659d711b72a028e9d18cef8b3221764b54ce511397cf64fa2a0 WatchSource:0}: Error finding container 86f3f825475e5659d711b72a028e9d18cef8b3221764b54ce511397cf64fa2a0: Status 404 returned error can't find the container with id 86f3f825475e5659d711b72a028e9d18cef8b3221764b54ce511397cf64fa2a0
Sep 16 18:46:51.199219 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:46:51.199202    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:46:51.713103 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:46:51.713067    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-8pq4w" event={"ID":"8dc7e5df-f9c8-401e-8cb6-a4a4bea77156","Type":"ContainerStarted","Data":"86f3f825475e5659d711b72a028e9d18cef8b3221764b54ce511397cf64fa2a0"}
Sep 16 18:46:55.731470 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:46:55.731437    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-8pq4w" event={"ID":"8dc7e5df-f9c8-401e-8cb6-a4a4bea77156","Type":"ContainerStarted","Data":"8e3929c86565ee6a129e66c493681a35e843b1ff86babd0491383749b17c13be"}
Sep 16 18:46:55.751543 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:46:55.751493    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-8pq4w" podStartSLOduration=1.611065502 podStartE2EDuration="5.751477714s" totalImagesPullingTime="4.140412212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:55.74993396 +0000 UTC m=+373.132823326" watchObservedRunningTime="2026-09-16 18:46:55.751477714 +0000 UTC m=+373.134367079"
Sep 16 18:46:59.353545 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:46:59.353509    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-846d5d4569-f72mc"]
Sep 16 18:46:59.357074 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:46:59.357054    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-846d5d4569-f72mc"
Sep 16 18:46:59.359959 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:46:59.359934    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 16 18:46:59.361156 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:46:59.361134    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-8qb4s\""
Sep 16 18:46:59.361242 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:46:59.361228    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 16 18:46:59.361557 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:46:59.361541    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 16 18:46:59.361851 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:46:59.361837    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:46:59.362810 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:46:59.362794    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 16 18:46:59.372850 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:46:59.372827    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-846d5d4569-f72mc"]
Sep 16 18:46:59.490041 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:46:59.490008    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2103e78d-58d1-4458-8fe1-903caef6cf25-manager-config\") pod \"lws-controller-manager-846d5d4569-f72mc\" (UID: \"2103e78d-58d1-4458-8fe1-903caef6cf25\") " pod="openshift-lws-operator/lws-controller-manager-846d5d4569-f72mc"
Sep 16 18:46:59.490041 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:46:59.490046    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2103e78d-58d1-4458-8fe1-903caef6cf25-metrics-cert\") pod \"lws-controller-manager-846d5d4569-f72mc\" (UID: \"2103e78d-58d1-4458-8fe1-903caef6cf25\") " pod="openshift-lws-operator/lws-controller-manager-846d5d4569-f72mc"
Sep 16 18:46:59.490251 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:46:59.490087    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2103e78d-58d1-4458-8fe1-903caef6cf25-cert\") pod \"lws-controller-manager-846d5d4569-f72mc\" (UID: \"2103e78d-58d1-4458-8fe1-903caef6cf25\") " pod="openshift-lws-operator/lws-controller-manager-846d5d4569-f72mc"
Sep 16 18:46:59.490251 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:46:59.490114    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sstvq\" (UniqueName: \"kubernetes.io/projected/2103e78d-58d1-4458-8fe1-903caef6cf25-kube-api-access-sstvq\") pod \"lws-controller-manager-846d5d4569-f72mc\" (UID: \"2103e78d-58d1-4458-8fe1-903caef6cf25\") " pod="openshift-lws-operator/lws-controller-manager-846d5d4569-f72mc"
Sep 16 18:46:59.590950 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:46:59.590914    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2103e78d-58d1-4458-8fe1-903caef6cf25-cert\") pod \"lws-controller-manager-846d5d4569-f72mc\" (UID: \"2103e78d-58d1-4458-8fe1-903caef6cf25\") " pod="openshift-lws-operator/lws-controller-manager-846d5d4569-f72mc"
Sep 16 18:46:59.591095 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:46:59.590954    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sstvq\" (UniqueName: \"kubernetes.io/projected/2103e78d-58d1-4458-8fe1-903caef6cf25-kube-api-access-sstvq\") pod \"lws-controller-manager-846d5d4569-f72mc\" (UID: \"2103e78d-58d1-4458-8fe1-903caef6cf25\") " pod="openshift-lws-operator/lws-controller-manager-846d5d4569-f72mc"
Sep 16 18:46:59.591168 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:46:59.591142    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2103e78d-58d1-4458-8fe1-903caef6cf25-manager-config\") pod \"lws-controller-manager-846d5d4569-f72mc\" (UID: \"2103e78d-58d1-4458-8fe1-903caef6cf25\") " pod="openshift-lws-operator/lws-controller-manager-846d5d4569-f72mc"
Sep 16 18:46:59.591227 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:46:59.591196    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2103e78d-58d1-4458-8fe1-903caef6cf25-metrics-cert\") pod \"lws-controller-manager-846d5d4569-f72mc\" (UID: \"2103e78d-58d1-4458-8fe1-903caef6cf25\") " pod="openshift-lws-operator/lws-controller-manager-846d5d4569-f72mc"
Sep 16 18:46:59.591765 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:46:59.591734    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2103e78d-58d1-4458-8fe1-903caef6cf25-manager-config\") pod \"lws-controller-manager-846d5d4569-f72mc\" (UID: \"2103e78d-58d1-4458-8fe1-903caef6cf25\") " pod="openshift-lws-operator/lws-controller-manager-846d5d4569-f72mc"
Sep 16 18:46:59.593604 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:46:59.593586    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2103e78d-58d1-4458-8fe1-903caef6cf25-cert\") pod \"lws-controller-manager-846d5d4569-f72mc\" (UID: \"2103e78d-58d1-4458-8fe1-903caef6cf25\") " pod="openshift-lws-operator/lws-controller-manager-846d5d4569-f72mc"
Sep 16 18:46:59.593780 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:46:59.593760    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2103e78d-58d1-4458-8fe1-903caef6cf25-metrics-cert\") pod \"lws-controller-manager-846d5d4569-f72mc\" (UID: \"2103e78d-58d1-4458-8fe1-903caef6cf25\") " pod="openshift-lws-operator/lws-controller-manager-846d5d4569-f72mc"
Sep 16 18:46:59.604443 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:46:59.604388    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sstvq\" (UniqueName: \"kubernetes.io/projected/2103e78d-58d1-4458-8fe1-903caef6cf25-kube-api-access-sstvq\") pod \"lws-controller-manager-846d5d4569-f72mc\" (UID: \"2103e78d-58d1-4458-8fe1-903caef6cf25\") " pod="openshift-lws-operator/lws-controller-manager-846d5d4569-f72mc"
Sep 16 18:46:59.666605 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:46:59.666585    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-846d5d4569-f72mc"
Sep 16 18:46:59.959365 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:46:59.959333    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-846d5d4569-f72mc"]
Sep 16 18:46:59.961503 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:46:59.961478    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2103e78d_58d1_4458_8fe1_903caef6cf25.slice/crio-5ef5ca3dc0c2a0bc982bc5c1067795b22cad0215420f9496bcb730bb5cfa6528 WatchSource:0}: Error finding container 5ef5ca3dc0c2a0bc982bc5c1067795b22cad0215420f9496bcb730bb5cfa6528: Status 404 returned error can't find the container with id 5ef5ca3dc0c2a0bc982bc5c1067795b22cad0215420f9496bcb730bb5cfa6528
Sep 16 18:47:00.755071 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:47:00.755008    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-846d5d4569-f72mc" event={"ID":"2103e78d-58d1-4458-8fe1-903caef6cf25","Type":"ContainerStarted","Data":"5ef5ca3dc0c2a0bc982bc5c1067795b22cad0215420f9496bcb730bb5cfa6528"}
Sep 16 18:47:02.765387 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:47:02.765339    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-846d5d4569-f72mc" event={"ID":"2103e78d-58d1-4458-8fe1-903caef6cf25","Type":"ContainerStarted","Data":"4cf97454e2a346fa1fa3317fd8305d1f9ddd369d4a0ca94f7b5bac3388461f84"}
Sep 16 18:47:02.765790 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:47:02.765451    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-846d5d4569-f72mc"
Sep 16 18:47:02.786598 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:47:02.786549    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-846d5d4569-f72mc" podStartSLOduration=1.5030596950000001 podStartE2EDuration="3.786518164s" totalImagesPullingTime="2.283458469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:02.784295119 +0000 UTC m=+380.167184483" watchObservedRunningTime="2026-09-16 18:47:02.786518164 +0000 UTC m=+380.169407529"
Sep 16 18:47:13.771366 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:47:13.771339    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-846d5d4569-f72mc"
Sep 16 18:48:23.147591 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:23.147552    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4g87j"]
Sep 16 18:48:23.151566 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:23.151541    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-4g87j"
Sep 16 18:48:23.154599 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:23.154576    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 18:48:23.155822 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:23.155800    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 16 18:48:23.155912 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:23.155801    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 18:48:23.155912 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:23.155804    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-pnvqr\""
Sep 16 18:48:23.161641 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:23.161602    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4g87j"]
Sep 16 18:48:23.195105 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:23.195081    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4g87j"]
Sep 16 18:48:23.202701 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:23.202675    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6d452793-9c5c-4869-b57e-650ab85fe261-config-file\") pod \"limitador-limitador-84994f7ffc-4g87j\" (UID: \"6d452793-9c5c-4869-b57e-650ab85fe261\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-4g87j"
Sep 16 18:48:23.202799 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:23.202716    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xxc6\" (UniqueName: \"kubernetes.io/projected/6d452793-9c5c-4869-b57e-650ab85fe261-kube-api-access-9xxc6\") pod \"limitador-limitador-84994f7ffc-4g87j\" (UID: \"6d452793-9c5c-4869-b57e-650ab85fe261\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-4g87j"
Sep 16 18:48:23.303500 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:23.303476    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6d452793-9c5c-4869-b57e-650ab85fe261-config-file\") pod \"limitador-limitador-84994f7ffc-4g87j\" (UID: \"6d452793-9c5c-4869-b57e-650ab85fe261\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-4g87j"
Sep 16 18:48:23.303672 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:23.303516    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xxc6\" (UniqueName: \"kubernetes.io/projected/6d452793-9c5c-4869-b57e-650ab85fe261-kube-api-access-9xxc6\") pod \"limitador-limitador-84994f7ffc-4g87j\" (UID: \"6d452793-9c5c-4869-b57e-650ab85fe261\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-4g87j"
Sep 16 18:48:23.304102 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:23.304082    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6d452793-9c5c-4869-b57e-650ab85fe261-config-file\") pod \"limitador-limitador-84994f7ffc-4g87j\" (UID: \"6d452793-9c5c-4869-b57e-650ab85fe261\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-4g87j"
Sep 16 18:48:23.314388 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:23.314367    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xxc6\" (UniqueName: \"kubernetes.io/projected/6d452793-9c5c-4869-b57e-650ab85fe261-kube-api-access-9xxc6\") pod \"limitador-limitador-84994f7ffc-4g87j\" (UID: \"6d452793-9c5c-4869-b57e-650ab85fe261\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-4g87j"
Sep 16 18:48:23.462739 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:23.462682    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-4g87j"
Sep 16 18:48:23.677986 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:23.677954    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-xtxjm"]
Sep 16 18:48:23.680225 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:23.680205    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-xtxjm"
Sep 16 18:48:23.684102 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:23.684083    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jf5d4\""
Sep 16 18:48:23.689629 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:23.689606    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-xtxjm"]
Sep 16 18:48:23.707979 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:23.707952    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5pvm\" (UniqueName: \"kubernetes.io/projected/ecdfc00d-8795-4549-ba9a-cfed848ee058-kube-api-access-j5pvm\") pod \"authorino-7bcfd8f9cd-xtxjm\" (UID: \"ecdfc00d-8795-4549-ba9a-cfed848ee058\") " pod="kuadrant-system/authorino-7bcfd8f9cd-xtxjm"
Sep 16 18:48:23.738164 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:23.738143    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4g87j"]
Sep 16 18:48:23.740067 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:48:23.740045    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d452793_9c5c_4869_b57e_650ab85fe261.slice/crio-a8a2991a3cf213cf24d0382043b9fe81ddaf3d9be7bea0e931b39084dc4a187e WatchSource:0}: Error finding container a8a2991a3cf213cf24d0382043b9fe81ddaf3d9be7bea0e931b39084dc4a187e: Status 404 returned error can't find the container with id a8a2991a3cf213cf24d0382043b9fe81ddaf3d9be7bea0e931b39084dc4a187e
Sep 16 18:48:23.808853 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:23.808828    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5pvm\" (UniqueName: \"kubernetes.io/projected/ecdfc00d-8795-4549-ba9a-cfed848ee058-kube-api-access-j5pvm\") pod \"authorino-7bcfd8f9cd-xtxjm\" (UID: \"ecdfc00d-8795-4549-ba9a-cfed848ee058\") " pod="kuadrant-system/authorino-7bcfd8f9cd-xtxjm"
Sep 16 18:48:23.818983 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:23.818962    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5pvm\" (UniqueName: \"kubernetes.io/projected/ecdfc00d-8795-4549-ba9a-cfed848ee058-kube-api-access-j5pvm\") pod \"authorino-7bcfd8f9cd-xtxjm\" (UID: \"ecdfc00d-8795-4549-ba9a-cfed848ee058\") " pod="kuadrant-system/authorino-7bcfd8f9cd-xtxjm"
Sep 16 18:48:23.989656 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:23.989592    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-xtxjm"
Sep 16 18:48:24.088250 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:24.088217    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-4g87j" event={"ID":"6d452793-9c5c-4869-b57e-650ab85fe261","Type":"ContainerStarted","Data":"a8a2991a3cf213cf24d0382043b9fe81ddaf3d9be7bea0e931b39084dc4a187e"}
Sep 16 18:48:24.256270 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:24.256208    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-xtxjm"]
Sep 16 18:48:24.259389 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:48:24.259349    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podecdfc00d_8795_4549_ba9a_cfed848ee058.slice/crio-5eb0749bc8182aa0831645d8c034decd8c8a955a801bc417fc936273b0245750 WatchSource:0}: Error finding container 5eb0749bc8182aa0831645d8c034decd8c8a955a801bc417fc936273b0245750: Status 404 returned error can't find the container with id 5eb0749bc8182aa0831645d8c034decd8c8a955a801bc417fc936273b0245750
Sep 16 18:48:25.094632 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:25.094506    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-4g87j" event={"ID":"6d452793-9c5c-4869-b57e-650ab85fe261","Type":"ContainerStarted","Data":"a8eb5ff2299f4c3a19ed6a1d41720720cc403d68c7b3ff828a6b9afedeaf521b"}
Sep 16 18:48:25.096029 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:25.096000    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-xtxjm" event={"ID":"ecdfc00d-8795-4549-ba9a-cfed848ee058","Type":"ContainerStarted","Data":"5eb0749bc8182aa0831645d8c034decd8c8a955a801bc417fc936273b0245750"}
Sep 16 18:48:25.113663 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:25.113590    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-4g87j" podStartSLOduration=0.887263754 podStartE2EDuration="2.113576004s" totalImagesPullingTime="1.22631225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:25.112604693 +0000 UTC m=+462.495494059" watchObservedRunningTime="2026-09-16 18:48:25.113576004 +0000 UTC m=+462.496465369"
Sep 16 18:48:26.101449 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:26.101413    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-xtxjm" event={"ID":"ecdfc00d-8795-4549-ba9a-cfed848ee058","Type":"ContainerStarted","Data":"dc04da82a0b9f5e9aa994c63a27d94471f2879446dcf58a422e3420ce160a418"}
Sep 16 18:48:26.101919 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:26.101627    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-4g87j"
Sep 16 18:48:26.119808 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:26.119765    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-xtxjm" podStartSLOduration=1.754245783 podStartE2EDuration="3.119749018s" totalImagesPullingTime="1.365503235s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:26.11849487 +0000 UTC m=+463.501384237" watchObservedRunningTime="2026-09-16 18:48:26.119749018 +0000 UTC m=+463.502638383"
Sep 16 18:48:28.694001 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:28.693967    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-xtxjm"]
Sep 16 18:48:28.694459 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:28.694108    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-xtxjm" podUID="ecdfc00d-8795-4549-ba9a-cfed848ee058" containerName="authorino" containerID="cri-o://dc04da82a0b9f5e9aa994c63a27d94471f2879446dcf58a422e3420ce160a418" gracePeriod=30
Sep 16 18:48:28.937327 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:28.937302    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-xtxjm"
Sep 16 18:48:29.053806 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:29.053750    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5pvm\" (UniqueName: \"kubernetes.io/projected/ecdfc00d-8795-4549-ba9a-cfed848ee058-kube-api-access-j5pvm\") pod \"ecdfc00d-8795-4549-ba9a-cfed848ee058\" (UID: \"ecdfc00d-8795-4549-ba9a-cfed848ee058\") "
Sep 16 18:48:29.056031 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:29.056002    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecdfc00d-8795-4549-ba9a-cfed848ee058-kube-api-access-j5pvm" (OuterVolumeSpecName: "kube-api-access-j5pvm") pod "ecdfc00d-8795-4549-ba9a-cfed848ee058" (UID: "ecdfc00d-8795-4549-ba9a-cfed848ee058"). InnerVolumeSpecName "kube-api-access-j5pvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:48:29.114311 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:29.114285    2577 generic.go:356] "Generic (PLEG): container finished" podID="ecdfc00d-8795-4549-ba9a-cfed848ee058" containerID="dc04da82a0b9f5e9aa994c63a27d94471f2879446dcf58a422e3420ce160a418" exitCode=0
Sep 16 18:48:29.114414 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:29.114320    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-xtxjm"
Sep 16 18:48:29.114414 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:29.114317    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-xtxjm" event={"ID":"ecdfc00d-8795-4549-ba9a-cfed848ee058","Type":"ContainerDied","Data":"dc04da82a0b9f5e9aa994c63a27d94471f2879446dcf58a422e3420ce160a418"}
Sep 16 18:48:29.114414 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:29.114372    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-xtxjm" event={"ID":"ecdfc00d-8795-4549-ba9a-cfed848ee058","Type":"ContainerDied","Data":"5eb0749bc8182aa0831645d8c034decd8c8a955a801bc417fc936273b0245750"}
Sep 16 18:48:29.114414 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:29.114385    2577 scope.go:117] "RemoveContainer" containerID="dc04da82a0b9f5e9aa994c63a27d94471f2879446dcf58a422e3420ce160a418"
Sep 16 18:48:29.122842 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:29.122826    2577 scope.go:117] "RemoveContainer" containerID="dc04da82a0b9f5e9aa994c63a27d94471f2879446dcf58a422e3420ce160a418"
Sep 16 18:48:29.123131 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:48:29.123114    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc04da82a0b9f5e9aa994c63a27d94471f2879446dcf58a422e3420ce160a418\": container with ID starting with dc04da82a0b9f5e9aa994c63a27d94471f2879446dcf58a422e3420ce160a418 not found: ID does not exist" containerID="dc04da82a0b9f5e9aa994c63a27d94471f2879446dcf58a422e3420ce160a418"
Sep 16 18:48:29.123178 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:29.123140    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc04da82a0b9f5e9aa994c63a27d94471f2879446dcf58a422e3420ce160a418"} err="failed to get container status \"dc04da82a0b9f5e9aa994c63a27d94471f2879446dcf58a422e3420ce160a418\": rpc error: code = NotFound desc = could not find container \"dc04da82a0b9f5e9aa994c63a27d94471f2879446dcf58a422e3420ce160a418\": container with ID starting with dc04da82a0b9f5e9aa994c63a27d94471f2879446dcf58a422e3420ce160a418 not found: ID does not exist"
Sep 16 18:48:29.144980 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:29.144958    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-xtxjm"]
Sep 16 18:48:29.146287 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:29.146265    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-xtxjm"]
Sep 16 18:48:29.154567 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:29.154546    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecdfc00d-8795-4549-ba9a-cfed848ee058" path="/var/lib/kubelet/pods/ecdfc00d-8795-4549-ba9a-cfed848ee058/volumes"
Sep 16 18:48:29.154958 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:29.154939    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5pvm\" (UniqueName: \"kubernetes.io/projected/ecdfc00d-8795-4549-ba9a-cfed848ee058-kube-api-access-j5pvm\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:48:37.105702 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:48:37.105669    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-4g87j"
Sep 16 18:50:55.154861 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:50:55.154833    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-sb42p"]
Sep 16 18:50:55.157101 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:50:55.155134    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecdfc00d-8795-4549-ba9a-cfed848ee058" containerName="authorino"
Sep 16 18:50:55.157101 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:50:55.155145    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecdfc00d-8795-4549-ba9a-cfed848ee058" containerName="authorino"
Sep 16 18:50:55.157101 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:50:55.155200    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecdfc00d-8795-4549-ba9a-cfed848ee058" containerName="authorino"
Sep 16 18:50:55.158009 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:50:55.157994    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-sb42p"
Sep 16 18:50:55.160427 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:50:55.160406    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 16 18:50:55.160817 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:50:55.160798    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 18:50:55.160906 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:50:55.160842    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 16 18:50:55.161915 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:50:55.161896    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 18:50:55.162016 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:50:55.161897    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-f9k9l\""
Sep 16 18:50:55.163948 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:50:55.163927    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-sb42p"]
Sep 16 18:50:55.304945 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:50:55.304923    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96f8bb5f-041a-48db-89a6-7ba9cf3192e1-model-cache\") pod \"s3-init-sb42p\" (UID: \"96f8bb5f-041a-48db-89a6-7ba9cf3192e1\") " pod="kserve/s3-init-sb42p"
Sep 16 18:50:55.305081 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:50:55.304964    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcrtf\" (UniqueName: \"kubernetes.io/projected/96f8bb5f-041a-48db-89a6-7ba9cf3192e1-kube-api-access-bcrtf\") pod \"s3-init-sb42p\" (UID: \"96f8bb5f-041a-48db-89a6-7ba9cf3192e1\") " pod="kserve/s3-init-sb42p"
Sep 16 18:50:55.406243 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:50:55.406189    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96f8bb5f-041a-48db-89a6-7ba9cf3192e1-model-cache\") pod \"s3-init-sb42p\" (UID: \"96f8bb5f-041a-48db-89a6-7ba9cf3192e1\") " pod="kserve/s3-init-sb42p"
Sep 16 18:50:55.406243 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:50:55.406217    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcrtf\" (UniqueName: \"kubernetes.io/projected/96f8bb5f-041a-48db-89a6-7ba9cf3192e1-kube-api-access-bcrtf\") pod \"s3-init-sb42p\" (UID: \"96f8bb5f-041a-48db-89a6-7ba9cf3192e1\") " pod="kserve/s3-init-sb42p"
Sep 16 18:50:55.406524 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:50:55.406508    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96f8bb5f-041a-48db-89a6-7ba9cf3192e1-model-cache\") pod \"s3-init-sb42p\" (UID: \"96f8bb5f-041a-48db-89a6-7ba9cf3192e1\") " pod="kserve/s3-init-sb42p"
Sep 16 18:50:55.416298 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:50:55.416278    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcrtf\" (UniqueName: \"kubernetes.io/projected/96f8bb5f-041a-48db-89a6-7ba9cf3192e1-kube-api-access-bcrtf\") pod \"s3-init-sb42p\" (UID: \"96f8bb5f-041a-48db-89a6-7ba9cf3192e1\") " pod="kserve/s3-init-sb42p"
Sep 16 18:50:55.467846 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:50:55.467824    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-sb42p"
Sep 16 18:50:55.740308 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:50:55.740286    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-sb42p"]
Sep 16 18:50:55.742461 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:50:55.742434    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96f8bb5f_041a_48db_89a6_7ba9cf3192e1.slice/crio-61c81b710eb3618569bcd6df6ab760fa4ca3c6dc9a6f420e3d900edd49fef218 WatchSource:0}: Error finding container 61c81b710eb3618569bcd6df6ab760fa4ca3c6dc9a6f420e3d900edd49fef218: Status 404 returned error can't find the container with id 61c81b710eb3618569bcd6df6ab760fa4ca3c6dc9a6f420e3d900edd49fef218
Sep 16 18:50:56.690605 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:50:56.690572    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-sb42p" event={"ID":"96f8bb5f-041a-48db-89a6-7ba9cf3192e1","Type":"ContainerStarted","Data":"61c81b710eb3618569bcd6df6ab760fa4ca3c6dc9a6f420e3d900edd49fef218"}
Sep 16 18:50:59.708149 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:50:59.708109    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-sb42p" event={"ID":"96f8bb5f-041a-48db-89a6-7ba9cf3192e1","Type":"ContainerStarted","Data":"fc72c1d2fd993130cb1f54d5a449cd50a79633da63c7740ecf722b0510626bda"}
Sep 16 18:51:00.714122 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:51:00.714092    2577 generic.go:356] "Generic (PLEG): container finished" podID="96f8bb5f-041a-48db-89a6-7ba9cf3192e1" containerID="fc72c1d2fd993130cb1f54d5a449cd50a79633da63c7740ecf722b0510626bda" exitCode=0
Sep 16 18:51:00.714418 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:51:00.714131    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-sb42p" event={"ID":"96f8bb5f-041a-48db-89a6-7ba9cf3192e1","Type":"ContainerDied","Data":"fc72c1d2fd993130cb1f54d5a449cd50a79633da63c7740ecf722b0510626bda"}
Sep 16 18:51:05.741890 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:51:05.741856    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-sb42p" event={"ID":"96f8bb5f-041a-48db-89a6-7ba9cf3192e1","Type":"ContainerStarted","Data":"6c4cceca0589a9079adb0ab88c0794bb9cfe563d9e09c8867cbbd38a631e93da"}
Sep 16 18:51:05.761671 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:51:05.761625    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-sb42p" podStartSLOduration=1.55852047 podStartE2EDuration="10.761610271s" totalImagesPullingTime="7.508179271s" totalInitContainerRuntime="1.69491053s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:50:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:51:05.760232734 +0000 UTC m=+623.143122101" watchObservedRunningTime="2026-09-16 18:51:05.761610271 +0000 UTC m=+623.144499635"
Sep 16 18:52:10.005169 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:10.005105    2577 generic.go:356] "Generic (PLEG): container finished" podID="96f8bb5f-041a-48db-89a6-7ba9cf3192e1" containerID="6c4cceca0589a9079adb0ab88c0794bb9cfe563d9e09c8867cbbd38a631e93da" exitCode=0
Sep 16 18:52:10.005634 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:10.005187    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-sb42p" event={"ID":"96f8bb5f-041a-48db-89a6-7ba9cf3192e1","Type":"ContainerDied","Data":"6c4cceca0589a9079adb0ab88c0794bb9cfe563d9e09c8867cbbd38a631e93da"}
Sep 16 18:52:11.133626 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:11.133601    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-sb42p"
Sep 16 18:52:11.284065 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:11.283980    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96f8bb5f-041a-48db-89a6-7ba9cf3192e1-model-cache\") pod \"96f8bb5f-041a-48db-89a6-7ba9cf3192e1\" (UID: \"96f8bb5f-041a-48db-89a6-7ba9cf3192e1\") "
Sep 16 18:52:11.284065 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:11.284022    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcrtf\" (UniqueName: \"kubernetes.io/projected/96f8bb5f-041a-48db-89a6-7ba9cf3192e1-kube-api-access-bcrtf\") pod \"96f8bb5f-041a-48db-89a6-7ba9cf3192e1\" (UID: \"96f8bb5f-041a-48db-89a6-7ba9cf3192e1\") "
Sep 16 18:52:11.286475 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:11.286443    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96f8bb5f-041a-48db-89a6-7ba9cf3192e1-kube-api-access-bcrtf" (OuterVolumeSpecName: "kube-api-access-bcrtf") pod "96f8bb5f-041a-48db-89a6-7ba9cf3192e1" (UID: "96f8bb5f-041a-48db-89a6-7ba9cf3192e1"). InnerVolumeSpecName "kube-api-access-bcrtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:52:11.302468 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:11.302442    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96f8bb5f-041a-48db-89a6-7ba9cf3192e1-model-cache" (OuterVolumeSpecName: "model-cache") pod "96f8bb5f-041a-48db-89a6-7ba9cf3192e1" (UID: "96f8bb5f-041a-48db-89a6-7ba9cf3192e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:52:11.385492 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:11.385470    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcrtf\" (UniqueName: \"kubernetes.io/projected/96f8bb5f-041a-48db-89a6-7ba9cf3192e1-kube-api-access-bcrtf\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:52:11.385492 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:11.385492    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96f8bb5f-041a-48db-89a6-7ba9cf3192e1-model-cache\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:52:12.014297 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:12.014266    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-sb42p"
Sep 16 18:52:12.014297 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:12.014289    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-sb42p" event={"ID":"96f8bb5f-041a-48db-89a6-7ba9cf3192e1","Type":"ContainerDied","Data":"61c81b710eb3618569bcd6df6ab760fa4ca3c6dc9a6f420e3d900edd49fef218"}
Sep 16 18:52:12.014509 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:12.014327    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61c81b710eb3618569bcd6df6ab760fa4ca3c6dc9a6f420e3d900edd49fef218"
Sep 16 18:52:33.550284 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:33.550253    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v8l5b"]
Sep 16 18:52:33.550711 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:33.550676    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96f8bb5f-041a-48db-89a6-7ba9cf3192e1" containerName="s3-init"
Sep 16 18:52:33.550711 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:33.550689    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="96f8bb5f-041a-48db-89a6-7ba9cf3192e1" containerName="s3-init"
Sep 16 18:52:33.550711 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:33.550702    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96f8bb5f-041a-48db-89a6-7ba9cf3192e1" containerName="download-hf-model"
Sep 16 18:52:33.550711 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:33.550707    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="96f8bb5f-041a-48db-89a6-7ba9cf3192e1" containerName="download-hf-model"
Sep 16 18:52:33.550869 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:33.550763    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="96f8bb5f-041a-48db-89a6-7ba9cf3192e1" containerName="s3-init"
Sep 16 18:52:33.554060 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:33.554043    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v8l5b"
Sep 16 18:52:33.558026 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:33.558000    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 16 18:52:33.558138 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:33.558024    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-pv4rj\""
Sep 16 18:52:33.558138 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:33.558006    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 16 18:52:33.558138 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:33.558111    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 16 18:52:33.566762 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:33.566742    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v8l5b"]
Sep 16 18:52:33.649381 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:33.649348    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04f31f32-25e9-4624-aabd-49f148cf6ac6-dshm\") pod \"fc-smoke-test-kserve-7f4687897-v8l5b\" (UID: \"04f31f32-25e9-4624-aabd-49f148cf6ac6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v8l5b"
Sep 16 18:52:33.649568 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:33.649398    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04f31f32-25e9-4624-aabd-49f148cf6ac6-home\") pod \"fc-smoke-test-kserve-7f4687897-v8l5b\" (UID: \"04f31f32-25e9-4624-aabd-49f148cf6ac6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v8l5b"
Sep 16 18:52:33.649568 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:33.649425    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmnz7\" (UniqueName: \"kubernetes.io/projected/04f31f32-25e9-4624-aabd-49f148cf6ac6-kube-api-access-fmnz7\") pod \"fc-smoke-test-kserve-7f4687897-v8l5b\" (UID: \"04f31f32-25e9-4624-aabd-49f148cf6ac6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v8l5b"
Sep 16 18:52:33.649568 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:33.649485    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04f31f32-25e9-4624-aabd-49f148cf6ac6-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-v8l5b\" (UID: \"04f31f32-25e9-4624-aabd-49f148cf6ac6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v8l5b"
Sep 16 18:52:33.649568 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:33.649545    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04f31f32-25e9-4624-aabd-49f148cf6ac6-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-v8l5b\" (UID: \"04f31f32-25e9-4624-aabd-49f148cf6ac6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v8l5b"
Sep 16 18:52:33.649760 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:33.649608    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04f31f32-25e9-4624-aabd-49f148cf6ac6-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-v8l5b\" (UID: \"04f31f32-25e9-4624-aabd-49f148cf6ac6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v8l5b"
Sep 16 18:52:33.750691 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:33.750662    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04f31f32-25e9-4624-aabd-49f148cf6ac6-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-v8l5b\" (UID: \"04f31f32-25e9-4624-aabd-49f148cf6ac6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v8l5b"
Sep 16 18:52:33.750825 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:33.750699    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04f31f32-25e9-4624-aabd-49f148cf6ac6-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-v8l5b\" (UID: \"04f31f32-25e9-4624-aabd-49f148cf6ac6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v8l5b"
Sep 16 18:52:33.750825 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:33.750730    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04f31f32-25e9-4624-aabd-49f148cf6ac6-dshm\") pod \"fc-smoke-test-kserve-7f4687897-v8l5b\" (UID: \"04f31f32-25e9-4624-aabd-49f148cf6ac6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v8l5b"
Sep 16 18:52:33.750825 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:33.750752    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04f31f32-25e9-4624-aabd-49f148cf6ac6-home\") pod \"fc-smoke-test-kserve-7f4687897-v8l5b\" (UID: \"04f31f32-25e9-4624-aabd-49f148cf6ac6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v8l5b"
Sep 16 18:52:33.750825 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:33.750770    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmnz7\" (UniqueName: \"kubernetes.io/projected/04f31f32-25e9-4624-aabd-49f148cf6ac6-kube-api-access-fmnz7\") pod \"fc-smoke-test-kserve-7f4687897-v8l5b\" (UID: \"04f31f32-25e9-4624-aabd-49f148cf6ac6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v8l5b"
Sep 16 18:52:33.750825 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:33.750793    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04f31f32-25e9-4624-aabd-49f148cf6ac6-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-v8l5b\" (UID: \"04f31f32-25e9-4624-aabd-49f148cf6ac6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v8l5b"
Sep 16 18:52:33.751125 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:33.751107    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04f31f32-25e9-4624-aabd-49f148cf6ac6-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-v8l5b\" (UID: \"04f31f32-25e9-4624-aabd-49f148cf6ac6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v8l5b"
Sep 16 18:52:33.751181 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:33.751141    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04f31f32-25e9-4624-aabd-49f148cf6ac6-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-v8l5b\" (UID: \"04f31f32-25e9-4624-aabd-49f148cf6ac6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v8l5b"
Sep 16 18:52:33.751181 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:33.751163    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04f31f32-25e9-4624-aabd-49f148cf6ac6-home\") pod \"fc-smoke-test-kserve-7f4687897-v8l5b\" (UID: \"04f31f32-25e9-4624-aabd-49f148cf6ac6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v8l5b"
Sep 16 18:52:33.753055 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:33.753034    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04f31f32-25e9-4624-aabd-49f148cf6ac6-dshm\") pod \"fc-smoke-test-kserve-7f4687897-v8l5b\" (UID: \"04f31f32-25e9-4624-aabd-49f148cf6ac6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v8l5b"
Sep 16 18:52:33.753336 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:33.753317    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04f31f32-25e9-4624-aabd-49f148cf6ac6-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-v8l5b\" (UID: \"04f31f32-25e9-4624-aabd-49f148cf6ac6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v8l5b"
Sep 16 18:52:33.761622 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:33.761601    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmnz7\" (UniqueName: \"kubernetes.io/projected/04f31f32-25e9-4624-aabd-49f148cf6ac6-kube-api-access-fmnz7\") pod \"fc-smoke-test-kserve-7f4687897-v8l5b\" (UID: \"04f31f32-25e9-4624-aabd-49f148cf6ac6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v8l5b"
Sep 16 18:52:33.863955 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:33.863888    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v8l5b"
Sep 16 18:52:34.144948 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:34.144882    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v8l5b"]
Sep 16 18:52:34.147132 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:52:34.147106    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04f31f32_25e9_4624_aabd_49f148cf6ac6.slice/crio-573df5d2a13a3a2768a30f1e192f097a90604c3ef7b31cf7ed1666c1e2a8e957 WatchSource:0}: Error finding container 573df5d2a13a3a2768a30f1e192f097a90604c3ef7b31cf7ed1666c1e2a8e957: Status 404 returned error can't find the container with id 573df5d2a13a3a2768a30f1e192f097a90604c3ef7b31cf7ed1666c1e2a8e957
Sep 16 18:52:34.149105 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:34.149088    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:52:35.109921 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:35.109844    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v8l5b" event={"ID":"04f31f32-25e9-4624-aabd-49f148cf6ac6","Type":"ContainerStarted","Data":"573df5d2a13a3a2768a30f1e192f097a90604c3ef7b31cf7ed1666c1e2a8e957"}
Sep 16 18:52:36.121282 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:36.121187    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v8l5b" event={"ID":"04f31f32-25e9-4624-aabd-49f148cf6ac6","Type":"ContainerStarted","Data":"f533e11ae0f603f6ffe43f3dff8a5289c3e48580ebb411e0fc971df2c999b520"}
Sep 16 18:52:36.141911 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:36.141861    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v8l5b" podStartSLOduration=1.55086291 podStartE2EDuration="3.141846167s" totalImagesPullingTime="1.590983257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:36.14079835 +0000 UTC m=+713.523687714" watchObservedRunningTime="2026-09-16 18:52:36.141846167 +0000 UTC m=+713.524735533"
Sep 16 18:52:36.864680 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:36.864646    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v8l5b"
Sep 16 18:52:39.213284 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:39.213250    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 18:52:39.216736 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:39.216716    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:52:39.219752 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:39.219732    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 16 18:52:39.219867 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:39.219752    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:52:39.219867 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:39.219783    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-zw8kr\""
Sep 16 18:52:39.219867 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:39.219832    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 16 18:52:39.220788 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:39.220772    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 16 18:52:39.221027 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:39.221001    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 16 18:52:39.221124 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:39.221043    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-xnzpq\""
Sep 16 18:52:39.230876 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:39.230857    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 18:52:39.296867 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:39.296831    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:52:39.296867 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:39.296868    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:52:39.297091 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:39.296889    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:52:39.297091 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:39.296946    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj9bs\" (UniqueName: \"kubernetes.io/projected/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-kube-api-access-pj9bs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:52:39.297091 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:39.297067    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:52:39.297247 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:39.297135    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:52:39.297247 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:39.297193    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:52:39.297346 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:39.297261    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:52:39.398730 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:39.398696    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:52:39.398730 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:39.398731    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:52:39.398962 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:39.398750    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:52:39.398962 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:39.398772    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj9bs\" (UniqueName: \"kubernetes.io/projected/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-kube-api-access-pj9bs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:52:39.398962 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:39.398813    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:52:39.398962 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:39.398844    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:52:39.398962 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:39.398885    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:52:39.398962 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:39.398940    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:52:39.399264 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:39.399090    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:52:39.399264 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:39.399214    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:52:39.399365 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:39.399285    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:52:39.399365 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:39.399352    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:52:39.399732 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:39.399483    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:52:39.401054 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:39.401034    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:52:39.401282 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:39.401259    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:52:39.408889 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:39.408869    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj9bs\" (UniqueName: \"kubernetes.io/projected/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-kube-api-access-pj9bs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:52:39.527582 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:39.527500    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:52:39.804646 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:39.804572    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 18:52:39.807668 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:52:39.807637    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4d184c4_8f57_48f3_bb49_4c80e4fd6b79.slice/crio-f784e44725622ac8d8de000485c529712f677af48597aa4a6c3cbe3669777b72 WatchSource:0}: Error finding container f784e44725622ac8d8de000485c529712f677af48597aa4a6c3cbe3669777b72: Status 404 returned error can't find the container with id f784e44725622ac8d8de000485c529712f677af48597aa4a6c3cbe3669777b72
Sep 16 18:52:40.139319 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:40.139283    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79","Type":"ContainerStarted","Data":"1606bd5a1a890903ca58dc85010ba7ac05ad41ab01a25901ac6c8bc021d47a61"}
Sep 16 18:52:40.139504 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:40.139344    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79","Type":"ContainerStarted","Data":"f784e44725622ac8d8de000485c529712f677af48597aa4a6c3cbe3669777b72"}
Sep 16 18:52:42.150122 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:42.150092    2577 generic.go:356] "Generic (PLEG): container finished" podID="f4d184c4-8f57-48f3-bb49-4c80e4fd6b79" containerID="1606bd5a1a890903ca58dc85010ba7ac05ad41ab01a25901ac6c8bc021d47a61" exitCode=0
Sep 16 18:52:42.150470 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:42.150159    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79","Type":"ContainerDied","Data":"1606bd5a1a890903ca58dc85010ba7ac05ad41ab01a25901ac6c8bc021d47a61"}
Sep 16 18:52:43.864898 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:43.864867    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v8l5b"
Sep 16 18:52:43.881595 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:43.881254    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v8l5b"
Sep 16 18:52:44.184161 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:52:44.183896    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v8l5b"
Sep 16 18:53:11.313239 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:53:11.313160    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79","Type":"ContainerStarted","Data":"c096f69c44cc0dacae8251218a22041fc6d3d7b78b782ea944fdb699b5d94e7e"}
Sep 16 18:53:11.341665 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:53:11.341609    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=1.470060218 podStartE2EDuration="32.341589135s" totalImagesPullingTime="28.838369223s" totalInitContainerRuntime="2.033159694s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:53:11.338106513 +0000 UTC m=+748.720995880" watchObservedRunningTime="2026-09-16 18:53:11.341589135 +0000 UTC m=+748.724478503"
Sep 16 18:53:11.527786 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:53:11.527756    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:53:15.492254 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:53:15.492209    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v8l5b"]
Sep 16 18:53:15.503094 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:53:15.503053    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 18:53:15.503094 ip-10-0-128-247 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:53:15.503094 ip-10-0-128-247 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 18:53:15.503094 ip-10-0-128-247 kubenswrapper[2577]:  > containerID="f533e11ae0f603f6ffe43f3dff8a5289c3e48580ebb411e0fc971df2c999b520" cmd=["/bin/sleep","15"]
Sep 16 18:53:15.503419 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:53:15.503120    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v8l5b" message="" err=<
Sep 16 18:53:15.503419 ip-10-0-128-247 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:53:15.503419 ip-10-0-128-247 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 18:53:15.503419 ip-10-0-128-247 kubenswrapper[2577]:  >
Sep 16 18:53:15.503419 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:53:15.503137    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 18:53:15.503419 ip-10-0-128-247 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:53:15.503419 ip-10-0-128-247 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 18:53:15.503419 ip-10-0-128-247 kubenswrapper[2577]:  > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v8l5b" podUID="04f31f32-25e9-4624-aabd-49f148cf6ac6" containerName="main" containerID="cri-o://f533e11ae0f603f6ffe43f3dff8a5289c3e48580ebb411e0fc971df2c999b520"
Sep 16 18:53:15.503419 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:53:15.503181    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v8l5b" podUID="04f31f32-25e9-4624-aabd-49f148cf6ac6" containerName="main" containerID="cri-o://f533e11ae0f603f6ffe43f3dff8a5289c3e48580ebb411e0fc971df2c999b520" gracePeriod=60
Sep 16 18:53:15.847210 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:53:15.847188    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v8l5b"
Sep 16 18:53:15.937551 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:53:15.937498    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04f31f32-25e9-4624-aabd-49f148cf6ac6-tls-certs\") pod \"04f31f32-25e9-4624-aabd-49f148cf6ac6\" (UID: \"04f31f32-25e9-4624-aabd-49f148cf6ac6\") "
Sep 16 18:53:15.937710 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:53:15.937575    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04f31f32-25e9-4624-aabd-49f148cf6ac6-dshm\") pod \"04f31f32-25e9-4624-aabd-49f148cf6ac6\" (UID: \"04f31f32-25e9-4624-aabd-49f148cf6ac6\") "
Sep 16 18:53:15.937710 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:53:15.937599    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmnz7\" (UniqueName: \"kubernetes.io/projected/04f31f32-25e9-4624-aabd-49f148cf6ac6-kube-api-access-fmnz7\") pod \"04f31f32-25e9-4624-aabd-49f148cf6ac6\" (UID: \"04f31f32-25e9-4624-aabd-49f148cf6ac6\") "
Sep 16 18:53:15.937710 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:53:15.937618    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04f31f32-25e9-4624-aabd-49f148cf6ac6-tmp-dir\") pod \"04f31f32-25e9-4624-aabd-49f148cf6ac6\" (UID: \"04f31f32-25e9-4624-aabd-49f148cf6ac6\") "
Sep 16 18:53:15.937710 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:53:15.937641    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04f31f32-25e9-4624-aabd-49f148cf6ac6-model-cache\") pod \"04f31f32-25e9-4624-aabd-49f148cf6ac6\" (UID: \"04f31f32-25e9-4624-aabd-49f148cf6ac6\") "
Sep 16 18:53:15.937710 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:53:15.937672    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04f31f32-25e9-4624-aabd-49f148cf6ac6-home\") pod \"04f31f32-25e9-4624-aabd-49f148cf6ac6\" (UID: \"04f31f32-25e9-4624-aabd-49f148cf6ac6\") "
Sep 16 18:53:15.937960 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:53:15.937918    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04f31f32-25e9-4624-aabd-49f148cf6ac6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "04f31f32-25e9-4624-aabd-49f148cf6ac6" (UID: "04f31f32-25e9-4624-aabd-49f148cf6ac6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:53:15.937960 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:53:15.937935    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04f31f32-25e9-4624-aabd-49f148cf6ac6-model-cache" (OuterVolumeSpecName: "model-cache") pod "04f31f32-25e9-4624-aabd-49f148cf6ac6" (UID: "04f31f32-25e9-4624-aabd-49f148cf6ac6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:53:15.938066 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:53:15.938044    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04f31f32-25e9-4624-aabd-49f148cf6ac6-home" (OuterVolumeSpecName: "home") pod "04f31f32-25e9-4624-aabd-49f148cf6ac6" (UID: "04f31f32-25e9-4624-aabd-49f148cf6ac6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:53:15.940026 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:53:15.939995    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04f31f32-25e9-4624-aabd-49f148cf6ac6-kube-api-access-fmnz7" (OuterVolumeSpecName: "kube-api-access-fmnz7") pod "04f31f32-25e9-4624-aabd-49f148cf6ac6" (UID: "04f31f32-25e9-4624-aabd-49f148cf6ac6"). InnerVolumeSpecName "kube-api-access-fmnz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:53:15.940220 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:53:15.940197    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04f31f32-25e9-4624-aabd-49f148cf6ac6-dshm" (OuterVolumeSpecName: "dshm") pod "04f31f32-25e9-4624-aabd-49f148cf6ac6" (UID: "04f31f32-25e9-4624-aabd-49f148cf6ac6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:53:15.940220 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:53:15.940204    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04f31f32-25e9-4624-aabd-49f148cf6ac6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "04f31f32-25e9-4624-aabd-49f148cf6ac6" (UID: "04f31f32-25e9-4624-aabd-49f148cf6ac6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:53:16.039106 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:53:16.039023    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04f31f32-25e9-4624-aabd-49f148cf6ac6-home\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:53:16.039106 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:53:16.039061    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04f31f32-25e9-4624-aabd-49f148cf6ac6-tls-certs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:53:16.039106 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:53:16.039072    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04f31f32-25e9-4624-aabd-49f148cf6ac6-dshm\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:53:16.039106 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:53:16.039084    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmnz7\" (UniqueName: \"kubernetes.io/projected/04f31f32-25e9-4624-aabd-49f148cf6ac6-kube-api-access-fmnz7\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:53:16.039106 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:53:16.039113    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04f31f32-25e9-4624-aabd-49f148cf6ac6-tmp-dir\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:53:16.039365 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:53:16.039123    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04f31f32-25e9-4624-aabd-49f148cf6ac6-model-cache\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:53:16.339520 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:53:16.339436    2577 generic.go:356] "Generic (PLEG): container finished" podID="04f31f32-25e9-4624-aabd-49f148cf6ac6" containerID="f533e11ae0f603f6ffe43f3dff8a5289c3e48580ebb411e0fc971df2c999b520" exitCode=0
Sep 16 18:53:16.339520 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:53:16.339484    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v8l5b"
Sep 16 18:53:16.339520 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:53:16.339499    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v8l5b" event={"ID":"04f31f32-25e9-4624-aabd-49f148cf6ac6","Type":"ContainerDied","Data":"f533e11ae0f603f6ffe43f3dff8a5289c3e48580ebb411e0fc971df2c999b520"}
Sep 16 18:53:16.339887 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:53:16.339559    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v8l5b" event={"ID":"04f31f32-25e9-4624-aabd-49f148cf6ac6","Type":"ContainerDied","Data":"573df5d2a13a3a2768a30f1e192f097a90604c3ef7b31cf7ed1666c1e2a8e957"}
Sep 16 18:53:16.339887 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:53:16.339581    2577 scope.go:117] "RemoveContainer" containerID="f533e11ae0f603f6ffe43f3dff8a5289c3e48580ebb411e0fc971df2c999b520"
Sep 16 18:53:16.350126 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:53:16.350111    2577 scope.go:117] "RemoveContainer" containerID="f533e11ae0f603f6ffe43f3dff8a5289c3e48580ebb411e0fc971df2c999b520"
Sep 16 18:53:16.350382 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:53:16.350362    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f533e11ae0f603f6ffe43f3dff8a5289c3e48580ebb411e0fc971df2c999b520\": container with ID starting with f533e11ae0f603f6ffe43f3dff8a5289c3e48580ebb411e0fc971df2c999b520 not found: ID does not exist" containerID="f533e11ae0f603f6ffe43f3dff8a5289c3e48580ebb411e0fc971df2c999b520"
Sep 16 18:53:16.350454 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:53:16.350389    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f533e11ae0f603f6ffe43f3dff8a5289c3e48580ebb411e0fc971df2c999b520"} err="failed to get container status \"f533e11ae0f603f6ffe43f3dff8a5289c3e48580ebb411e0fc971df2c999b520\": rpc error: code = NotFound desc = could not find container \"f533e11ae0f603f6ffe43f3dff8a5289c3e48580ebb411e0fc971df2c999b520\": container with ID starting with f533e11ae0f603f6ffe43f3dff8a5289c3e48580ebb411e0fc971df2c999b520 not found: ID does not exist"
Sep 16 18:53:16.362650 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:53:16.362628    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v8l5b"]
Sep 16 18:53:16.368153 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:53:16.368132    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-v8l5b"]
Sep 16 18:53:17.154710 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:53:17.154676    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04f31f32-25e9-4624-aabd-49f148cf6ac6" path="/var/lib/kubelet/pods/04f31f32-25e9-4624-aabd-49f148cf6ac6/volumes"
Sep 16 18:53:19.527905 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:53:19.527874    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:53:19.529584 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:53:19.529518    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f4d184c4-8f57-48f3-bb49-4c80e4fd6b79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Sep 16 18:53:29.528178 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:53:29.528135    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f4d184c4-8f57-48f3-bb49-4c80e4fd6b79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Sep 16 18:53:39.528827 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:53:39.528783    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f4d184c4-8f57-48f3-bb49-4c80e4fd6b79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Sep 16 18:53:49.528053 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:53:49.528013    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f4d184c4-8f57-48f3-bb49-4c80e4fd6b79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Sep 16 18:53:59.528936 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:53:59.528892    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f4d184c4-8f57-48f3-bb49-4c80e4fd6b79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Sep 16 18:54:09.528300 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:54:09.528203    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f4d184c4-8f57-48f3-bb49-4c80e4fd6b79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Sep 16 18:54:19.528751 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:54:19.528705    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f4d184c4-8f57-48f3-bb49-4c80e4fd6b79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Sep 16 18:54:29.528501 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:54:29.528454    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f4d184c4-8f57-48f3-bb49-4c80e4fd6b79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Sep 16 18:54:39.537260 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:54:39.537226    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:54:39.544744 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:54:39.544722    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:54:50.787726 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:54:50.787695    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 18:55:05.809905 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:55:05.809858    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f4d184c4-8f57-48f3-bb49-4c80e4fd6b79" containerName="main" containerID="cri-o://c096f69c44cc0dacae8251218a22041fc6d3d7b78b782ea944fdb699b5d94e7e" gracePeriod=45
Sep 16 18:55:06.482806 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:55:06.482784    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:06.655157 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:55:06.655133    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pj9bs\" (UniqueName: \"kubernetes.io/projected/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-kube-api-access-pj9bs\") pod \"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79\" (UID: \"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79\") "
Sep 16 18:55:06.655298 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:55:06.655167    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-dshm\") pod \"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79\" (UID: \"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79\") "
Sep 16 18:55:06.655298 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:55:06.655192    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-kserve-provision-location\") pod \"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79\" (UID: \"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79\") "
Sep 16 18:55:06.655298 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:55:06.655234    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-tmp-dir\") pod \"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79\" (UID: \"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79\") "
Sep 16 18:55:06.655298 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:55:06.655254    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-model-cache\") pod \"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79\" (UID: \"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79\") "
Sep 16 18:55:06.655298 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:55:06.655294    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-cabundle-cert\") pod \"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79\" (UID: \"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79\") "
Sep 16 18:55:06.655592 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:55:06.655321    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-tls-certs\") pod \"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79\" (UID: \"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79\") "
Sep 16 18:55:06.655592 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:55:06.655361    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-home\") pod \"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79\" (UID: \"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79\") "
Sep 16 18:55:06.655709 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:55:06.655613    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-model-cache" (OuterVolumeSpecName: "model-cache") pod "f4d184c4-8f57-48f3-bb49-4c80e4fd6b79" (UID: "f4d184c4-8f57-48f3-bb49-4c80e4fd6b79"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:06.656116 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:55:06.656083    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f4d184c4-8f57-48f3-bb49-4c80e4fd6b79" (UID: "f4d184c4-8f57-48f3-bb49-4c80e4fd6b79"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:55:06.656234 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:55:06.656210    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-home" (OuterVolumeSpecName: "home") pod "f4d184c4-8f57-48f3-bb49-4c80e4fd6b79" (UID: "f4d184c4-8f57-48f3-bb49-4c80e4fd6b79"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:06.657555 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:55:06.657513    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-dshm" (OuterVolumeSpecName: "dshm") pod "f4d184c4-8f57-48f3-bb49-4c80e4fd6b79" (UID: "f4d184c4-8f57-48f3-bb49-4c80e4fd6b79"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:06.657555 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:55:06.657518    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-kube-api-access-pj9bs" (OuterVolumeSpecName: "kube-api-access-pj9bs") pod "f4d184c4-8f57-48f3-bb49-4c80e4fd6b79" (UID: "f4d184c4-8f57-48f3-bb49-4c80e4fd6b79"). InnerVolumeSpecName "kube-api-access-pj9bs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:55:06.658333 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:55:06.658309    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f4d184c4-8f57-48f3-bb49-4c80e4fd6b79" (UID: "f4d184c4-8f57-48f3-bb49-4c80e4fd6b79"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:55:06.667680 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:55:06.667655    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f4d184c4-8f57-48f3-bb49-4c80e4fd6b79" (UID: "f4d184c4-8f57-48f3-bb49-4c80e4fd6b79"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:06.676136 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:55:06.676112    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f4d184c4-8f57-48f3-bb49-4c80e4fd6b79" (UID: "f4d184c4-8f57-48f3-bb49-4c80e4fd6b79"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:06.756831 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:55:06.756804    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-tls-certs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:55:06.756831 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:55:06.756827    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-home\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:55:06.756964 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:55:06.756841    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj9bs\" (UniqueName: \"kubernetes.io/projected/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-kube-api-access-pj9bs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:55:06.756964 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:55:06.756862    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-dshm\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:55:06.756964 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:55:06.756875    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-kserve-provision-location\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:55:06.756964 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:55:06.756888    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-tmp-dir\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:55:06.756964 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:55:06.756902    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-model-cache\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:55:06.756964 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:55:06.756913    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79-cabundle-cert\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:55:06.843725 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:55:06.843695    2577 generic.go:356] "Generic (PLEG): container finished" podID="f4d184c4-8f57-48f3-bb49-4c80e4fd6b79" containerID="c096f69c44cc0dacae8251218a22041fc6d3d7b78b782ea944fdb699b5d94e7e" exitCode=0
Sep 16 18:55:06.844037 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:55:06.843742    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:55:06.844037 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:55:06.843742    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79","Type":"ContainerDied","Data":"c096f69c44cc0dacae8251218a22041fc6d3d7b78b782ea944fdb699b5d94e7e"}
Sep 16 18:55:06.844037 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:55:06.843877    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"f4d184c4-8f57-48f3-bb49-4c80e4fd6b79","Type":"ContainerDied","Data":"f784e44725622ac8d8de000485c529712f677af48597aa4a6c3cbe3669777b72"}
Sep 16 18:55:06.844037 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:55:06.843908    2577 scope.go:117] "RemoveContainer" containerID="c096f69c44cc0dacae8251218a22041fc6d3d7b78b782ea944fdb699b5d94e7e"
Sep 16 18:55:06.852815 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:55:06.852800    2577 scope.go:117] "RemoveContainer" containerID="1606bd5a1a890903ca58dc85010ba7ac05ad41ab01a25901ac6c8bc021d47a61"
Sep 16 18:55:06.860176 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:55:06.860159    2577 scope.go:117] "RemoveContainer" containerID="c096f69c44cc0dacae8251218a22041fc6d3d7b78b782ea944fdb699b5d94e7e"
Sep 16 18:55:06.860434 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:55:06.860416    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c096f69c44cc0dacae8251218a22041fc6d3d7b78b782ea944fdb699b5d94e7e\": container with ID starting with c096f69c44cc0dacae8251218a22041fc6d3d7b78b782ea944fdb699b5d94e7e not found: ID does not exist" containerID="c096f69c44cc0dacae8251218a22041fc6d3d7b78b782ea944fdb699b5d94e7e"
Sep 16 18:55:06.860486 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:55:06.860443    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c096f69c44cc0dacae8251218a22041fc6d3d7b78b782ea944fdb699b5d94e7e"} err="failed to get container status \"c096f69c44cc0dacae8251218a22041fc6d3d7b78b782ea944fdb699b5d94e7e\": rpc error: code = NotFound desc = could not find container \"c096f69c44cc0dacae8251218a22041fc6d3d7b78b782ea944fdb699b5d94e7e\": container with ID starting with c096f69c44cc0dacae8251218a22041fc6d3d7b78b782ea944fdb699b5d94e7e not found: ID does not exist"
Sep 16 18:55:06.860486 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:55:06.860461    2577 scope.go:117] "RemoveContainer" containerID="1606bd5a1a890903ca58dc85010ba7ac05ad41ab01a25901ac6c8bc021d47a61"
Sep 16 18:55:06.860701 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:55:06.860684    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1606bd5a1a890903ca58dc85010ba7ac05ad41ab01a25901ac6c8bc021d47a61\": container with ID starting with 1606bd5a1a890903ca58dc85010ba7ac05ad41ab01a25901ac6c8bc021d47a61 not found: ID does not exist" containerID="1606bd5a1a890903ca58dc85010ba7ac05ad41ab01a25901ac6c8bc021d47a61"
Sep 16 18:55:06.860748 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:55:06.860707    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1606bd5a1a890903ca58dc85010ba7ac05ad41ab01a25901ac6c8bc021d47a61"} err="failed to get container status \"1606bd5a1a890903ca58dc85010ba7ac05ad41ab01a25901ac6c8bc021d47a61\": rpc error: code = NotFound desc = could not find container \"1606bd5a1a890903ca58dc85010ba7ac05ad41ab01a25901ac6c8bc021d47a61\": container with ID starting with 1606bd5a1a890903ca58dc85010ba7ac05ad41ab01a25901ac6c8bc021d47a61 not found: ID does not exist"
Sep 16 18:55:06.870129 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:55:06.870105    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 18:55:06.873806 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:55:06.873787    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 18:55:07.154905 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:55:07.154873    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4d184c4-8f57-48f3-bb49-4c80e4fd6b79" path="/var/lib/kubelet/pods/f4d184c4-8f57-48f3-bb49-4c80e4fd6b79/volumes"
Sep 16 18:56:15.243245 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:15.243207    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll"]
Sep 16 18:56:15.245625 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:15.243722    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4d184c4-8f57-48f3-bb49-4c80e4fd6b79" containerName="storage-initializer"
Sep 16 18:56:15.245625 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:15.243738    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4d184c4-8f57-48f3-bb49-4c80e4fd6b79" containerName="storage-initializer"
Sep 16 18:56:15.245625 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:15.243753    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04f31f32-25e9-4624-aabd-49f148cf6ac6" containerName="main"
Sep 16 18:56:15.245625 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:15.243762    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="04f31f32-25e9-4624-aabd-49f148cf6ac6" containerName="main"
Sep 16 18:56:15.245625 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:15.243771    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4d184c4-8f57-48f3-bb49-4c80e4fd6b79" containerName="main"
Sep 16 18:56:15.245625 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:15.243810    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4d184c4-8f57-48f3-bb49-4c80e4fd6b79" containerName="main"
Sep 16 18:56:15.245625 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:15.243895    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="04f31f32-25e9-4624-aabd-49f148cf6ac6" containerName="main"
Sep 16 18:56:15.245625 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:15.243917    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4d184c4-8f57-48f3-bb49-4c80e4fd6b79" containerName="main"
Sep 16 18:56:15.247119 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:15.247103    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll"
Sep 16 18:56:15.251824 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:15.251520    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Sep 16 18:56:15.251824 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:15.251567    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Sep 16 18:56:15.251824 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:15.251703    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-c8gl5\""
Sep 16 18:56:15.252340 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:15.252318    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Sep 16 18:56:15.255358 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:15.255332    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll"]
Sep 16 18:56:15.366469 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:15.366441    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa93a6a3-618e-49b8-a556-7722824960b1-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll\" (UID: \"aa93a6a3-618e-49b8-a556-7722824960b1\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll"
Sep 16 18:56:15.366601 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:15.366471    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa93a6a3-618e-49b8-a556-7722824960b1-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll\" (UID: \"aa93a6a3-618e-49b8-a556-7722824960b1\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll"
Sep 16 18:56:15.366601 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:15.366498    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa93a6a3-618e-49b8-a556-7722824960b1-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll\" (UID: \"aa93a6a3-618e-49b8-a556-7722824960b1\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll"
Sep 16 18:56:15.366692 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:15.366608    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa93a6a3-618e-49b8-a556-7722824960b1-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll\" (UID: \"aa93a6a3-618e-49b8-a556-7722824960b1\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll"
Sep 16 18:56:15.366692 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:15.366627    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa93a6a3-618e-49b8-a556-7722824960b1-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll\" (UID: \"aa93a6a3-618e-49b8-a556-7722824960b1\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll"
Sep 16 18:56:15.366692 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:15.366647    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9nxt\" (UniqueName: \"kubernetes.io/projected/aa93a6a3-618e-49b8-a556-7722824960b1-kube-api-access-d9nxt\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll\" (UID: \"aa93a6a3-618e-49b8-a556-7722824960b1\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll"
Sep 16 18:56:15.467223 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:15.467197    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa93a6a3-618e-49b8-a556-7722824960b1-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll\" (UID: \"aa93a6a3-618e-49b8-a556-7722824960b1\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll"
Sep 16 18:56:15.467324 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:15.467236    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa93a6a3-618e-49b8-a556-7722824960b1-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll\" (UID: \"aa93a6a3-618e-49b8-a556-7722824960b1\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll"
Sep 16 18:56:15.467324 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:15.467253    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa93a6a3-618e-49b8-a556-7722824960b1-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll\" (UID: \"aa93a6a3-618e-49b8-a556-7722824960b1\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll"
Sep 16 18:56:15.467324 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:15.467273    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9nxt\" (UniqueName: \"kubernetes.io/projected/aa93a6a3-618e-49b8-a556-7722824960b1-kube-api-access-d9nxt\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll\" (UID: \"aa93a6a3-618e-49b8-a556-7722824960b1\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll"
Sep 16 18:56:15.467479 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:15.467334    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa93a6a3-618e-49b8-a556-7722824960b1-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll\" (UID: \"aa93a6a3-618e-49b8-a556-7722824960b1\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll"
Sep 16 18:56:15.467479 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:15.467351    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa93a6a3-618e-49b8-a556-7722824960b1-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll\" (UID: \"aa93a6a3-618e-49b8-a556-7722824960b1\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll"
Sep 16 18:56:15.467621 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:15.467578    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa93a6a3-618e-49b8-a556-7722824960b1-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll\" (UID: \"aa93a6a3-618e-49b8-a556-7722824960b1\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll"
Sep 16 18:56:15.467679 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:15.467640    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa93a6a3-618e-49b8-a556-7722824960b1-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll\" (UID: \"aa93a6a3-618e-49b8-a556-7722824960b1\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll"
Sep 16 18:56:15.467735 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:15.467717    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa93a6a3-618e-49b8-a556-7722824960b1-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll\" (UID: \"aa93a6a3-618e-49b8-a556-7722824960b1\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll"
Sep 16 18:56:15.469658 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:15.469636    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa93a6a3-618e-49b8-a556-7722824960b1-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll\" (UID: \"aa93a6a3-618e-49b8-a556-7722824960b1\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll"
Sep 16 18:56:15.470058 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:15.470036    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa93a6a3-618e-49b8-a556-7722824960b1-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll\" (UID: \"aa93a6a3-618e-49b8-a556-7722824960b1\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll"
Sep 16 18:56:15.477799 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:15.477780    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9nxt\" (UniqueName: \"kubernetes.io/projected/aa93a6a3-618e-49b8-a556-7722824960b1-kube-api-access-d9nxt\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll\" (UID: \"aa93a6a3-618e-49b8-a556-7722824960b1\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll"
Sep 16 18:56:15.560053 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:15.560005    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll"
Sep 16 18:56:15.841169 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:15.841100    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll"]
Sep 16 18:56:15.844509 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:56:15.844482    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa93a6a3_618e_49b8_a556_7722824960b1.slice/crio-8555a72c6f0a214a9e014e2ada7ff50c8b76fdb3f2b5269a2879eab940d96c84 WatchSource:0}: Error finding container 8555a72c6f0a214a9e014e2ada7ff50c8b76fdb3f2b5269a2879eab940d96c84: Status 404 returned error can't find the container with id 8555a72c6f0a214a9e014e2ada7ff50c8b76fdb3f2b5269a2879eab940d96c84
Sep 16 18:56:16.156971 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:16.156922    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll" event={"ID":"aa93a6a3-618e-49b8-a556-7722824960b1","Type":"ContainerStarted","Data":"bfe214e833fab5450ff759f28b581184b84d346b5dcd01b9603aa988ba8bd770"}
Sep 16 18:56:16.157145 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:16.156984    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll" event={"ID":"aa93a6a3-618e-49b8-a556-7722824960b1","Type":"ContainerStarted","Data":"8555a72c6f0a214a9e014e2ada7ff50c8b76fdb3f2b5269a2879eab940d96c84"}
Sep 16 18:56:16.181442 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:16.181400    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll" podStartSLOduration=1.181385588 podStartE2EDuration="1.181385588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:56:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:56:16.180986449 +0000 UTC m=+933.563875815" watchObservedRunningTime="2026-09-16 18:56:16.181385588 +0000 UTC m=+933.564274953"
Sep 16 18:56:16.560333 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:16.560253    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll"
Sep 16 18:56:22.156816 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:22.156779    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll"]
Sep 16 18:56:22.166156 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:56:22.166126    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 18:56:22.166156 ip-10-0-128-247 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:56:22.166156 ip-10-0-128-247 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 18:56:22.166156 ip-10-0-128-247 kubenswrapper[2577]:  > containerID="bfe214e833fab5450ff759f28b581184b84d346b5dcd01b9603aa988ba8bd770" cmd=["/bin/sleep","15"]
Sep 16 18:56:22.166418 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:22.166171    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll" message="" err=<
Sep 16 18:56:22.166418 ip-10-0-128-247 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:56:22.166418 ip-10-0-128-247 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 18:56:22.166418 ip-10-0-128-247 kubenswrapper[2577]:  >
Sep 16 18:56:22.166418 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:56:22.166184    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 18:56:22.166418 ip-10-0-128-247 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:56:22.166418 ip-10-0-128-247 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 18:56:22.166418 ip-10-0-128-247 kubenswrapper[2577]:  > pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll" podUID="aa93a6a3-618e-49b8-a556-7722824960b1" containerName="main" containerID="cri-o://bfe214e833fab5450ff759f28b581184b84d346b5dcd01b9603aa988ba8bd770"
Sep 16 18:56:22.166418 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:22.166217    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll" podUID="aa93a6a3-618e-49b8-a556-7722824960b1" containerName="main" containerID="cri-o://bfe214e833fab5450ff759f28b581184b84d346b5dcd01b9603aa988ba8bd770" gracePeriod=60
Sep 16 18:56:22.405718 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:22.405692    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll"
Sep 16 18:56:22.522397 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:22.522336    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa93a6a3-618e-49b8-a556-7722824960b1-model-cache\") pod \"aa93a6a3-618e-49b8-a556-7722824960b1\" (UID: \"aa93a6a3-618e-49b8-a556-7722824960b1\") "
Sep 16 18:56:22.522397 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:22.522369    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa93a6a3-618e-49b8-a556-7722824960b1-home\") pod \"aa93a6a3-618e-49b8-a556-7722824960b1\" (UID: \"aa93a6a3-618e-49b8-a556-7722824960b1\") "
Sep 16 18:56:22.522397 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:22.522399    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa93a6a3-618e-49b8-a556-7722824960b1-tmp-dir\") pod \"aa93a6a3-618e-49b8-a556-7722824960b1\" (UID: \"aa93a6a3-618e-49b8-a556-7722824960b1\") "
Sep 16 18:56:22.522657 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:22.522425    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa93a6a3-618e-49b8-a556-7722824960b1-dshm\") pod \"aa93a6a3-618e-49b8-a556-7722824960b1\" (UID: \"aa93a6a3-618e-49b8-a556-7722824960b1\") "
Sep 16 18:56:22.522657 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:22.522444    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa93a6a3-618e-49b8-a556-7722824960b1-tls-certs\") pod \"aa93a6a3-618e-49b8-a556-7722824960b1\" (UID: \"aa93a6a3-618e-49b8-a556-7722824960b1\") "
Sep 16 18:56:22.522657 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:22.522512    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9nxt\" (UniqueName: \"kubernetes.io/projected/aa93a6a3-618e-49b8-a556-7722824960b1-kube-api-access-d9nxt\") pod \"aa93a6a3-618e-49b8-a556-7722824960b1\" (UID: \"aa93a6a3-618e-49b8-a556-7722824960b1\") "
Sep 16 18:56:22.522795 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:22.522655    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa93a6a3-618e-49b8-a556-7722824960b1-home" (OuterVolumeSpecName: "home") pod "aa93a6a3-618e-49b8-a556-7722824960b1" (UID: "aa93a6a3-618e-49b8-a556-7722824960b1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:22.522795 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:22.522665    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa93a6a3-618e-49b8-a556-7722824960b1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aa93a6a3-618e-49b8-a556-7722824960b1" (UID: "aa93a6a3-618e-49b8-a556-7722824960b1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:22.522795 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:22.522678    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa93a6a3-618e-49b8-a556-7722824960b1-model-cache" (OuterVolumeSpecName: "model-cache") pod "aa93a6a3-618e-49b8-a556-7722824960b1" (UID: "aa93a6a3-618e-49b8-a556-7722824960b1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:22.522944 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:22.522844    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa93a6a3-618e-49b8-a556-7722824960b1-model-cache\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:56:22.522944 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:22.522858    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa93a6a3-618e-49b8-a556-7722824960b1-home\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:56:22.522944 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:22.522868    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa93a6a3-618e-49b8-a556-7722824960b1-tmp-dir\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:56:22.524687 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:22.524665    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa93a6a3-618e-49b8-a556-7722824960b1-dshm" (OuterVolumeSpecName: "dshm") pod "aa93a6a3-618e-49b8-a556-7722824960b1" (UID: "aa93a6a3-618e-49b8-a556-7722824960b1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:22.524798 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:22.524774    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa93a6a3-618e-49b8-a556-7722824960b1-kube-api-access-d9nxt" (OuterVolumeSpecName: "kube-api-access-d9nxt") pod "aa93a6a3-618e-49b8-a556-7722824960b1" (UID: "aa93a6a3-618e-49b8-a556-7722824960b1"). InnerVolumeSpecName "kube-api-access-d9nxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:56:22.524872 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:22.524856    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa93a6a3-618e-49b8-a556-7722824960b1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aa93a6a3-618e-49b8-a556-7722824960b1" (UID: "aa93a6a3-618e-49b8-a556-7722824960b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:56:22.624016 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:22.623994    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa93a6a3-618e-49b8-a556-7722824960b1-dshm\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:56:22.624016 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:22.624018    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa93a6a3-618e-49b8-a556-7722824960b1-tls-certs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:56:22.624199 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:22.624031    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9nxt\" (UniqueName: \"kubernetes.io/projected/aa93a6a3-618e-49b8-a556-7722824960b1-kube-api-access-d9nxt\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:56:23.186222 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:23.186191    2577 generic.go:356] "Generic (PLEG): container finished" podID="aa93a6a3-618e-49b8-a556-7722824960b1" containerID="bfe214e833fab5450ff759f28b581184b84d346b5dcd01b9603aa988ba8bd770" exitCode=0
Sep 16 18:56:23.186626 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:23.186235    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll"
Sep 16 18:56:23.186626 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:23.186264    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll" event={"ID":"aa93a6a3-618e-49b8-a556-7722824960b1","Type":"ContainerDied","Data":"bfe214e833fab5450ff759f28b581184b84d346b5dcd01b9603aa988ba8bd770"}
Sep 16 18:56:23.186626 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:23.186311    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll" event={"ID":"aa93a6a3-618e-49b8-a556-7722824960b1","Type":"ContainerDied","Data":"8555a72c6f0a214a9e014e2ada7ff50c8b76fdb3f2b5269a2879eab940d96c84"}
Sep 16 18:56:23.186626 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:23.186333    2577 scope.go:117] "RemoveContainer" containerID="bfe214e833fab5450ff759f28b581184b84d346b5dcd01b9603aa988ba8bd770"
Sep 16 18:56:23.195366 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:23.195352    2577 scope.go:117] "RemoveContainer" containerID="bfe214e833fab5450ff759f28b581184b84d346b5dcd01b9603aa988ba8bd770"
Sep 16 18:56:23.195674 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:56:23.195643    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfe214e833fab5450ff759f28b581184b84d346b5dcd01b9603aa988ba8bd770\": container with ID starting with bfe214e833fab5450ff759f28b581184b84d346b5dcd01b9603aa988ba8bd770 not found: ID does not exist" containerID="bfe214e833fab5450ff759f28b581184b84d346b5dcd01b9603aa988ba8bd770"
Sep 16 18:56:23.195753 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:23.195673    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfe214e833fab5450ff759f28b581184b84d346b5dcd01b9603aa988ba8bd770"} err="failed to get container status \"bfe214e833fab5450ff759f28b581184b84d346b5dcd01b9603aa988ba8bd770\": rpc error: code = NotFound desc = could not find container \"bfe214e833fab5450ff759f28b581184b84d346b5dcd01b9603aa988ba8bd770\": container with ID starting with bfe214e833fab5450ff759f28b581184b84d346b5dcd01b9603aa988ba8bd770 not found: ID does not exist"
Sep 16 18:56:23.205886 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:23.205863    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll"]
Sep 16 18:56:23.213238 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:23.213214    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-n57ll"]
Sep 16 18:56:25.159789 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:25.156189    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa93a6a3-618e-49b8-a556-7722824960b1" path="/var/lib/kubelet/pods/aa93a6a3-618e-49b8-a556-7722824960b1/volumes"
Sep 16 18:56:42.639847 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:42.639814    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck"]
Sep 16 18:56:42.640360 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:42.640222    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa93a6a3-618e-49b8-a556-7722824960b1" containerName="main"
Sep 16 18:56:42.640360 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:42.640240    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa93a6a3-618e-49b8-a556-7722824960b1" containerName="main"
Sep 16 18:56:42.640360 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:42.640333    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa93a6a3-618e-49b8-a556-7722824960b1" containerName="main"
Sep 16 18:56:42.643155 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:42.643135    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck"
Sep 16 18:56:42.645938 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:42.645912    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 16 18:56:42.647221 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:42.647199    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 16 18:56:42.647356 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:42.647334    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Sep 16 18:56:42.647408 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:42.647341    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-7s2cc\""
Sep 16 18:56:42.654687 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:42.654663    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck"]
Sep 16 18:56:42.660951 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:42.660918    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ad745ba-be5c-45da-8101-177a3fb7edda-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck\" (UID: \"4ad745ba-be5c-45da-8101-177a3fb7edda\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck"
Sep 16 18:56:42.661039 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:42.660962    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ad745ba-be5c-45da-8101-177a3fb7edda-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck\" (UID: \"4ad745ba-be5c-45da-8101-177a3fb7edda\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck"
Sep 16 18:56:42.661101 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:42.661080    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk77c\" (UniqueName: \"kubernetes.io/projected/4ad745ba-be5c-45da-8101-177a3fb7edda-kube-api-access-xk77c\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck\" (UID: \"4ad745ba-be5c-45da-8101-177a3fb7edda\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck"
Sep 16 18:56:42.661138 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:42.661121    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ad745ba-be5c-45da-8101-177a3fb7edda-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck\" (UID: \"4ad745ba-be5c-45da-8101-177a3fb7edda\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck"
Sep 16 18:56:42.661206 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:42.661188    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ad745ba-be5c-45da-8101-177a3fb7edda-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck\" (UID: \"4ad745ba-be5c-45da-8101-177a3fb7edda\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck"
Sep 16 18:56:42.661244 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:42.661227    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ad745ba-be5c-45da-8101-177a3fb7edda-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck\" (UID: \"4ad745ba-be5c-45da-8101-177a3fb7edda\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck"
Sep 16 18:56:42.762101 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:42.762077    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk77c\" (UniqueName: \"kubernetes.io/projected/4ad745ba-be5c-45da-8101-177a3fb7edda-kube-api-access-xk77c\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck\" (UID: \"4ad745ba-be5c-45da-8101-177a3fb7edda\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck"
Sep 16 18:56:42.762220 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:42.762117    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ad745ba-be5c-45da-8101-177a3fb7edda-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck\" (UID: \"4ad745ba-be5c-45da-8101-177a3fb7edda\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck"
Sep 16 18:56:42.762220 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:42.762151    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ad745ba-be5c-45da-8101-177a3fb7edda-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck\" (UID: \"4ad745ba-be5c-45da-8101-177a3fb7edda\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck"
Sep 16 18:56:42.762220 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:42.762175    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ad745ba-be5c-45da-8101-177a3fb7edda-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck\" (UID: \"4ad745ba-be5c-45da-8101-177a3fb7edda\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck"
Sep 16 18:56:42.762220 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:42.762209    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ad745ba-be5c-45da-8101-177a3fb7edda-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck\" (UID: \"4ad745ba-be5c-45da-8101-177a3fb7edda\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck"
Sep 16 18:56:42.762415 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:42.762247    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ad745ba-be5c-45da-8101-177a3fb7edda-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck\" (UID: \"4ad745ba-be5c-45da-8101-177a3fb7edda\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck"
Sep 16 18:56:42.762558 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:42.762510    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ad745ba-be5c-45da-8101-177a3fb7edda-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck\" (UID: \"4ad745ba-be5c-45da-8101-177a3fb7edda\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck"
Sep 16 18:56:42.762634 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:42.762570    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ad745ba-be5c-45da-8101-177a3fb7edda-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck\" (UID: \"4ad745ba-be5c-45da-8101-177a3fb7edda\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck"
Sep 16 18:56:42.762700 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:42.762681    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ad745ba-be5c-45da-8101-177a3fb7edda-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck\" (UID: \"4ad745ba-be5c-45da-8101-177a3fb7edda\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck"
Sep 16 18:56:42.764693 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:42.764673    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ad745ba-be5c-45da-8101-177a3fb7edda-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck\" (UID: \"4ad745ba-be5c-45da-8101-177a3fb7edda\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck"
Sep 16 18:56:42.764841 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:42.764824    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ad745ba-be5c-45da-8101-177a3fb7edda-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck\" (UID: \"4ad745ba-be5c-45da-8101-177a3fb7edda\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck"
Sep 16 18:56:42.771259 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:42.771236    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk77c\" (UniqueName: \"kubernetes.io/projected/4ad745ba-be5c-45da-8101-177a3fb7edda-kube-api-access-xk77c\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck\" (UID: \"4ad745ba-be5c-45da-8101-177a3fb7edda\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck"
Sep 16 18:56:42.955072 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:42.954987    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck"
Sep 16 18:56:43.229087 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:43.229065    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck"]
Sep 16 18:56:43.231113 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:56:43.231084    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ad745ba_be5c_45da_8101_177a3fb7edda.slice/crio-f156595a559f1c6ab8d459af08f3d751558cfaaf159e0fe17bb40e75bf9321fc WatchSource:0}: Error finding container f156595a559f1c6ab8d459af08f3d751558cfaaf159e0fe17bb40e75bf9321fc: Status 404 returned error can't find the container with id f156595a559f1c6ab8d459af08f3d751558cfaaf159e0fe17bb40e75bf9321fc
Sep 16 18:56:43.268015 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:43.267989    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck" event={"ID":"4ad745ba-be5c-45da-8101-177a3fb7edda","Type":"ContainerStarted","Data":"f156595a559f1c6ab8d459af08f3d751558cfaaf159e0fe17bb40e75bf9321fc"}
Sep 16 18:56:44.273389 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:44.273356    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck" event={"ID":"4ad745ba-be5c-45da-8101-177a3fb7edda","Type":"ContainerStarted","Data":"cd403f9113ca3f969e9b57deed37085ca0794303bb3a8acde7c322f0008b70cb"}
Sep 16 18:56:44.293063 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:44.293018    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck" podStartSLOduration=2.293002424 podStartE2EDuration="2.293002424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:56:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:56:44.291886752 +0000 UTC m=+961.674776117" watchObservedRunningTime="2026-09-16 18:56:44.293002424 +0000 UTC m=+961.675891788"
Sep 16 18:56:44.955837 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:44.955803    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck"
Sep 16 18:56:45.823031 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:45.823003    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck"]
Sep 16 18:56:46.288095 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:56:46.288066    2577 secret.go:189] Couldn't get secret e2e-test-gateway-section-name-propagation-94799d44/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs: secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Sep 16 18:56:46.288340 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:56:46.288132    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ad745ba-be5c-45da-8101-177a3fb7edda-tls-certs podName:4ad745ba-be5c-45da-8101-177a3fb7edda nodeName:}" failed. No retries permitted until 2026-09-16 18:56:46.788117299 +0000 UTC m=+964.171006641 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/4ad745ba-be5c-45da-8101-177a3fb7edda-tls-certs") pod "gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck" (UID: "4ad745ba-be5c-45da-8101-177a3fb7edda") : secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Sep 16 18:56:46.792983 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:56:46.792954    2577 secret.go:189] Couldn't get secret e2e-test-gateway-section-name-propagation-94799d44/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs: secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Sep 16 18:56:46.793146 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:56:46.793020    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ad745ba-be5c-45da-8101-177a3fb7edda-tls-certs podName:4ad745ba-be5c-45da-8101-177a3fb7edda nodeName:}" failed. No retries permitted until 2026-09-16 18:56:47.793004507 +0000 UTC m=+965.175893875 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/4ad745ba-be5c-45da-8101-177a3fb7edda-tls-certs") pod "gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck" (UID: "4ad745ba-be5c-45da-8101-177a3fb7edda") : secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Sep 16 18:56:47.295502 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:56:47.295459    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 18:56:47.295502 ip-10-0-128-247 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:56:47.295502 ip-10-0-128-247 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 18:56:47.295502 ip-10-0-128-247 kubenswrapper[2577]:  > containerID="cd403f9113ca3f969e9b57deed37085ca0794303bb3a8acde7c322f0008b70cb" cmd=["/bin/sleep","15"]
Sep 16 18:56:47.296112 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:47.295513    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck" message="" err=<
Sep 16 18:56:47.296112 ip-10-0-128-247 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:56:47.296112 ip-10-0-128-247 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 18:56:47.296112 ip-10-0-128-247 kubenswrapper[2577]:  >
Sep 16 18:56:47.296112 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:56:47.295545    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 18:56:47.296112 ip-10-0-128-247 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:56:47.296112 ip-10-0-128-247 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 18:56:47.296112 ip-10-0-128-247 kubenswrapper[2577]:  > pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck" podUID="4ad745ba-be5c-45da-8101-177a3fb7edda" containerName="main" containerID="cri-o://cd403f9113ca3f969e9b57deed37085ca0794303bb3a8acde7c322f0008b70cb"
Sep 16 18:56:47.296112 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:47.295591    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck" podUID="4ad745ba-be5c-45da-8101-177a3fb7edda" containerName="main" containerID="cri-o://cd403f9113ca3f969e9b57deed37085ca0794303bb3a8acde7c322f0008b70cb" gracePeriod=60
Sep 16 18:56:47.534497 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:47.534471    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck"
Sep 16 18:56:47.597581 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:47.597507    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ad745ba-be5c-45da-8101-177a3fb7edda-dshm\") pod \"4ad745ba-be5c-45da-8101-177a3fb7edda\" (UID: \"4ad745ba-be5c-45da-8101-177a3fb7edda\") "
Sep 16 18:56:47.597717 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:47.597587    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ad745ba-be5c-45da-8101-177a3fb7edda-model-cache\") pod \"4ad745ba-be5c-45da-8101-177a3fb7edda\" (UID: \"4ad745ba-be5c-45da-8101-177a3fb7edda\") "
Sep 16 18:56:47.597717 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:47.597608    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xk77c\" (UniqueName: \"kubernetes.io/projected/4ad745ba-be5c-45da-8101-177a3fb7edda-kube-api-access-xk77c\") pod \"4ad745ba-be5c-45da-8101-177a3fb7edda\" (UID: \"4ad745ba-be5c-45da-8101-177a3fb7edda\") "
Sep 16 18:56:47.597717 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:47.597632    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ad745ba-be5c-45da-8101-177a3fb7edda-home\") pod \"4ad745ba-be5c-45da-8101-177a3fb7edda\" (UID: \"4ad745ba-be5c-45da-8101-177a3fb7edda\") "
Sep 16 18:56:47.597717 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:47.597681    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ad745ba-be5c-45da-8101-177a3fb7edda-tls-certs\") pod \"4ad745ba-be5c-45da-8101-177a3fb7edda\" (UID: \"4ad745ba-be5c-45da-8101-177a3fb7edda\") "
Sep 16 18:56:47.597717 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:47.597714    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ad745ba-be5c-45da-8101-177a3fb7edda-tmp-dir\") pod \"4ad745ba-be5c-45da-8101-177a3fb7edda\" (UID: \"4ad745ba-be5c-45da-8101-177a3fb7edda\") "
Sep 16 18:56:47.597937 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:47.597874    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad745ba-be5c-45da-8101-177a3fb7edda-model-cache" (OuterVolumeSpecName: "model-cache") pod "4ad745ba-be5c-45da-8101-177a3fb7edda" (UID: "4ad745ba-be5c-45da-8101-177a3fb7edda"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:47.597937 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:47.597891    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad745ba-be5c-45da-8101-177a3fb7edda-home" (OuterVolumeSpecName: "home") pod "4ad745ba-be5c-45da-8101-177a3fb7edda" (UID: "4ad745ba-be5c-45da-8101-177a3fb7edda"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:47.598044 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:47.597979    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ad745ba-be5c-45da-8101-177a3fb7edda-model-cache\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:56:47.598044 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:47.597996    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ad745ba-be5c-45da-8101-177a3fb7edda-home\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:56:47.598136 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:47.598084    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad745ba-be5c-45da-8101-177a3fb7edda-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4ad745ba-be5c-45da-8101-177a3fb7edda" (UID: "4ad745ba-be5c-45da-8101-177a3fb7edda"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:47.600054 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:47.600028    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ad745ba-be5c-45da-8101-177a3fb7edda-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4ad745ba-be5c-45da-8101-177a3fb7edda" (UID: "4ad745ba-be5c-45da-8101-177a3fb7edda"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:56:47.600329 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:47.600311    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad745ba-be5c-45da-8101-177a3fb7edda-dshm" (OuterVolumeSpecName: "dshm") pod "4ad745ba-be5c-45da-8101-177a3fb7edda" (UID: "4ad745ba-be5c-45da-8101-177a3fb7edda"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:47.600329 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:47.600315    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad745ba-be5c-45da-8101-177a3fb7edda-kube-api-access-xk77c" (OuterVolumeSpecName: "kube-api-access-xk77c") pod "4ad745ba-be5c-45da-8101-177a3fb7edda" (UID: "4ad745ba-be5c-45da-8101-177a3fb7edda"). InnerVolumeSpecName "kube-api-access-xk77c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:56:47.699145 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:47.699113    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ad745ba-be5c-45da-8101-177a3fb7edda-dshm\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:56:47.699145 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:47.699138    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk77c\" (UniqueName: \"kubernetes.io/projected/4ad745ba-be5c-45da-8101-177a3fb7edda-kube-api-access-xk77c\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:56:47.699145 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:47.699151    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ad745ba-be5c-45da-8101-177a3fb7edda-tls-certs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:56:47.699355 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:47.699160    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ad745ba-be5c-45da-8101-177a3fb7edda-tmp-dir\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:56:48.291735 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:48.291698    2577 generic.go:356] "Generic (PLEG): container finished" podID="4ad745ba-be5c-45da-8101-177a3fb7edda" containerID="cd403f9113ca3f969e9b57deed37085ca0794303bb3a8acde7c322f0008b70cb" exitCode=0
Sep 16 18:56:48.291937 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:48.291750    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck"
Sep 16 18:56:48.291937 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:48.291801    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck" event={"ID":"4ad745ba-be5c-45da-8101-177a3fb7edda","Type":"ContainerDied","Data":"cd403f9113ca3f969e9b57deed37085ca0794303bb3a8acde7c322f0008b70cb"}
Sep 16 18:56:48.291937 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:48.291875    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck" event={"ID":"4ad745ba-be5c-45da-8101-177a3fb7edda","Type":"ContainerDied","Data":"f156595a559f1c6ab8d459af08f3d751558cfaaf159e0fe17bb40e75bf9321fc"}
Sep 16 18:56:48.291937 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:48.291907    2577 scope.go:117] "RemoveContainer" containerID="cd403f9113ca3f969e9b57deed37085ca0794303bb3a8acde7c322f0008b70cb"
Sep 16 18:56:48.302088 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:48.301923    2577 scope.go:117] "RemoveContainer" containerID="cd403f9113ca3f969e9b57deed37085ca0794303bb3a8acde7c322f0008b70cb"
Sep 16 18:56:48.302334 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:56:48.302245    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd403f9113ca3f969e9b57deed37085ca0794303bb3a8acde7c322f0008b70cb\": container with ID starting with cd403f9113ca3f969e9b57deed37085ca0794303bb3a8acde7c322f0008b70cb not found: ID does not exist" containerID="cd403f9113ca3f969e9b57deed37085ca0794303bb3a8acde7c322f0008b70cb"
Sep 16 18:56:48.302334 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:48.302271    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd403f9113ca3f969e9b57deed37085ca0794303bb3a8acde7c322f0008b70cb"} err="failed to get container status \"cd403f9113ca3f969e9b57deed37085ca0794303bb3a8acde7c322f0008b70cb\": rpc error: code = NotFound desc = could not find container \"cd403f9113ca3f969e9b57deed37085ca0794303bb3a8acde7c322f0008b70cb\": container with ID starting with cd403f9113ca3f969e9b57deed37085ca0794303bb3a8acde7c322f0008b70cb not found: ID does not exist"
Sep 16 18:56:48.316707 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:48.316680    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck"]
Sep 16 18:56:48.324163 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:48.324139    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfkbvck"]
Sep 16 18:56:49.160543 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:56:49.160502    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ad745ba-be5c-45da-8101-177a3fb7edda" path="/var/lib/kubelet/pods/4ad745ba-be5c-45da-8101-177a3fb7edda/volumes"
Sep 16 18:57:00.369557 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:00.369514    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp"]
Sep 16 18:57:00.369899 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:00.369866    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ad745ba-be5c-45da-8101-177a3fb7edda" containerName="main"
Sep 16 18:57:00.369899 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:00.369880    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ad745ba-be5c-45da-8101-177a3fb7edda" containerName="main"
Sep 16 18:57:00.369980 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:00.369955    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ad745ba-be5c-45da-8101-177a3fb7edda" containerName="main"
Sep 16 18:57:00.374242 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:00.374227    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp"
Sep 16 18:57:00.377225 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:00.377201    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 16 18:57:00.377373 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:00.377272    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 16 18:57:00.377373 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:00.377312    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 16 18:57:00.377497 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:00.377458    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-f4rfs\""
Sep 16 18:57:00.377497 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:00.377483    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Sep 16 18:57:00.378468 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:00.378451    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:57:00.388695 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:00.388676    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp"]
Sep 16 18:57:00.393817 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:00.393794    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66344676-0c38-4f6e-bc8a-bb19b064f1f9-tls-certs\") pod \"auth-enabled-test-kserve-5df9cf6f97-l7scp\" (UID: \"66344676-0c38-4f6e-bc8a-bb19b064f1f9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp"
Sep 16 18:57:00.393916 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:00.393836    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66344676-0c38-4f6e-bc8a-bb19b064f1f9-kserve-provision-location\") pod \"auth-enabled-test-kserve-5df9cf6f97-l7scp\" (UID: \"66344676-0c38-4f6e-bc8a-bb19b064f1f9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp"
Sep 16 18:57:00.393916 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:00.393863    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66344676-0c38-4f6e-bc8a-bb19b064f1f9-cabundle-cert\") pod \"auth-enabled-test-kserve-5df9cf6f97-l7scp\" (UID: \"66344676-0c38-4f6e-bc8a-bb19b064f1f9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp"
Sep 16 18:57:00.394004 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:00.393967    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sv5f\" (UniqueName: \"kubernetes.io/projected/66344676-0c38-4f6e-bc8a-bb19b064f1f9-kube-api-access-2sv5f\") pod \"auth-enabled-test-kserve-5df9cf6f97-l7scp\" (UID: \"66344676-0c38-4f6e-bc8a-bb19b064f1f9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp"
Sep 16 18:57:00.394004 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:00.393990    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66344676-0c38-4f6e-bc8a-bb19b064f1f9-dshm\") pod \"auth-enabled-test-kserve-5df9cf6f97-l7scp\" (UID: \"66344676-0c38-4f6e-bc8a-bb19b064f1f9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp"
Sep 16 18:57:00.394090 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:00.394019    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66344676-0c38-4f6e-bc8a-bb19b064f1f9-home\") pod \"auth-enabled-test-kserve-5df9cf6f97-l7scp\" (UID: \"66344676-0c38-4f6e-bc8a-bb19b064f1f9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp"
Sep 16 18:57:00.394090 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:00.394033    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66344676-0c38-4f6e-bc8a-bb19b064f1f9-model-cache\") pod \"auth-enabled-test-kserve-5df9cf6f97-l7scp\" (UID: \"66344676-0c38-4f6e-bc8a-bb19b064f1f9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp"
Sep 16 18:57:00.394090 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:00.394048    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66344676-0c38-4f6e-bc8a-bb19b064f1f9-tmp-dir\") pod \"auth-enabled-test-kserve-5df9cf6f97-l7scp\" (UID: \"66344676-0c38-4f6e-bc8a-bb19b064f1f9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp"
Sep 16 18:57:00.495009 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:00.494973    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66344676-0c38-4f6e-bc8a-bb19b064f1f9-kserve-provision-location\") pod \"auth-enabled-test-kserve-5df9cf6f97-l7scp\" (UID: \"66344676-0c38-4f6e-bc8a-bb19b064f1f9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp"
Sep 16 18:57:00.495009 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:00.495011    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66344676-0c38-4f6e-bc8a-bb19b064f1f9-cabundle-cert\") pod \"auth-enabled-test-kserve-5df9cf6f97-l7scp\" (UID: \"66344676-0c38-4f6e-bc8a-bb19b064f1f9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp"
Sep 16 18:57:00.495332 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:00.495051    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sv5f\" (UniqueName: \"kubernetes.io/projected/66344676-0c38-4f6e-bc8a-bb19b064f1f9-kube-api-access-2sv5f\") pod \"auth-enabled-test-kserve-5df9cf6f97-l7scp\" (UID: \"66344676-0c38-4f6e-bc8a-bb19b064f1f9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp"
Sep 16 18:57:00.495332 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:00.495069    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66344676-0c38-4f6e-bc8a-bb19b064f1f9-dshm\") pod \"auth-enabled-test-kserve-5df9cf6f97-l7scp\" (UID: \"66344676-0c38-4f6e-bc8a-bb19b064f1f9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp"
Sep 16 18:57:00.495332 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:00.495095    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66344676-0c38-4f6e-bc8a-bb19b064f1f9-home\") pod \"auth-enabled-test-kserve-5df9cf6f97-l7scp\" (UID: \"66344676-0c38-4f6e-bc8a-bb19b064f1f9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp"
Sep 16 18:57:00.495332 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:00.495110    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66344676-0c38-4f6e-bc8a-bb19b064f1f9-model-cache\") pod \"auth-enabled-test-kserve-5df9cf6f97-l7scp\" (UID: \"66344676-0c38-4f6e-bc8a-bb19b064f1f9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp"
Sep 16 18:57:00.495332 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:00.495126    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66344676-0c38-4f6e-bc8a-bb19b064f1f9-tmp-dir\") pod \"auth-enabled-test-kserve-5df9cf6f97-l7scp\" (UID: \"66344676-0c38-4f6e-bc8a-bb19b064f1f9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp"
Sep 16 18:57:00.495332 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:00.495162    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66344676-0c38-4f6e-bc8a-bb19b064f1f9-tls-certs\") pod \"auth-enabled-test-kserve-5df9cf6f97-l7scp\" (UID: \"66344676-0c38-4f6e-bc8a-bb19b064f1f9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp"
Sep 16 18:57:00.495662 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:00.495432    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66344676-0c38-4f6e-bc8a-bb19b064f1f9-home\") pod \"auth-enabled-test-kserve-5df9cf6f97-l7scp\" (UID: \"66344676-0c38-4f6e-bc8a-bb19b064f1f9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp"
Sep 16 18:57:00.495662 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:00.495453    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66344676-0c38-4f6e-bc8a-bb19b064f1f9-kserve-provision-location\") pod \"auth-enabled-test-kserve-5df9cf6f97-l7scp\" (UID: \"66344676-0c38-4f6e-bc8a-bb19b064f1f9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp"
Sep 16 18:57:00.495662 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:00.495606    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66344676-0c38-4f6e-bc8a-bb19b064f1f9-tmp-dir\") pod \"auth-enabled-test-kserve-5df9cf6f97-l7scp\" (UID: \"66344676-0c38-4f6e-bc8a-bb19b064f1f9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp"
Sep 16 18:57:00.495765 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:00.495658    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66344676-0c38-4f6e-bc8a-bb19b064f1f9-model-cache\") pod \"auth-enabled-test-kserve-5df9cf6f97-l7scp\" (UID: \"66344676-0c38-4f6e-bc8a-bb19b064f1f9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp"
Sep 16 18:57:00.495878 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:00.495847    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66344676-0c38-4f6e-bc8a-bb19b064f1f9-cabundle-cert\") pod \"auth-enabled-test-kserve-5df9cf6f97-l7scp\" (UID: \"66344676-0c38-4f6e-bc8a-bb19b064f1f9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp"
Sep 16 18:57:00.497747 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:00.497728    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66344676-0c38-4f6e-bc8a-bb19b064f1f9-dshm\") pod \"auth-enabled-test-kserve-5df9cf6f97-l7scp\" (UID: \"66344676-0c38-4f6e-bc8a-bb19b064f1f9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp"
Sep 16 18:57:00.497889 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:00.497874    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66344676-0c38-4f6e-bc8a-bb19b064f1f9-tls-certs\") pod \"auth-enabled-test-kserve-5df9cf6f97-l7scp\" (UID: \"66344676-0c38-4f6e-bc8a-bb19b064f1f9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp"
Sep 16 18:57:00.504589 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:00.504569    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sv5f\" (UniqueName: \"kubernetes.io/projected/66344676-0c38-4f6e-bc8a-bb19b064f1f9-kube-api-access-2sv5f\") pod \"auth-enabled-test-kserve-5df9cf6f97-l7scp\" (UID: \"66344676-0c38-4f6e-bc8a-bb19b064f1f9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp"
Sep 16 18:57:00.683992 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:00.683961    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp"
Sep 16 18:57:00.961680 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:00.961658    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp"]
Sep 16 18:57:00.964473 ip-10-0-128-247 kubenswrapper[2577]: W0916 18:57:00.964441    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66344676_0c38_4f6e_bc8a_bb19b064f1f9.slice/crio-796c5d60b7b59187484a8b3e94bea631be36de9c40e1ed211ba2a4f28993c21a WatchSource:0}: Error finding container 796c5d60b7b59187484a8b3e94bea631be36de9c40e1ed211ba2a4f28993c21a: Status 404 returned error can't find the container with id 796c5d60b7b59187484a8b3e94bea631be36de9c40e1ed211ba2a4f28993c21a
Sep 16 18:57:01.350091 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:01.350012    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp" event={"ID":"66344676-0c38-4f6e-bc8a-bb19b064f1f9","Type":"ContainerStarted","Data":"d76898af1b235bec3690a0cf3eb119d79e8ebd9b03f97a1260c050155ad4d491"}
Sep 16 18:57:01.350091 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:01.350073    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp" event={"ID":"66344676-0c38-4f6e-bc8a-bb19b064f1f9","Type":"ContainerStarted","Data":"796c5d60b7b59187484a8b3e94bea631be36de9c40e1ed211ba2a4f28993c21a"}
Sep 16 18:57:03.365113 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:03.365034    2577 generic.go:356] "Generic (PLEG): container finished" podID="66344676-0c38-4f6e-bc8a-bb19b064f1f9" containerID="d76898af1b235bec3690a0cf3eb119d79e8ebd9b03f97a1260c050155ad4d491" exitCode=0
Sep 16 18:57:03.365113 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:03.365081    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp" event={"ID":"66344676-0c38-4f6e-bc8a-bb19b064f1f9","Type":"ContainerDied","Data":"d76898af1b235bec3690a0cf3eb119d79e8ebd9b03f97a1260c050155ad4d491"}
Sep 16 18:57:04.371259 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:04.371222    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp" event={"ID":"66344676-0c38-4f6e-bc8a-bb19b064f1f9","Type":"ContainerStarted","Data":"1911584924b56c8bf89fbf62932238532300323cdc3250e180384c976c347082"}
Sep 16 18:57:04.416945 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:04.416895    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp" podStartSLOduration=2.366689328 podStartE2EDuration="4.416881382s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.050192054s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:57:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:57:04.416027942 +0000 UTC m=+981.798917309" watchObservedRunningTime="2026-09-16 18:57:04.416881382 +0000 UTC m=+981.799770747"
Sep 16 18:57:04.684798 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:04.684758    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp"
Sep 16 18:57:10.684593 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:10.684557    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp"
Sep 16 18:57:10.685990 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:10.685963    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp" podUID="66344676-0c38-4f6e-bc8a-bb19b064f1f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Sep 16 18:57:20.684589 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:20.684522    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp" podUID="66344676-0c38-4f6e-bc8a-bb19b064f1f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Sep 16 18:57:30.684868 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:30.684826    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp" podUID="66344676-0c38-4f6e-bc8a-bb19b064f1f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Sep 16 18:57:40.684575 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:40.684506    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp" podUID="66344676-0c38-4f6e-bc8a-bb19b064f1f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Sep 16 18:57:50.684381 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:57:50.684341    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp" podUID="66344676-0c38-4f6e-bc8a-bb19b064f1f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Sep 16 18:58:00.684485 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:58:00.684440    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp" podUID="66344676-0c38-4f6e-bc8a-bb19b064f1f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Sep 16 18:58:10.684889 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:58:10.684850    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp" podUID="66344676-0c38-4f6e-bc8a-bb19b064f1f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Sep 16 18:58:20.684721 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:58:20.684686    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp" podUID="66344676-0c38-4f6e-bc8a-bb19b064f1f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Sep 16 18:58:30.685288 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:58:30.685244    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp" podUID="66344676-0c38-4f6e-bc8a-bb19b064f1f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Sep 16 18:58:40.684848 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:58:40.684763    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp" podUID="66344676-0c38-4f6e-bc8a-bb19b064f1f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Sep 16 18:58:50.695092 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:58:50.695060    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp"
Sep 16 18:58:50.702824 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:58:50.702801    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp"
Sep 16 18:58:56.012016 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:58:56.011982    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp"]
Sep 16 18:59:11.034653 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:59:11.034606    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp" podUID="66344676-0c38-4f6e-bc8a-bb19b064f1f9" containerName="main" containerID="cri-o://1911584924b56c8bf89fbf62932238532300323cdc3250e180384c976c347082" gracePeriod=45
Sep 16 18:59:11.771654 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:59:11.771633    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp"
Sep 16 18:59:11.876626 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:59:11.876560    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sv5f\" (UniqueName: \"kubernetes.io/projected/66344676-0c38-4f6e-bc8a-bb19b064f1f9-kube-api-access-2sv5f\") pod \"66344676-0c38-4f6e-bc8a-bb19b064f1f9\" (UID: \"66344676-0c38-4f6e-bc8a-bb19b064f1f9\") "
Sep 16 18:59:11.876626 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:59:11.876623    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66344676-0c38-4f6e-bc8a-bb19b064f1f9-home\") pod \"66344676-0c38-4f6e-bc8a-bb19b064f1f9\" (UID: \"66344676-0c38-4f6e-bc8a-bb19b064f1f9\") "
Sep 16 18:59:11.876813 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:59:11.876676    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66344676-0c38-4f6e-bc8a-bb19b064f1f9-cabundle-cert\") pod \"66344676-0c38-4f6e-bc8a-bb19b064f1f9\" (UID: \"66344676-0c38-4f6e-bc8a-bb19b064f1f9\") "
Sep 16 18:59:11.876813 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:59:11.876707    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66344676-0c38-4f6e-bc8a-bb19b064f1f9-tls-certs\") pod \"66344676-0c38-4f6e-bc8a-bb19b064f1f9\" (UID: \"66344676-0c38-4f6e-bc8a-bb19b064f1f9\") "
Sep 16 18:59:11.876813 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:59:11.876729    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66344676-0c38-4f6e-bc8a-bb19b064f1f9-tmp-dir\") pod \"66344676-0c38-4f6e-bc8a-bb19b064f1f9\" (UID: \"66344676-0c38-4f6e-bc8a-bb19b064f1f9\") "
Sep 16 18:59:11.876813 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:59:11.876786    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66344676-0c38-4f6e-bc8a-bb19b064f1f9-kserve-provision-location\") pod \"66344676-0c38-4f6e-bc8a-bb19b064f1f9\" (UID: \"66344676-0c38-4f6e-bc8a-bb19b064f1f9\") "
Sep 16 18:59:11.876813 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:59:11.876809    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66344676-0c38-4f6e-bc8a-bb19b064f1f9-dshm\") pod \"66344676-0c38-4f6e-bc8a-bb19b064f1f9\" (UID: \"66344676-0c38-4f6e-bc8a-bb19b064f1f9\") "
Sep 16 18:59:11.877046 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:59:11.876835    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66344676-0c38-4f6e-bc8a-bb19b064f1f9-model-cache\") pod \"66344676-0c38-4f6e-bc8a-bb19b064f1f9\" (UID: \"66344676-0c38-4f6e-bc8a-bb19b064f1f9\") "
Sep 16 18:59:11.877125 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:59:11.877085    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66344676-0c38-4f6e-bc8a-bb19b064f1f9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "66344676-0c38-4f6e-bc8a-bb19b064f1f9" (UID: "66344676-0c38-4f6e-bc8a-bb19b064f1f9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:59:11.877317 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:59:11.877284    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66344676-0c38-4f6e-bc8a-bb19b064f1f9-model-cache" (OuterVolumeSpecName: "model-cache") pod "66344676-0c38-4f6e-bc8a-bb19b064f1f9" (UID: "66344676-0c38-4f6e-bc8a-bb19b064f1f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:11.877602 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:59:11.877572    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66344676-0c38-4f6e-bc8a-bb19b064f1f9-home" (OuterVolumeSpecName: "home") pod "66344676-0c38-4f6e-bc8a-bb19b064f1f9" (UID: "66344676-0c38-4f6e-bc8a-bb19b064f1f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:11.879232 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:59:11.879204    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66344676-0c38-4f6e-bc8a-bb19b064f1f9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "66344676-0c38-4f6e-bc8a-bb19b064f1f9" (UID: "66344676-0c38-4f6e-bc8a-bb19b064f1f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:59:11.879319 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:59:11.879278    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66344676-0c38-4f6e-bc8a-bb19b064f1f9-kube-api-access-2sv5f" (OuterVolumeSpecName: "kube-api-access-2sv5f") pod "66344676-0c38-4f6e-bc8a-bb19b064f1f9" (UID: "66344676-0c38-4f6e-bc8a-bb19b064f1f9"). InnerVolumeSpecName "kube-api-access-2sv5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:59:11.879505 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:59:11.879477    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66344676-0c38-4f6e-bc8a-bb19b064f1f9-dshm" (OuterVolumeSpecName: "dshm") pod "66344676-0c38-4f6e-bc8a-bb19b064f1f9" (UID: "66344676-0c38-4f6e-bc8a-bb19b064f1f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:11.890357 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:59:11.890332    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66344676-0c38-4f6e-bc8a-bb19b064f1f9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "66344676-0c38-4f6e-bc8a-bb19b064f1f9" (UID: "66344676-0c38-4f6e-bc8a-bb19b064f1f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:11.899361 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:59:11.899339    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66344676-0c38-4f6e-bc8a-bb19b064f1f9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "66344676-0c38-4f6e-bc8a-bb19b064f1f9" (UID: "66344676-0c38-4f6e-bc8a-bb19b064f1f9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:11.931713 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:59:11.931685    2577 generic.go:356] "Generic (PLEG): container finished" podID="66344676-0c38-4f6e-bc8a-bb19b064f1f9" containerID="1911584924b56c8bf89fbf62932238532300323cdc3250e180384c976c347082" exitCode=0
Sep 16 18:59:11.931796 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:59:11.931742    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp" event={"ID":"66344676-0c38-4f6e-bc8a-bb19b064f1f9","Type":"ContainerDied","Data":"1911584924b56c8bf89fbf62932238532300323cdc3250e180384c976c347082"}
Sep 16 18:59:11.931847 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:59:11.931817    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp" event={"ID":"66344676-0c38-4f6e-bc8a-bb19b064f1f9","Type":"ContainerDied","Data":"796c5d60b7b59187484a8b3e94bea631be36de9c40e1ed211ba2a4f28993c21a"}
Sep 16 18:59:11.931847 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:59:11.931746    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp"
Sep 16 18:59:11.931954 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:59:11.931839    2577 scope.go:117] "RemoveContainer" containerID="1911584924b56c8bf89fbf62932238532300323cdc3250e180384c976c347082"
Sep 16 18:59:11.941261 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:59:11.941245    2577 scope.go:117] "RemoveContainer" containerID="d76898af1b235bec3690a0cf3eb119d79e8ebd9b03f97a1260c050155ad4d491"
Sep 16 18:59:11.948701 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:59:11.948686    2577 scope.go:117] "RemoveContainer" containerID="1911584924b56c8bf89fbf62932238532300323cdc3250e180384c976c347082"
Sep 16 18:59:11.948954 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:59:11.948936    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1911584924b56c8bf89fbf62932238532300323cdc3250e180384c976c347082\": container with ID starting with 1911584924b56c8bf89fbf62932238532300323cdc3250e180384c976c347082 not found: ID does not exist" containerID="1911584924b56c8bf89fbf62932238532300323cdc3250e180384c976c347082"
Sep 16 18:59:11.949021 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:59:11.948964    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1911584924b56c8bf89fbf62932238532300323cdc3250e180384c976c347082"} err="failed to get container status \"1911584924b56c8bf89fbf62932238532300323cdc3250e180384c976c347082\": rpc error: code = NotFound desc = could not find container \"1911584924b56c8bf89fbf62932238532300323cdc3250e180384c976c347082\": container with ID starting with 1911584924b56c8bf89fbf62932238532300323cdc3250e180384c976c347082 not found: ID does not exist"
Sep 16 18:59:11.949021 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:59:11.948987    2577 scope.go:117] "RemoveContainer" containerID="d76898af1b235bec3690a0cf3eb119d79e8ebd9b03f97a1260c050155ad4d491"
Sep 16 18:59:11.949242 ip-10-0-128-247 kubenswrapper[2577]: E0916 18:59:11.949224    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d76898af1b235bec3690a0cf3eb119d79e8ebd9b03f97a1260c050155ad4d491\": container with ID starting with d76898af1b235bec3690a0cf3eb119d79e8ebd9b03f97a1260c050155ad4d491 not found: ID does not exist" containerID="d76898af1b235bec3690a0cf3eb119d79e8ebd9b03f97a1260c050155ad4d491"
Sep 16 18:59:11.949281 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:59:11.949248    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d76898af1b235bec3690a0cf3eb119d79e8ebd9b03f97a1260c050155ad4d491"} err="failed to get container status \"d76898af1b235bec3690a0cf3eb119d79e8ebd9b03f97a1260c050155ad4d491\": rpc error: code = NotFound desc = could not find container \"d76898af1b235bec3690a0cf3eb119d79e8ebd9b03f97a1260c050155ad4d491\": container with ID starting with d76898af1b235bec3690a0cf3eb119d79e8ebd9b03f97a1260c050155ad4d491 not found: ID does not exist"
Sep 16 18:59:11.957295 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:59:11.957274    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp"]
Sep 16 18:59:11.961385 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:59:11.961365    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp"]
Sep 16 18:59:11.978043 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:59:11.978017    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sv5f\" (UniqueName: \"kubernetes.io/projected/66344676-0c38-4f6e-bc8a-bb19b064f1f9-kube-api-access-2sv5f\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:59:11.978043 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:59:11.978041    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66344676-0c38-4f6e-bc8a-bb19b064f1f9-home\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:59:11.978178 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:59:11.978056    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66344676-0c38-4f6e-bc8a-bb19b064f1f9-cabundle-cert\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:59:11.978178 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:59:11.978068    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66344676-0c38-4f6e-bc8a-bb19b064f1f9-tls-certs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:59:11.978178 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:59:11.978079    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66344676-0c38-4f6e-bc8a-bb19b064f1f9-tmp-dir\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:59:11.978178 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:59:11.978090    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66344676-0c38-4f6e-bc8a-bb19b064f1f9-kserve-provision-location\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:59:11.978178 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:59:11.978102    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66344676-0c38-4f6e-bc8a-bb19b064f1f9-dshm\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:59:11.978178 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:59:11.978114    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66344676-0c38-4f6e-bc8a-bb19b064f1f9-model-cache\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 18:59:12.696683 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:59:12.696643    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5df9cf6f97-l7scp" podUID="66344676-0c38-4f6e-bc8a-bb19b064f1f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": context deadline exceeded"
Sep 16 18:59:13.155425 ip-10-0-128-247 kubenswrapper[2577]: I0916 18:59:13.155396    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66344676-0c38-4f6e-bc8a-bb19b064f1f9" path="/var/lib/kubelet/pods/66344676-0c38-4f6e-bc8a-bb19b064f1f9/volumes"
Sep 16 19:00:20.412295 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:20.412262    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw"]
Sep 16 19:00:20.412692 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:20.412594    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66344676-0c38-4f6e-bc8a-bb19b064f1f9" containerName="storage-initializer"
Sep 16 19:00:20.412692 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:20.412608    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="66344676-0c38-4f6e-bc8a-bb19b064f1f9" containerName="storage-initializer"
Sep 16 19:00:20.412692 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:20.412630    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66344676-0c38-4f6e-bc8a-bb19b064f1f9" containerName="main"
Sep 16 19:00:20.412692 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:20.412639    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="66344676-0c38-4f6e-bc8a-bb19b064f1f9" containerName="main"
Sep 16 19:00:20.412692 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:20.412693    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="66344676-0c38-4f6e-bc8a-bb19b064f1f9" containerName="main"
Sep 16 19:00:20.415615 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:20.415597    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw"
Sep 16 19:00:20.419617 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:20.419596    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Sep 16 19:00:20.420072 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:20.420056    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 16 19:00:20.420149 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:20.420133    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 16 19:00:20.420194 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:20.420133    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 16 19:00:20.420803 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:20.420789    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:00:20.433118 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:20.433100    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-8v8vp\""
Sep 16 19:00:20.475360 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:20.475335    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrl9q\" (UniqueName: \"kubernetes.io/projected/a36d2855-5787-4eea-8fe0-1a564db10646-kube-api-access-mrl9q\") pod \"auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw\" (UID: \"a36d2855-5787-4eea-8fe0-1a564db10646\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw"
Sep 16 19:00:20.475465 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:20.475378    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a36d2855-5787-4eea-8fe0-1a564db10646-tls-certs\") pod \"auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw\" (UID: \"a36d2855-5787-4eea-8fe0-1a564db10646\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw"
Sep 16 19:00:20.475465 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:20.475403    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a36d2855-5787-4eea-8fe0-1a564db10646-cabundle-cert\") pod \"auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw\" (UID: \"a36d2855-5787-4eea-8fe0-1a564db10646\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw"
Sep 16 19:00:20.475465 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:20.475425    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a36d2855-5787-4eea-8fe0-1a564db10646-home\") pod \"auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw\" (UID: \"a36d2855-5787-4eea-8fe0-1a564db10646\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw"
Sep 16 19:00:20.475676 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:20.475513    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a36d2855-5787-4eea-8fe0-1a564db10646-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw\" (UID: \"a36d2855-5787-4eea-8fe0-1a564db10646\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw"
Sep 16 19:00:20.475676 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:20.475594    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a36d2855-5787-4eea-8fe0-1a564db10646-model-cache\") pod \"auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw\" (UID: \"a36d2855-5787-4eea-8fe0-1a564db10646\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw"
Sep 16 19:00:20.475676 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:20.475646    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a36d2855-5787-4eea-8fe0-1a564db10646-dshm\") pod \"auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw\" (UID: \"a36d2855-5787-4eea-8fe0-1a564db10646\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw"
Sep 16 19:00:20.475803 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:20.475680    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a36d2855-5787-4eea-8fe0-1a564db10646-tmp-dir\") pod \"auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw\" (UID: \"a36d2855-5787-4eea-8fe0-1a564db10646\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw"
Sep 16 19:00:20.521203 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:20.521154    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw"]
Sep 16 19:00:20.576746 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:20.576725    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a36d2855-5787-4eea-8fe0-1a564db10646-dshm\") pod \"auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw\" (UID: \"a36d2855-5787-4eea-8fe0-1a564db10646\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw"
Sep 16 19:00:20.576831 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:20.576749    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a36d2855-5787-4eea-8fe0-1a564db10646-tmp-dir\") pod \"auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw\" (UID: \"a36d2855-5787-4eea-8fe0-1a564db10646\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw"
Sep 16 19:00:20.576831 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:20.576777    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrl9q\" (UniqueName: \"kubernetes.io/projected/a36d2855-5787-4eea-8fe0-1a564db10646-kube-api-access-mrl9q\") pod \"auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw\" (UID: \"a36d2855-5787-4eea-8fe0-1a564db10646\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw"
Sep 16 19:00:20.576831 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:20.576802    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a36d2855-5787-4eea-8fe0-1a564db10646-tls-certs\") pod \"auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw\" (UID: \"a36d2855-5787-4eea-8fe0-1a564db10646\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw"
Sep 16 19:00:20.576831 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:20.576817    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a36d2855-5787-4eea-8fe0-1a564db10646-cabundle-cert\") pod \"auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw\" (UID: \"a36d2855-5787-4eea-8fe0-1a564db10646\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw"
Sep 16 19:00:20.577037 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:20.576832    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a36d2855-5787-4eea-8fe0-1a564db10646-home\") pod \"auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw\" (UID: \"a36d2855-5787-4eea-8fe0-1a564db10646\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw"
Sep 16 19:00:20.577037 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:20.576873    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a36d2855-5787-4eea-8fe0-1a564db10646-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw\" (UID: \"a36d2855-5787-4eea-8fe0-1a564db10646\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw"
Sep 16 19:00:20.577037 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:20.576904    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a36d2855-5787-4eea-8fe0-1a564db10646-model-cache\") pod \"auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw\" (UID: \"a36d2855-5787-4eea-8fe0-1a564db10646\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw"
Sep 16 19:00:20.577187 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:20.577167    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a36d2855-5787-4eea-8fe0-1a564db10646-tmp-dir\") pod \"auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw\" (UID: \"a36d2855-5787-4eea-8fe0-1a564db10646\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw"
Sep 16 19:00:20.577484 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:20.577464    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a36d2855-5787-4eea-8fe0-1a564db10646-home\") pod \"auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw\" (UID: \"a36d2855-5787-4eea-8fe0-1a564db10646\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw"
Sep 16 19:00:20.577590 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:20.577508    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a36d2855-5787-4eea-8fe0-1a564db10646-model-cache\") pod \"auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw\" (UID: \"a36d2855-5787-4eea-8fe0-1a564db10646\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw"
Sep 16 19:00:20.577690 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:20.577674    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a36d2855-5787-4eea-8fe0-1a564db10646-cabundle-cert\") pod \"auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw\" (UID: \"a36d2855-5787-4eea-8fe0-1a564db10646\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw"
Sep 16 19:00:20.577791 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:20.577770    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a36d2855-5787-4eea-8fe0-1a564db10646-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw\" (UID: \"a36d2855-5787-4eea-8fe0-1a564db10646\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw"
Sep 16 19:00:20.579158 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:20.579134    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a36d2855-5787-4eea-8fe0-1a564db10646-dshm\") pod \"auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw\" (UID: \"a36d2855-5787-4eea-8fe0-1a564db10646\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw"
Sep 16 19:00:20.579393 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:20.579375    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a36d2855-5787-4eea-8fe0-1a564db10646-tls-certs\") pod \"auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw\" (UID: \"a36d2855-5787-4eea-8fe0-1a564db10646\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw"
Sep 16 19:00:20.587778 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:20.587759    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrl9q\" (UniqueName: \"kubernetes.io/projected/a36d2855-5787-4eea-8fe0-1a564db10646-kube-api-access-mrl9q\") pod \"auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw\" (UID: \"a36d2855-5787-4eea-8fe0-1a564db10646\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw"
Sep 16 19:00:20.725747 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:20.725721    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw"
Sep 16 19:00:21.005174 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:21.005151    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw"]
Sep 16 19:00:21.007513 ip-10-0-128-247 kubenswrapper[2577]: W0916 19:00:21.007487    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda36d2855_5787_4eea_8fe0_1a564db10646.slice/crio-ae99daa3dace3a4494c34695f15b66afec3be61024f4867f67708dda3ef5b122 WatchSource:0}: Error finding container ae99daa3dace3a4494c34695f15b66afec3be61024f4867f67708dda3ef5b122: Status 404 returned error can't find the container with id ae99daa3dace3a4494c34695f15b66afec3be61024f4867f67708dda3ef5b122
Sep 16 19:00:21.226043 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:21.226006    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw" event={"ID":"a36d2855-5787-4eea-8fe0-1a564db10646","Type":"ContainerStarted","Data":"80bd53e5fa447dd4e3190b374e6a01ac7e635d7dc340e4c5595b9c8c255d11f9"}
Sep 16 19:00:21.226191 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:21.226058    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw" event={"ID":"a36d2855-5787-4eea-8fe0-1a564db10646","Type":"ContainerStarted","Data":"ae99daa3dace3a4494c34695f15b66afec3be61024f4867f67708dda3ef5b122"}
Sep 16 19:00:23.235178 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:23.235148    2577 generic.go:356] "Generic (PLEG): container finished" podID="a36d2855-5787-4eea-8fe0-1a564db10646" containerID="80bd53e5fa447dd4e3190b374e6a01ac7e635d7dc340e4c5595b9c8c255d11f9" exitCode=0
Sep 16 19:00:23.235488 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:23.235189    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw" event={"ID":"a36d2855-5787-4eea-8fe0-1a564db10646","Type":"ContainerDied","Data":"80bd53e5fa447dd4e3190b374e6a01ac7e635d7dc340e4c5595b9c8c255d11f9"}
Sep 16 19:00:24.248459 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:24.248417    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw" event={"ID":"a36d2855-5787-4eea-8fe0-1a564db10646","Type":"ContainerStarted","Data":"79ca9c1491d23cdcd83d03195b02d6f75c648372c059fdf233efab7e5dff6edb"}
Sep 16 19:00:24.283411 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:24.283356    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw" podStartSLOduration=2.264154114 podStartE2EDuration="4.283340754s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.01918664s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:00:24.282082729 +0000 UTC m=+1181.664972093" watchObservedRunningTime="2026-09-16 19:00:24.283340754 +0000 UTC m=+1181.666230116"
Sep 16 19:00:24.726427 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:24.726394    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw"
Sep 16 19:00:30.726468 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:30.726417    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw"
Sep 16 19:00:30.728019 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:30.727983    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw" podUID="a36d2855-5787-4eea-8fe0-1a564db10646" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Sep 16 19:00:40.726880 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:40.726840    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw" podUID="a36d2855-5787-4eea-8fe0-1a564db10646" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Sep 16 19:00:50.726827 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:00:50.726767    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw" podUID="a36d2855-5787-4eea-8fe0-1a564db10646" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Sep 16 19:01:00.726504 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:01:00.726459    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw" podUID="a36d2855-5787-4eea-8fe0-1a564db10646" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Sep 16 19:01:10.726305 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:01:10.726260    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw" podUID="a36d2855-5787-4eea-8fe0-1a564db10646" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Sep 16 19:01:20.726979 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:01:20.726933    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw" podUID="a36d2855-5787-4eea-8fe0-1a564db10646" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Sep 16 19:01:30.727165 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:01:30.727121    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw" podUID="a36d2855-5787-4eea-8fe0-1a564db10646" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Sep 16 19:01:40.727016 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:01:40.726925    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw" podUID="a36d2855-5787-4eea-8fe0-1a564db10646" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Sep 16 19:01:50.726621 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:01:50.726570    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw" podUID="a36d2855-5787-4eea-8fe0-1a564db10646" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Sep 16 19:02:00.726703 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:02:00.726658    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw" podUID="a36d2855-5787-4eea-8fe0-1a564db10646" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Sep 16 19:02:10.736768 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:02:10.736732    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw"
Sep 16 19:02:10.744732 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:02:10.744705    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw"
Sep 16 19:02:12.561523 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:02:12.561488    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw"]
Sep 16 19:02:27.737847 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:02:27.737803    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw" podUID="a36d2855-5787-4eea-8fe0-1a564db10646" containerName="main" containerID="cri-o://79ca9c1491d23cdcd83d03195b02d6f75c648372c059fdf233efab7e5dff6edb" gracePeriod=45
Sep 16 19:02:28.483472 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:02:28.483450    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw"
Sep 16 19:02:28.610703 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:02:28.610643    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a36d2855-5787-4eea-8fe0-1a564db10646-tmp-dir\") pod \"a36d2855-5787-4eea-8fe0-1a564db10646\" (UID: \"a36d2855-5787-4eea-8fe0-1a564db10646\") "
Sep 16 19:02:28.610703 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:02:28.610677    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a36d2855-5787-4eea-8fe0-1a564db10646-dshm\") pod \"a36d2855-5787-4eea-8fe0-1a564db10646\" (UID: \"a36d2855-5787-4eea-8fe0-1a564db10646\") "
Sep 16 19:02:28.610703 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:02:28.610692    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a36d2855-5787-4eea-8fe0-1a564db10646-home\") pod \"a36d2855-5787-4eea-8fe0-1a564db10646\" (UID: \"a36d2855-5787-4eea-8fe0-1a564db10646\") "
Sep 16 19:02:28.610950 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:02:28.610708    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a36d2855-5787-4eea-8fe0-1a564db10646-kserve-provision-location\") pod \"a36d2855-5787-4eea-8fe0-1a564db10646\" (UID: \"a36d2855-5787-4eea-8fe0-1a564db10646\") "
Sep 16 19:02:28.610950 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:02:28.610746    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a36d2855-5787-4eea-8fe0-1a564db10646-tls-certs\") pod \"a36d2855-5787-4eea-8fe0-1a564db10646\" (UID: \"a36d2855-5787-4eea-8fe0-1a564db10646\") "
Sep 16 19:02:28.610950 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:02:28.610791    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a36d2855-5787-4eea-8fe0-1a564db10646-model-cache\") pod \"a36d2855-5787-4eea-8fe0-1a564db10646\" (UID: \"a36d2855-5787-4eea-8fe0-1a564db10646\") "
Sep 16 19:02:28.610950 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:02:28.610823    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrl9q\" (UniqueName: \"kubernetes.io/projected/a36d2855-5787-4eea-8fe0-1a564db10646-kube-api-access-mrl9q\") pod \"a36d2855-5787-4eea-8fe0-1a564db10646\" (UID: \"a36d2855-5787-4eea-8fe0-1a564db10646\") "
Sep 16 19:02:28.610950 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:02:28.610943    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a36d2855-5787-4eea-8fe0-1a564db10646-cabundle-cert\") pod \"a36d2855-5787-4eea-8fe0-1a564db10646\" (UID: \"a36d2855-5787-4eea-8fe0-1a564db10646\") "
Sep 16 19:02:28.611204 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:02:28.611121    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a36d2855-5787-4eea-8fe0-1a564db10646-model-cache" (OuterVolumeSpecName: "model-cache") pod "a36d2855-5787-4eea-8fe0-1a564db10646" (UID: "a36d2855-5787-4eea-8fe0-1a564db10646"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:02:28.611204 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:02:28.611197    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a36d2855-5787-4eea-8fe0-1a564db10646-model-cache\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:02:28.611484 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:02:28.611452    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a36d2855-5787-4eea-8fe0-1a564db10646-home" (OuterVolumeSpecName: "home") pod "a36d2855-5787-4eea-8fe0-1a564db10646" (UID: "a36d2855-5787-4eea-8fe0-1a564db10646"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:02:28.611624 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:02:28.611545    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a36d2855-5787-4eea-8fe0-1a564db10646-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a36d2855-5787-4eea-8fe0-1a564db10646" (UID: "a36d2855-5787-4eea-8fe0-1a564db10646"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:02:28.613452 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:02:28.613421    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a36d2855-5787-4eea-8fe0-1a564db10646-kube-api-access-mrl9q" (OuterVolumeSpecName: "kube-api-access-mrl9q") pod "a36d2855-5787-4eea-8fe0-1a564db10646" (UID: "a36d2855-5787-4eea-8fe0-1a564db10646"). InnerVolumeSpecName "kube-api-access-mrl9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:02:28.613601 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:02:28.613582    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a36d2855-5787-4eea-8fe0-1a564db10646-dshm" (OuterVolumeSpecName: "dshm") pod "a36d2855-5787-4eea-8fe0-1a564db10646" (UID: "a36d2855-5787-4eea-8fe0-1a564db10646"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:02:28.613663 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:02:28.613620    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a36d2855-5787-4eea-8fe0-1a564db10646-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a36d2855-5787-4eea-8fe0-1a564db10646" (UID: "a36d2855-5787-4eea-8fe0-1a564db10646"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:02:28.623917 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:02:28.623893    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a36d2855-5787-4eea-8fe0-1a564db10646-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a36d2855-5787-4eea-8fe0-1a564db10646" (UID: "a36d2855-5787-4eea-8fe0-1a564db10646"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:02:28.634496 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:02:28.634465    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a36d2855-5787-4eea-8fe0-1a564db10646-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a36d2855-5787-4eea-8fe0-1a564db10646" (UID: "a36d2855-5787-4eea-8fe0-1a564db10646"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:02:28.712573 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:02:28.712552    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a36d2855-5787-4eea-8fe0-1a564db10646-tmp-dir\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:02:28.712573 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:02:28.712571    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a36d2855-5787-4eea-8fe0-1a564db10646-dshm\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:02:28.712683 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:02:28.712580    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a36d2855-5787-4eea-8fe0-1a564db10646-home\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:02:28.712683 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:02:28.712588    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a36d2855-5787-4eea-8fe0-1a564db10646-kserve-provision-location\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:02:28.712683 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:02:28.712596    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a36d2855-5787-4eea-8fe0-1a564db10646-tls-certs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:02:28.712683 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:02:28.712607    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrl9q\" (UniqueName: \"kubernetes.io/projected/a36d2855-5787-4eea-8fe0-1a564db10646-kube-api-access-mrl9q\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:02:28.712683 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:02:28.712615    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a36d2855-5787-4eea-8fe0-1a564db10646-cabundle-cert\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:02:28.783748 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:02:28.783721    2577 generic.go:356] "Generic (PLEG): container finished" podID="a36d2855-5787-4eea-8fe0-1a564db10646" containerID="79ca9c1491d23cdcd83d03195b02d6f75c648372c059fdf233efab7e5dff6edb" exitCode=0
Sep 16 19:02:28.784065 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:02:28.783776    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw"
Sep 16 19:02:28.784065 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:02:28.783795    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw" event={"ID":"a36d2855-5787-4eea-8fe0-1a564db10646","Type":"ContainerDied","Data":"79ca9c1491d23cdcd83d03195b02d6f75c648372c059fdf233efab7e5dff6edb"}
Sep 16 19:02:28.784065 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:02:28.783862    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw" event={"ID":"a36d2855-5787-4eea-8fe0-1a564db10646","Type":"ContainerDied","Data":"ae99daa3dace3a4494c34695f15b66afec3be61024f4867f67708dda3ef5b122"}
Sep 16 19:02:28.784065 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:02:28.783885    2577 scope.go:117] "RemoveContainer" containerID="79ca9c1491d23cdcd83d03195b02d6f75c648372c059fdf233efab7e5dff6edb"
Sep 16 19:02:28.792648 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:02:28.792625    2577 scope.go:117] "RemoveContainer" containerID="80bd53e5fa447dd4e3190b374e6a01ac7e635d7dc340e4c5595b9c8c255d11f9"
Sep 16 19:02:28.799904 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:02:28.799883    2577 scope.go:117] "RemoveContainer" containerID="79ca9c1491d23cdcd83d03195b02d6f75c648372c059fdf233efab7e5dff6edb"
Sep 16 19:02:28.800133 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:02:28.800117    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79ca9c1491d23cdcd83d03195b02d6f75c648372c059fdf233efab7e5dff6edb\": container with ID starting with 79ca9c1491d23cdcd83d03195b02d6f75c648372c059fdf233efab7e5dff6edb not found: ID does not exist" containerID="79ca9c1491d23cdcd83d03195b02d6f75c648372c059fdf233efab7e5dff6edb"
Sep 16 19:02:28.800187 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:02:28.800141    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79ca9c1491d23cdcd83d03195b02d6f75c648372c059fdf233efab7e5dff6edb"} err="failed to get container status \"79ca9c1491d23cdcd83d03195b02d6f75c648372c059fdf233efab7e5dff6edb\": rpc error: code = NotFound desc = could not find container \"79ca9c1491d23cdcd83d03195b02d6f75c648372c059fdf233efab7e5dff6edb\": container with ID starting with 79ca9c1491d23cdcd83d03195b02d6f75c648372c059fdf233efab7e5dff6edb not found: ID does not exist"
Sep 16 19:02:28.800187 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:02:28.800159    2577 scope.go:117] "RemoveContainer" containerID="80bd53e5fa447dd4e3190b374e6a01ac7e635d7dc340e4c5595b9c8c255d11f9"
Sep 16 19:02:28.800405 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:02:28.800392    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80bd53e5fa447dd4e3190b374e6a01ac7e635d7dc340e4c5595b9c8c255d11f9\": container with ID starting with 80bd53e5fa447dd4e3190b374e6a01ac7e635d7dc340e4c5595b9c8c255d11f9 not found: ID does not exist" containerID="80bd53e5fa447dd4e3190b374e6a01ac7e635d7dc340e4c5595b9c8c255d11f9"
Sep 16 19:02:28.800449 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:02:28.800408    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80bd53e5fa447dd4e3190b374e6a01ac7e635d7dc340e4c5595b9c8c255d11f9"} err="failed to get container status \"80bd53e5fa447dd4e3190b374e6a01ac7e635d7dc340e4c5595b9c8c255d11f9\": rpc error: code = NotFound desc = could not find container \"80bd53e5fa447dd4e3190b374e6a01ac7e635d7dc340e4c5595b9c8c255d11f9\": container with ID starting with 80bd53e5fa447dd4e3190b374e6a01ac7e635d7dc340e4c5595b9c8c255d11f9 not found: ID does not exist"
Sep 16 19:02:28.808388 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:02:28.808368    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw"]
Sep 16 19:02:28.814664 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:02:28.814644    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f67bf7f5c-qgxxw"]
Sep 16 19:02:29.161277 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:02:29.161244    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a36d2855-5787-4eea-8fe0-1a564db10646" path="/var/lib/kubelet/pods/a36d2855-5787-4eea-8fe0-1a564db10646/volumes"
Sep 16 19:03:20.586952 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:20.586883    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8"]
Sep 16 19:03:20.587314 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:20.587195    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a36d2855-5787-4eea-8fe0-1a564db10646" containerName="main"
Sep 16 19:03:20.587314 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:20.587207    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a36d2855-5787-4eea-8fe0-1a564db10646" containerName="main"
Sep 16 19:03:20.587314 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:20.587217    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a36d2855-5787-4eea-8fe0-1a564db10646" containerName="storage-initializer"
Sep 16 19:03:20.587314 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:20.587222    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a36d2855-5787-4eea-8fe0-1a564db10646" containerName="storage-initializer"
Sep 16 19:03:20.587314 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:20.587282    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a36d2855-5787-4eea-8fe0-1a564db10646" containerName="main"
Sep 16 19:03:20.590422 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:20.590404    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8"
Sep 16 19:03:20.593779 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:20.593756    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-knkkj\""
Sep 16 19:03:20.593899 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:20.593757    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:03:20.594355 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:20.594338    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 16 19:03:20.595047 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:20.595027    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 16 19:03:20.595145 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:20.595066    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Sep 16 19:03:20.595249 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:20.595235    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 16 19:03:20.603419 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:20.603398    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8"]
Sep 16 19:03:20.696616 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:20.696581    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-model-cache\") pod \"auth-disabled-test-kserve-6b7b464494-npxg8\" (UID: \"8c4be60c-637f-47ea-ac7d-295f5b1dfb79\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8"
Sep 16 19:03:20.696724 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:20.696625    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-kserve-provision-location\") pod \"auth-disabled-test-kserve-6b7b464494-npxg8\" (UID: \"8c4be60c-637f-47ea-ac7d-295f5b1dfb79\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8"
Sep 16 19:03:20.696724 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:20.696687    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-tls-certs\") pod \"auth-disabled-test-kserve-6b7b464494-npxg8\" (UID: \"8c4be60c-637f-47ea-ac7d-295f5b1dfb79\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8"
Sep 16 19:03:20.696724 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:20.696715    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-home\") pod \"auth-disabled-test-kserve-6b7b464494-npxg8\" (UID: \"8c4be60c-637f-47ea-ac7d-295f5b1dfb79\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8"
Sep 16 19:03:20.696824 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:20.696744    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-dshm\") pod \"auth-disabled-test-kserve-6b7b464494-npxg8\" (UID: \"8c4be60c-637f-47ea-ac7d-295f5b1dfb79\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8"
Sep 16 19:03:20.696824 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:20.696763    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-cabundle-cert\") pod \"auth-disabled-test-kserve-6b7b464494-npxg8\" (UID: \"8c4be60c-637f-47ea-ac7d-295f5b1dfb79\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8"
Sep 16 19:03:20.696824 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:20.696791    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87w5f\" (UniqueName: \"kubernetes.io/projected/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-kube-api-access-87w5f\") pod \"auth-disabled-test-kserve-6b7b464494-npxg8\" (UID: \"8c4be60c-637f-47ea-ac7d-295f5b1dfb79\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8"
Sep 16 19:03:20.696918 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:20.696827    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-tmp-dir\") pod \"auth-disabled-test-kserve-6b7b464494-npxg8\" (UID: \"8c4be60c-637f-47ea-ac7d-295f5b1dfb79\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8"
Sep 16 19:03:20.797833 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:20.797813    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-cabundle-cert\") pod \"auth-disabled-test-kserve-6b7b464494-npxg8\" (UID: \"8c4be60c-637f-47ea-ac7d-295f5b1dfb79\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8"
Sep 16 19:03:20.797931 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:20.797843    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87w5f\" (UniqueName: \"kubernetes.io/projected/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-kube-api-access-87w5f\") pod \"auth-disabled-test-kserve-6b7b464494-npxg8\" (UID: \"8c4be60c-637f-47ea-ac7d-295f5b1dfb79\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8"
Sep 16 19:03:20.797931 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:20.797864    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-tmp-dir\") pod \"auth-disabled-test-kserve-6b7b464494-npxg8\" (UID: \"8c4be60c-637f-47ea-ac7d-295f5b1dfb79\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8"
Sep 16 19:03:20.798029 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:20.797981    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-model-cache\") pod \"auth-disabled-test-kserve-6b7b464494-npxg8\" (UID: \"8c4be60c-637f-47ea-ac7d-295f5b1dfb79\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8"
Sep 16 19:03:20.798029 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:20.798018    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-kserve-provision-location\") pod \"auth-disabled-test-kserve-6b7b464494-npxg8\" (UID: \"8c4be60c-637f-47ea-ac7d-295f5b1dfb79\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8"
Sep 16 19:03:20.798109 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:20.798072    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-tls-certs\") pod \"auth-disabled-test-kserve-6b7b464494-npxg8\" (UID: \"8c4be60c-637f-47ea-ac7d-295f5b1dfb79\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8"
Sep 16 19:03:20.798156 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:20.798105    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-home\") pod \"auth-disabled-test-kserve-6b7b464494-npxg8\" (UID: \"8c4be60c-637f-47ea-ac7d-295f5b1dfb79\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8"
Sep 16 19:03:20.798156 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:20.798147    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-dshm\") pod \"auth-disabled-test-kserve-6b7b464494-npxg8\" (UID: \"8c4be60c-637f-47ea-ac7d-295f5b1dfb79\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8"
Sep 16 19:03:20.798304 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:20.798283    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-tmp-dir\") pod \"auth-disabled-test-kserve-6b7b464494-npxg8\" (UID: \"8c4be60c-637f-47ea-ac7d-295f5b1dfb79\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8"
Sep 16 19:03:20.798379 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:20.798357    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-kserve-provision-location\") pod \"auth-disabled-test-kserve-6b7b464494-npxg8\" (UID: \"8c4be60c-637f-47ea-ac7d-295f5b1dfb79\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8"
Sep 16 19:03:20.798439 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:20.798396    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-model-cache\") pod \"auth-disabled-test-kserve-6b7b464494-npxg8\" (UID: \"8c4be60c-637f-47ea-ac7d-295f5b1dfb79\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8"
Sep 16 19:03:20.798512 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:20.798488    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-home\") pod \"auth-disabled-test-kserve-6b7b464494-npxg8\" (UID: \"8c4be60c-637f-47ea-ac7d-295f5b1dfb79\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8"
Sep 16 19:03:20.798623 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:20.798607    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-cabundle-cert\") pod \"auth-disabled-test-kserve-6b7b464494-npxg8\" (UID: \"8c4be60c-637f-47ea-ac7d-295f5b1dfb79\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8"
Sep 16 19:03:20.800570 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:20.800546    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-dshm\") pod \"auth-disabled-test-kserve-6b7b464494-npxg8\" (UID: \"8c4be60c-637f-47ea-ac7d-295f5b1dfb79\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8"
Sep 16 19:03:20.800811 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:20.800793    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-tls-certs\") pod \"auth-disabled-test-kserve-6b7b464494-npxg8\" (UID: \"8c4be60c-637f-47ea-ac7d-295f5b1dfb79\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8"
Sep 16 19:03:20.809571 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:20.809550    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87w5f\" (UniqueName: \"kubernetes.io/projected/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-kube-api-access-87w5f\") pod \"auth-disabled-test-kserve-6b7b464494-npxg8\" (UID: \"8c4be60c-637f-47ea-ac7d-295f5b1dfb79\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8"
Sep 16 19:03:20.901740 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:20.901692    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8"
Sep 16 19:03:21.186223 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:21.186006    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8"]
Sep 16 19:03:21.189260 ip-10-0-128-247 kubenswrapper[2577]: W0916 19:03:21.189232    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c4be60c_637f_47ea_ac7d_295f5b1dfb79.slice/crio-0ee1f3ffbfe9b68768034d1ef91c44716022ca6d97c9e260ebfa59a41d3d7d3c WatchSource:0}: Error finding container 0ee1f3ffbfe9b68768034d1ef91c44716022ca6d97c9e260ebfa59a41d3d7d3c: Status 404 returned error can't find the container with id 0ee1f3ffbfe9b68768034d1ef91c44716022ca6d97c9e260ebfa59a41d3d7d3c
Sep 16 19:03:22.000905 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:22.000868    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8" event={"ID":"8c4be60c-637f-47ea-ac7d-295f5b1dfb79","Type":"ContainerStarted","Data":"311a14db9692c9e8216814a9be5b0b6d13f4cbd4c8df19064f115f3dbf98375a"}
Sep 16 19:03:22.001268 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:22.000924    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8" event={"ID":"8c4be60c-637f-47ea-ac7d-295f5b1dfb79","Type":"ContainerStarted","Data":"0ee1f3ffbfe9b68768034d1ef91c44716022ca6d97c9e260ebfa59a41d3d7d3c"}
Sep 16 19:03:24.011867 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:24.011821    2577 generic.go:356] "Generic (PLEG): container finished" podID="8c4be60c-637f-47ea-ac7d-295f5b1dfb79" containerID="311a14db9692c9e8216814a9be5b0b6d13f4cbd4c8df19064f115f3dbf98375a" exitCode=0
Sep 16 19:03:24.012342 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:24.011888    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8" event={"ID":"8c4be60c-637f-47ea-ac7d-295f5b1dfb79","Type":"ContainerDied","Data":"311a14db9692c9e8216814a9be5b0b6d13f4cbd4c8df19064f115f3dbf98375a"}
Sep 16 19:03:25.019819 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:25.019779    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8" event={"ID":"8c4be60c-637f-47ea-ac7d-295f5b1dfb79","Type":"ContainerStarted","Data":"acf5c64b2310710f695f1f6ee8c2f0f0877aab39d07093eca171712731a35818"}
Sep 16 19:03:25.056309 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:25.056251    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8" podStartSLOduration=2.989830006 podStartE2EDuration="5.056233133s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.066403127s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:03:25.05464352 +0000 UTC m=+1362.437532887" watchObservedRunningTime="2026-09-16 19:03:25.056233133 +0000 UTC m=+1362.439122498"
Sep 16 19:03:25.902624 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:25.902584    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8"
Sep 16 19:03:30.902456 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:30.902414    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8"
Sep 16 19:03:30.904109 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:30.904080    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8" podUID="8c4be60c-637f-47ea-ac7d-295f5b1dfb79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Sep 16 19:03:40.903023 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:40.902972    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8" podUID="8c4be60c-637f-47ea-ac7d-295f5b1dfb79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Sep 16 19:03:50.902926 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:03:50.902882    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8" podUID="8c4be60c-637f-47ea-ac7d-295f5b1dfb79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Sep 16 19:04:00.902349 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:04:00.902302    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8" podUID="8c4be60c-637f-47ea-ac7d-295f5b1dfb79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Sep 16 19:04:10.902456 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:04:10.902414    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8" podUID="8c4be60c-637f-47ea-ac7d-295f5b1dfb79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Sep 16 19:04:20.902699 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:04:20.902653    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8" podUID="8c4be60c-637f-47ea-ac7d-295f5b1dfb79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Sep 16 19:04:30.902599 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:04:30.902561    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8" podUID="8c4be60c-637f-47ea-ac7d-295f5b1dfb79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Sep 16 19:04:40.902646 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:04:40.902551    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8" podUID="8c4be60c-637f-47ea-ac7d-295f5b1dfb79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Sep 16 19:04:50.902577 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:04:50.902516    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8" podUID="8c4be60c-637f-47ea-ac7d-295f5b1dfb79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Sep 16 19:05:00.902918 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:00.902878    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8" podUID="8c4be60c-637f-47ea-ac7d-295f5b1dfb79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Sep 16 19:05:10.912300 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:10.912263    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8"
Sep 16 19:05:10.920192 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:10.920170    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8"
Sep 16 19:05:15.991211 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:15.991178    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8"]
Sep 16 19:05:28.442904 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:28.442867    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w"]
Sep 16 19:05:28.445595 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:28.445574    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w"
Sep 16 19:05:28.450390 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:28.450363    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 16 19:05:28.450390 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:28.450379    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 16 19:05:28.450390 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:28.450386    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-cvks6\""
Sep 16 19:05:28.450672 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:28.450397    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 16 19:05:28.457497 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:28.457479    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w"]
Sep 16 19:05:28.579388 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:28.579361    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da3cab48-3d0d-47e3-9a10-5043778ff7b0-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w\" (UID: \"da3cab48-3d0d-47e3-9a10-5043778ff7b0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w"
Sep 16 19:05:28.579587 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:28.579400    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da3cab48-3d0d-47e3-9a10-5043778ff7b0-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w\" (UID: \"da3cab48-3d0d-47e3-9a10-5043778ff7b0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w"
Sep 16 19:05:28.579587 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:28.579420    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da3cab48-3d0d-47e3-9a10-5043778ff7b0-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w\" (UID: \"da3cab48-3d0d-47e3-9a10-5043778ff7b0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w"
Sep 16 19:05:28.579587 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:28.579439    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da3cab48-3d0d-47e3-9a10-5043778ff7b0-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w\" (UID: \"da3cab48-3d0d-47e3-9a10-5043778ff7b0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w"
Sep 16 19:05:28.579587 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:28.579454    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqlc7\" (UniqueName: \"kubernetes.io/projected/da3cab48-3d0d-47e3-9a10-5043778ff7b0-kube-api-access-pqlc7\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w\" (UID: \"da3cab48-3d0d-47e3-9a10-5043778ff7b0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w"
Sep 16 19:05:28.579587 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:28.579555    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da3cab48-3d0d-47e3-9a10-5043778ff7b0-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w\" (UID: \"da3cab48-3d0d-47e3-9a10-5043778ff7b0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w"
Sep 16 19:05:28.680438 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:28.680407    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da3cab48-3d0d-47e3-9a10-5043778ff7b0-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w\" (UID: \"da3cab48-3d0d-47e3-9a10-5043778ff7b0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w"
Sep 16 19:05:28.680639 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:28.680483    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da3cab48-3d0d-47e3-9a10-5043778ff7b0-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w\" (UID: \"da3cab48-3d0d-47e3-9a10-5043778ff7b0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w"
Sep 16 19:05:28.680639 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:28.680512    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da3cab48-3d0d-47e3-9a10-5043778ff7b0-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w\" (UID: \"da3cab48-3d0d-47e3-9a10-5043778ff7b0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w"
Sep 16 19:05:28.680639 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:28.680558    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da3cab48-3d0d-47e3-9a10-5043778ff7b0-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w\" (UID: \"da3cab48-3d0d-47e3-9a10-5043778ff7b0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w"
Sep 16 19:05:28.680639 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:28.680581    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da3cab48-3d0d-47e3-9a10-5043778ff7b0-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w\" (UID: \"da3cab48-3d0d-47e3-9a10-5043778ff7b0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w"
Sep 16 19:05:28.680639 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:28.680605    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqlc7\" (UniqueName: \"kubernetes.io/projected/da3cab48-3d0d-47e3-9a10-5043778ff7b0-kube-api-access-pqlc7\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w\" (UID: \"da3cab48-3d0d-47e3-9a10-5043778ff7b0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w"
Sep 16 19:05:28.680885 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:28.680827    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da3cab48-3d0d-47e3-9a10-5043778ff7b0-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w\" (UID: \"da3cab48-3d0d-47e3-9a10-5043778ff7b0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w"
Sep 16 19:05:28.680943 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:28.680897    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da3cab48-3d0d-47e3-9a10-5043778ff7b0-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w\" (UID: \"da3cab48-3d0d-47e3-9a10-5043778ff7b0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w"
Sep 16 19:05:28.681001 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:28.680983    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da3cab48-3d0d-47e3-9a10-5043778ff7b0-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w\" (UID: \"da3cab48-3d0d-47e3-9a10-5043778ff7b0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w"
Sep 16 19:05:28.683120 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:28.683093    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da3cab48-3d0d-47e3-9a10-5043778ff7b0-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w\" (UID: \"da3cab48-3d0d-47e3-9a10-5043778ff7b0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w"
Sep 16 19:05:28.683227 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:28.683131    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da3cab48-3d0d-47e3-9a10-5043778ff7b0-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w\" (UID: \"da3cab48-3d0d-47e3-9a10-5043778ff7b0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w"
Sep 16 19:05:28.691596 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:28.691574    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqlc7\" (UniqueName: \"kubernetes.io/projected/da3cab48-3d0d-47e3-9a10-5043778ff7b0-kube-api-access-pqlc7\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w\" (UID: \"da3cab48-3d0d-47e3-9a10-5043778ff7b0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w"
Sep 16 19:05:28.756163 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:28.756106    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w"
Sep 16 19:05:29.033317 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:29.033294    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w"]
Sep 16 19:05:29.035316 ip-10-0-128-247 kubenswrapper[2577]: W0916 19:05:29.035289    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda3cab48_3d0d_47e3_9a10_5043778ff7b0.slice/crio-a1e7eae6884c86e2f2c745d5be38e6e2ff0939e5aaed9091130e4d34eb226f50 WatchSource:0}: Error finding container a1e7eae6884c86e2f2c745d5be38e6e2ff0939e5aaed9091130e4d34eb226f50: Status 404 returned error can't find the container with id a1e7eae6884c86e2f2c745d5be38e6e2ff0939e5aaed9091130e4d34eb226f50
Sep 16 19:05:29.553859 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:29.553827    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w" event={"ID":"da3cab48-3d0d-47e3-9a10-5043778ff7b0","Type":"ContainerStarted","Data":"65564e7a4548f6c345d0f64d0fe833e83eb4141a88c91a468b8c82e20acc9bd1"}
Sep 16 19:05:29.554302 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:29.553879    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w" event={"ID":"da3cab48-3d0d-47e3-9a10-5043778ff7b0","Type":"ContainerStarted","Data":"a1e7eae6884c86e2f2c745d5be38e6e2ff0939e5aaed9091130e4d34eb226f50"}
Sep 16 19:05:29.576897 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:29.576851    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w" podStartSLOduration=1.576837211 podStartE2EDuration="1.576837211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:29.574060954 +0000 UTC m=+1486.956950318" watchObservedRunningTime="2026-09-16 19:05:29.576837211 +0000 UTC m=+1486.959726575"
Sep 16 19:05:29.757213 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:29.757177    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w"
Sep 16 19:05:31.014097 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:31.014057    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8" podUID="8c4be60c-637f-47ea-ac7d-295f5b1dfb79" containerName="main" containerID="cri-o://acf5c64b2310710f695f1f6ee8c2f0f0877aab39d07093eca171712731a35818" gracePeriod=45
Sep 16 19:05:31.564372 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:31.564332    2577 generic.go:356] "Generic (PLEG): container finished" podID="8c4be60c-637f-47ea-ac7d-295f5b1dfb79" containerID="acf5c64b2310710f695f1f6ee8c2f0f0877aab39d07093eca171712731a35818" exitCode=0
Sep 16 19:05:31.564509 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:31.564401    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8" event={"ID":"8c4be60c-637f-47ea-ac7d-295f5b1dfb79","Type":"ContainerDied","Data":"acf5c64b2310710f695f1f6ee8c2f0f0877aab39d07093eca171712731a35818"}
Sep 16 19:05:31.671373 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:31.671352    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8"
Sep 16 19:05:31.806310 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:31.806239    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-home\") pod \"8c4be60c-637f-47ea-ac7d-295f5b1dfb79\" (UID: \"8c4be60c-637f-47ea-ac7d-295f5b1dfb79\") "
Sep 16 19:05:31.806310 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:31.806294    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-tls-certs\") pod \"8c4be60c-637f-47ea-ac7d-295f5b1dfb79\" (UID: \"8c4be60c-637f-47ea-ac7d-295f5b1dfb79\") "
Sep 16 19:05:31.806503 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:31.806330    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-kserve-provision-location\") pod \"8c4be60c-637f-47ea-ac7d-295f5b1dfb79\" (UID: \"8c4be60c-637f-47ea-ac7d-295f5b1dfb79\") "
Sep 16 19:05:31.806503 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:31.806353    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87w5f\" (UniqueName: \"kubernetes.io/projected/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-kube-api-access-87w5f\") pod \"8c4be60c-637f-47ea-ac7d-295f5b1dfb79\" (UID: \"8c4be60c-637f-47ea-ac7d-295f5b1dfb79\") "
Sep 16 19:05:31.806503 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:31.806378    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-tmp-dir\") pod \"8c4be60c-637f-47ea-ac7d-295f5b1dfb79\" (UID: \"8c4be60c-637f-47ea-ac7d-295f5b1dfb79\") "
Sep 16 19:05:31.806503 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:31.806425    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-cabundle-cert\") pod \"8c4be60c-637f-47ea-ac7d-295f5b1dfb79\" (UID: \"8c4be60c-637f-47ea-ac7d-295f5b1dfb79\") "
Sep 16 19:05:31.806503 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:31.806450    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-model-cache\") pod \"8c4be60c-637f-47ea-ac7d-295f5b1dfb79\" (UID: \"8c4be60c-637f-47ea-ac7d-295f5b1dfb79\") "
Sep 16 19:05:31.806503 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:31.806485    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-dshm\") pod \"8c4be60c-637f-47ea-ac7d-295f5b1dfb79\" (UID: \"8c4be60c-637f-47ea-ac7d-295f5b1dfb79\") "
Sep 16 19:05:31.806856 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:31.806821    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-model-cache" (OuterVolumeSpecName: "model-cache") pod "8c4be60c-637f-47ea-ac7d-295f5b1dfb79" (UID: "8c4be60c-637f-47ea-ac7d-295f5b1dfb79"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:05:31.806919 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:31.806907    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8c4be60c-637f-47ea-ac7d-295f5b1dfb79" (UID: "8c4be60c-637f-47ea-ac7d-295f5b1dfb79"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:05:31.806975 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:31.806946    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-home" (OuterVolumeSpecName: "home") pod "8c4be60c-637f-47ea-ac7d-295f5b1dfb79" (UID: "8c4be60c-637f-47ea-ac7d-295f5b1dfb79"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:05:31.809160 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:31.809129    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-dshm" (OuterVolumeSpecName: "dshm") pod "8c4be60c-637f-47ea-ac7d-295f5b1dfb79" (UID: "8c4be60c-637f-47ea-ac7d-295f5b1dfb79"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:05:31.809287 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:31.809263    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8c4be60c-637f-47ea-ac7d-295f5b1dfb79" (UID: "8c4be60c-637f-47ea-ac7d-295f5b1dfb79"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:05:31.809354 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:31.809311    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-kube-api-access-87w5f" (OuterVolumeSpecName: "kube-api-access-87w5f") pod "8c4be60c-637f-47ea-ac7d-295f5b1dfb79" (UID: "8c4be60c-637f-47ea-ac7d-295f5b1dfb79"). InnerVolumeSpecName "kube-api-access-87w5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:05:31.820917 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:31.820894    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8c4be60c-637f-47ea-ac7d-295f5b1dfb79" (UID: "8c4be60c-637f-47ea-ac7d-295f5b1dfb79"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:05:31.829619 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:31.829596    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8c4be60c-637f-47ea-ac7d-295f5b1dfb79" (UID: "8c4be60c-637f-47ea-ac7d-295f5b1dfb79"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:05:31.907172 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:31.907153    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-cabundle-cert\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:05:31.907172 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:31.907172    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-model-cache\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:05:31.907273 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:31.907180    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-dshm\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:05:31.907273 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:31.907187    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-home\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:05:31.907273 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:31.907194    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-tls-certs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:05:31.907273 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:31.907201    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-kserve-provision-location\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:05:31.907273 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:31.907208    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87w5f\" (UniqueName: \"kubernetes.io/projected/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-kube-api-access-87w5f\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:05:31.907273 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:31.907216    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c4be60c-637f-47ea-ac7d-295f5b1dfb79-tmp-dir\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:05:32.570939 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:32.570905    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8" event={"ID":"8c4be60c-637f-47ea-ac7d-295f5b1dfb79","Type":"ContainerDied","Data":"0ee1f3ffbfe9b68768034d1ef91c44716022ca6d97c9e260ebfa59a41d3d7d3c"}
Sep 16 19:05:32.570939 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:32.570939    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8"
Sep 16 19:05:32.571383 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:32.570979    2577 scope.go:117] "RemoveContainer" containerID="acf5c64b2310710f695f1f6ee8c2f0f0877aab39d07093eca171712731a35818"
Sep 16 19:05:32.580421 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:32.580403    2577 scope.go:117] "RemoveContainer" containerID="311a14db9692c9e8216814a9be5b0b6d13f4cbd4c8df19064f115f3dbf98375a"
Sep 16 19:05:32.597358 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:32.597323    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8"]
Sep 16 19:05:32.604159 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:32.604136    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6b7b464494-npxg8"]
Sep 16 19:05:33.154669 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:33.154640    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c4be60c-637f-47ea-ac7d-295f5b1dfb79" path="/var/lib/kubelet/pods/8c4be60c-637f-47ea-ac7d-295f5b1dfb79/volumes"
Sep 16 19:05:38.757208 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:38.757173    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w"
Sep 16 19:05:38.769634 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:38.769612    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w"
Sep 16 19:05:39.618433 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:05:39.618398    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w"
Sep 16 19:06:52.615587 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:06:52.615552    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w"]
Sep 16 19:06:52.625613 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:06:52.625575    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:06:52.625613 ip-10-0-128-247 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:06:52.625613 ip-10-0-128-247 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 19:06:52.625613 ip-10-0-128-247 kubenswrapper[2577]:  > containerID="65564e7a4548f6c345d0f64d0fe833e83eb4141a88c91a468b8c82e20acc9bd1" cmd=["/bin/sleep","15"]
Sep 16 19:06:52.625841 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:06:52.625635    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w" message="" err=<
Sep 16 19:06:52.625841 ip-10-0-128-247 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:06:52.625841 ip-10-0-128-247 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 19:06:52.625841 ip-10-0-128-247 kubenswrapper[2577]:  >
Sep 16 19:06:52.625841 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:06:52.625650    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:06:52.625841 ip-10-0-128-247 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:06:52.625841 ip-10-0-128-247 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 19:06:52.625841 ip-10-0-128-247 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w" podUID="da3cab48-3d0d-47e3-9a10-5043778ff7b0" containerName="main" containerID="cri-o://65564e7a4548f6c345d0f64d0fe833e83eb4141a88c91a468b8c82e20acc9bd1"
Sep 16 19:06:52.625841 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:06:52.625702    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w" podUID="da3cab48-3d0d-47e3-9a10-5043778ff7b0" containerName="main" containerID="cri-o://65564e7a4548f6c345d0f64d0fe833e83eb4141a88c91a468b8c82e20acc9bd1" gracePeriod=60
Sep 16 19:06:52.914490 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:06:52.914458    2577 generic.go:356] "Generic (PLEG): container finished" podID="da3cab48-3d0d-47e3-9a10-5043778ff7b0" containerID="65564e7a4548f6c345d0f64d0fe833e83eb4141a88c91a468b8c82e20acc9bd1" exitCode=0
Sep 16 19:06:52.914637 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:06:52.914557    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w" event={"ID":"da3cab48-3d0d-47e3-9a10-5043778ff7b0","Type":"ContainerDied","Data":"65564e7a4548f6c345d0f64d0fe833e83eb4141a88c91a468b8c82e20acc9bd1"}
Sep 16 19:06:52.961316 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:06:52.961286    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w"
Sep 16 19:06:53.115106 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:06:53.115080    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da3cab48-3d0d-47e3-9a10-5043778ff7b0-tmp-dir\") pod \"da3cab48-3d0d-47e3-9a10-5043778ff7b0\" (UID: \"da3cab48-3d0d-47e3-9a10-5043778ff7b0\") "
Sep 16 19:06:53.115106 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:06:53.115108    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da3cab48-3d0d-47e3-9a10-5043778ff7b0-home\") pod \"da3cab48-3d0d-47e3-9a10-5043778ff7b0\" (UID: \"da3cab48-3d0d-47e3-9a10-5043778ff7b0\") "
Sep 16 19:06:53.115300 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:06:53.115130    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da3cab48-3d0d-47e3-9a10-5043778ff7b0-tls-certs\") pod \"da3cab48-3d0d-47e3-9a10-5043778ff7b0\" (UID: \"da3cab48-3d0d-47e3-9a10-5043778ff7b0\") "
Sep 16 19:06:53.115300 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:06:53.115172    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da3cab48-3d0d-47e3-9a10-5043778ff7b0-dshm\") pod \"da3cab48-3d0d-47e3-9a10-5043778ff7b0\" (UID: \"da3cab48-3d0d-47e3-9a10-5043778ff7b0\") "
Sep 16 19:06:53.115300 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:06:53.115197    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqlc7\" (UniqueName: \"kubernetes.io/projected/da3cab48-3d0d-47e3-9a10-5043778ff7b0-kube-api-access-pqlc7\") pod \"da3cab48-3d0d-47e3-9a10-5043778ff7b0\" (UID: \"da3cab48-3d0d-47e3-9a10-5043778ff7b0\") "
Sep 16 19:06:53.115300 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:06:53.115240    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da3cab48-3d0d-47e3-9a10-5043778ff7b0-model-cache\") pod \"da3cab48-3d0d-47e3-9a10-5043778ff7b0\" (UID: \"da3cab48-3d0d-47e3-9a10-5043778ff7b0\") "
Sep 16 19:06:53.115508 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:06:53.115358    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da3cab48-3d0d-47e3-9a10-5043778ff7b0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "da3cab48-3d0d-47e3-9a10-5043778ff7b0" (UID: "da3cab48-3d0d-47e3-9a10-5043778ff7b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:06:53.115508 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:06:53.115371    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da3cab48-3d0d-47e3-9a10-5043778ff7b0-home" (OuterVolumeSpecName: "home") pod "da3cab48-3d0d-47e3-9a10-5043778ff7b0" (UID: "da3cab48-3d0d-47e3-9a10-5043778ff7b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:06:53.115633 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:06:53.115514    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da3cab48-3d0d-47e3-9a10-5043778ff7b0-tmp-dir\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:06:53.115633 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:06:53.115559    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da3cab48-3d0d-47e3-9a10-5043778ff7b0-home\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:06:53.115700 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:06:53.115663    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da3cab48-3d0d-47e3-9a10-5043778ff7b0-model-cache" (OuterVolumeSpecName: "model-cache") pod "da3cab48-3d0d-47e3-9a10-5043778ff7b0" (UID: "da3cab48-3d0d-47e3-9a10-5043778ff7b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:06:53.117747 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:06:53.117717    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da3cab48-3d0d-47e3-9a10-5043778ff7b0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "da3cab48-3d0d-47e3-9a10-5043778ff7b0" (UID: "da3cab48-3d0d-47e3-9a10-5043778ff7b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:06:53.117898 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:06:53.117791    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da3cab48-3d0d-47e3-9a10-5043778ff7b0-dshm" (OuterVolumeSpecName: "dshm") pod "da3cab48-3d0d-47e3-9a10-5043778ff7b0" (UID: "da3cab48-3d0d-47e3-9a10-5043778ff7b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:06:53.117949 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:06:53.117893    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da3cab48-3d0d-47e3-9a10-5043778ff7b0-kube-api-access-pqlc7" (OuterVolumeSpecName: "kube-api-access-pqlc7") pod "da3cab48-3d0d-47e3-9a10-5043778ff7b0" (UID: "da3cab48-3d0d-47e3-9a10-5043778ff7b0"). InnerVolumeSpecName "kube-api-access-pqlc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:06:53.216285 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:06:53.216260    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da3cab48-3d0d-47e3-9a10-5043778ff7b0-tls-certs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:06:53.216285 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:06:53.216285    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da3cab48-3d0d-47e3-9a10-5043778ff7b0-dshm\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:06:53.216414 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:06:53.216294    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqlc7\" (UniqueName: \"kubernetes.io/projected/da3cab48-3d0d-47e3-9a10-5043778ff7b0-kube-api-access-pqlc7\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:06:53.216414 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:06:53.216304    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da3cab48-3d0d-47e3-9a10-5043778ff7b0-model-cache\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:06:53.920135 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:06:53.920099    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w" event={"ID":"da3cab48-3d0d-47e3-9a10-5043778ff7b0","Type":"ContainerDied","Data":"a1e7eae6884c86e2f2c745d5be38e6e2ff0939e5aaed9091130e4d34eb226f50"}
Sep 16 19:06:53.920135 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:06:53.920134    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w"
Sep 16 19:06:53.920665 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:06:53.920162    2577 scope.go:117] "RemoveContainer" containerID="65564e7a4548f6c345d0f64d0fe833e83eb4141a88c91a468b8c82e20acc9bd1"
Sep 16 19:06:53.946274 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:06:53.946240    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w"]
Sep 16 19:06:53.955243 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:06:53.955221    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-h5r4w"]
Sep 16 19:06:55.155007 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:06:55.154974    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da3cab48-3d0d-47e3-9a10-5043778ff7b0" path="/var/lib/kubelet/pods/da3cab48-3d0d-47e3-9a10-5043778ff7b0/volumes"
Sep 16 19:07:45.342190 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:07:45.342152    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq"]
Sep 16 19:07:45.342984 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:07:45.342960    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da3cab48-3d0d-47e3-9a10-5043778ff7b0" containerName="main"
Sep 16 19:07:45.343074 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:07:45.342987    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="da3cab48-3d0d-47e3-9a10-5043778ff7b0" containerName="main"
Sep 16 19:07:45.343074 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:07:45.343021    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c4be60c-637f-47ea-ac7d-295f5b1dfb79" containerName="main"
Sep 16 19:07:45.343074 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:07:45.343030    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c4be60c-637f-47ea-ac7d-295f5b1dfb79" containerName="main"
Sep 16 19:07:45.343074 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:07:45.343041    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c4be60c-637f-47ea-ac7d-295f5b1dfb79" containerName="storage-initializer"
Sep 16 19:07:45.343074 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:07:45.343049    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c4be60c-637f-47ea-ac7d-295f5b1dfb79" containerName="storage-initializer"
Sep 16 19:07:45.343293 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:07:45.343214    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="da3cab48-3d0d-47e3-9a10-5043778ff7b0" containerName="main"
Sep 16 19:07:45.343293 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:07:45.343238    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c4be60c-637f-47ea-ac7d-295f5b1dfb79" containerName="main"
Sep 16 19:07:45.349304 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:07:45.349277    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq"
Sep 16 19:07:45.352407 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:07:45.352373    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 16 19:07:45.353767 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:07:45.353748    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-67jn8\""
Sep 16 19:07:45.353829 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:07:45.353767    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 16 19:07:45.353829 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:07:45.353785    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 16 19:07:45.357321 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:07:45.357299    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq"]
Sep 16 19:07:45.402050 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:07:45.402019    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e28674b2-3e97-4240-aeb2-024a332b4caf-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq\" (UID: \"e28674b2-3e97-4240-aeb2-024a332b4caf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq"
Sep 16 19:07:45.402188 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:07:45.402086    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lvg9\" (UniqueName: \"kubernetes.io/projected/e28674b2-3e97-4240-aeb2-024a332b4caf-kube-api-access-2lvg9\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq\" (UID: \"e28674b2-3e97-4240-aeb2-024a332b4caf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq"
Sep 16 19:07:45.402188 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:07:45.402107    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e28674b2-3e97-4240-aeb2-024a332b4caf-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq\" (UID: \"e28674b2-3e97-4240-aeb2-024a332b4caf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq"
Sep 16 19:07:45.402188 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:07:45.402124    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e28674b2-3e97-4240-aeb2-024a332b4caf-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq\" (UID: \"e28674b2-3e97-4240-aeb2-024a332b4caf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq"
Sep 16 19:07:45.402188 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:07:45.402152    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e28674b2-3e97-4240-aeb2-024a332b4caf-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq\" (UID: \"e28674b2-3e97-4240-aeb2-024a332b4caf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq"
Sep 16 19:07:45.402376 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:07:45.402196    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e28674b2-3e97-4240-aeb2-024a332b4caf-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq\" (UID: \"e28674b2-3e97-4240-aeb2-024a332b4caf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq"
Sep 16 19:07:45.503549 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:07:45.503506    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e28674b2-3e97-4240-aeb2-024a332b4caf-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq\" (UID: \"e28674b2-3e97-4240-aeb2-024a332b4caf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq"
Sep 16 19:07:45.503683 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:07:45.503551    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e28674b2-3e97-4240-aeb2-024a332b4caf-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq\" (UID: \"e28674b2-3e97-4240-aeb2-024a332b4caf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq"
Sep 16 19:07:45.503683 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:07:45.503663    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e28674b2-3e97-4240-aeb2-024a332b4caf-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq\" (UID: \"e28674b2-3e97-4240-aeb2-024a332b4caf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq"
Sep 16 19:07:45.503792 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:07:45.503745    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lvg9\" (UniqueName: \"kubernetes.io/projected/e28674b2-3e97-4240-aeb2-024a332b4caf-kube-api-access-2lvg9\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq\" (UID: \"e28674b2-3e97-4240-aeb2-024a332b4caf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq"
Sep 16 19:07:45.503792 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:07:45.503768    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e28674b2-3e97-4240-aeb2-024a332b4caf-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq\" (UID: \"e28674b2-3e97-4240-aeb2-024a332b4caf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq"
Sep 16 19:07:45.503792 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:07:45.503787    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e28674b2-3e97-4240-aeb2-024a332b4caf-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq\" (UID: \"e28674b2-3e97-4240-aeb2-024a332b4caf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq"
Sep 16 19:07:45.503933 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:07:45.503886    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e28674b2-3e97-4240-aeb2-024a332b4caf-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq\" (UID: \"e28674b2-3e97-4240-aeb2-024a332b4caf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq"
Sep 16 19:07:45.504104 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:07:45.504086    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e28674b2-3e97-4240-aeb2-024a332b4caf-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq\" (UID: \"e28674b2-3e97-4240-aeb2-024a332b4caf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq"
Sep 16 19:07:45.504247 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:07:45.504227    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e28674b2-3e97-4240-aeb2-024a332b4caf-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq\" (UID: \"e28674b2-3e97-4240-aeb2-024a332b4caf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq"
Sep 16 19:07:45.505921 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:07:45.505896    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e28674b2-3e97-4240-aeb2-024a332b4caf-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq\" (UID: \"e28674b2-3e97-4240-aeb2-024a332b4caf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq"
Sep 16 19:07:45.506228 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:07:45.506209    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e28674b2-3e97-4240-aeb2-024a332b4caf-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq\" (UID: \"e28674b2-3e97-4240-aeb2-024a332b4caf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq"
Sep 16 19:07:45.513812 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:07:45.513790    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lvg9\" (UniqueName: \"kubernetes.io/projected/e28674b2-3e97-4240-aeb2-024a332b4caf-kube-api-access-2lvg9\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq\" (UID: \"e28674b2-3e97-4240-aeb2-024a332b4caf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq"
Sep 16 19:07:45.661275 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:07:45.661238    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq"
Sep 16 19:07:45.938010 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:07:45.937975    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq"]
Sep 16 19:07:45.940249 ip-10-0-128-247 kubenswrapper[2577]: W0916 19:07:45.940210    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode28674b2_3e97_4240_aeb2_024a332b4caf.slice/crio-a5f9f096242b07ed7457168e0871b44be1c40706c8a40e191487ce322906d069 WatchSource:0}: Error finding container a5f9f096242b07ed7457168e0871b44be1c40706c8a40e191487ce322906d069: Status 404 returned error can't find the container with id a5f9f096242b07ed7457168e0871b44be1c40706c8a40e191487ce322906d069
Sep 16 19:07:46.148715 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:07:46.148684    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq" event={"ID":"e28674b2-3e97-4240-aeb2-024a332b4caf","Type":"ContainerStarted","Data":"563e9b96cb56c26eb04f3f1bcd48b6c7851ca1070ff04156dd527808adb66049"}
Sep 16 19:07:46.148852 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:07:46.148738    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq" event={"ID":"e28674b2-3e97-4240-aeb2-024a332b4caf","Type":"ContainerStarted","Data":"a5f9f096242b07ed7457168e0871b44be1c40706c8a40e191487ce322906d069"}
Sep 16 19:07:46.167992 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:07:46.167953    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq" podStartSLOduration=1.167939255 podStartE2EDuration="1.167939255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:46.167506011 +0000 UTC m=+1623.550395379" watchObservedRunningTime="2026-09-16 19:07:46.167939255 +0000 UTC m=+1623.550828671"
Sep 16 19:07:46.661989 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:07:46.661956    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq"
Sep 16 19:07:55.662078 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:07:55.662043    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq"
Sep 16 19:07:55.674639 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:07:55.674616    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq"
Sep 16 19:07:56.198731 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:07:56.198704    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq"
Sep 16 19:08:24.032718 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:08:24.032680    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq"]
Sep 16 19:08:24.041573 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:08:24.041519    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:08:24.041573 ip-10-0-128-247 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:08:24.041573 ip-10-0-128-247 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 19:08:24.041573 ip-10-0-128-247 kubenswrapper[2577]:  > containerID="563e9b96cb56c26eb04f3f1bcd48b6c7851ca1070ff04156dd527808adb66049" cmd=["/bin/sleep","15"]
Sep 16 19:08:24.041839 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:08:24.041588    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq" message="" err=<
Sep 16 19:08:24.041839 ip-10-0-128-247 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:08:24.041839 ip-10-0-128-247 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 19:08:24.041839 ip-10-0-128-247 kubenswrapper[2577]:  >
Sep 16 19:08:24.041839 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:08:24.041600    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:08:24.041839 ip-10-0-128-247 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:08:24.041839 ip-10-0-128-247 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 19:08:24.041839 ip-10-0-128-247 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq" podUID="e28674b2-3e97-4240-aeb2-024a332b4caf" containerName="main" containerID="cri-o://563e9b96cb56c26eb04f3f1bcd48b6c7851ca1070ff04156dd527808adb66049"
Sep 16 19:08:24.041839 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:08:24.041634    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq" podUID="e28674b2-3e97-4240-aeb2-024a332b4caf" containerName="main" containerID="cri-o://563e9b96cb56c26eb04f3f1bcd48b6c7851ca1070ff04156dd527808adb66049" gracePeriod=60
Sep 16 19:08:24.188025 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:08:24.187992    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq" podUID="e28674b2-3e97-4240-aeb2-024a332b4caf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Sep 16 19:08:24.301201 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:08:24.301123    2577 generic.go:356] "Generic (PLEG): container finished" podID="e28674b2-3e97-4240-aeb2-024a332b4caf" containerID="563e9b96cb56c26eb04f3f1bcd48b6c7851ca1070ff04156dd527808adb66049" exitCode=0
Sep 16 19:08:24.301201 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:08:24.301191    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq" event={"ID":"e28674b2-3e97-4240-aeb2-024a332b4caf","Type":"ContainerDied","Data":"563e9b96cb56c26eb04f3f1bcd48b6c7851ca1070ff04156dd527808adb66049"}
Sep 16 19:08:24.391332 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:08:24.391305    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq"
Sep 16 19:08:24.496752 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:08:24.496729    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e28674b2-3e97-4240-aeb2-024a332b4caf-tmp-dir\") pod \"e28674b2-3e97-4240-aeb2-024a332b4caf\" (UID: \"e28674b2-3e97-4240-aeb2-024a332b4caf\") "
Sep 16 19:08:24.496898 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:08:24.496773    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2lvg9\" (UniqueName: \"kubernetes.io/projected/e28674b2-3e97-4240-aeb2-024a332b4caf-kube-api-access-2lvg9\") pod \"e28674b2-3e97-4240-aeb2-024a332b4caf\" (UID: \"e28674b2-3e97-4240-aeb2-024a332b4caf\") "
Sep 16 19:08:24.496898 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:08:24.496806    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e28674b2-3e97-4240-aeb2-024a332b4caf-tls-certs\") pod \"e28674b2-3e97-4240-aeb2-024a332b4caf\" (UID: \"e28674b2-3e97-4240-aeb2-024a332b4caf\") "
Sep 16 19:08:24.496898 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:08:24.496852    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e28674b2-3e97-4240-aeb2-024a332b4caf-model-cache\") pod \"e28674b2-3e97-4240-aeb2-024a332b4caf\" (UID: \"e28674b2-3e97-4240-aeb2-024a332b4caf\") "
Sep 16 19:08:24.496898 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:08:24.496887    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e28674b2-3e97-4240-aeb2-024a332b4caf-dshm\") pod \"e28674b2-3e97-4240-aeb2-024a332b4caf\" (UID: \"e28674b2-3e97-4240-aeb2-024a332b4caf\") "
Sep 16 19:08:24.497121 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:08:24.496953    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e28674b2-3e97-4240-aeb2-024a332b4caf-home\") pod \"e28674b2-3e97-4240-aeb2-024a332b4caf\" (UID: \"e28674b2-3e97-4240-aeb2-024a332b4caf\") "
Sep 16 19:08:24.497121 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:08:24.496977    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e28674b2-3e97-4240-aeb2-024a332b4caf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e28674b2-3e97-4240-aeb2-024a332b4caf" (UID: "e28674b2-3e97-4240-aeb2-024a332b4caf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:24.497232 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:08:24.497130    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e28674b2-3e97-4240-aeb2-024a332b4caf-model-cache" (OuterVolumeSpecName: "model-cache") pod "e28674b2-3e97-4240-aeb2-024a332b4caf" (UID: "e28674b2-3e97-4240-aeb2-024a332b4caf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:24.497232 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:08:24.497200    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e28674b2-3e97-4240-aeb2-024a332b4caf-model-cache\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:08:24.497232 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:08:24.497214    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e28674b2-3e97-4240-aeb2-024a332b4caf-tmp-dir\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:08:24.497383 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:08:24.497246    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e28674b2-3e97-4240-aeb2-024a332b4caf-home" (OuterVolumeSpecName: "home") pod "e28674b2-3e97-4240-aeb2-024a332b4caf" (UID: "e28674b2-3e97-4240-aeb2-024a332b4caf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:24.499131 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:08:24.499102    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e28674b2-3e97-4240-aeb2-024a332b4caf-dshm" (OuterVolumeSpecName: "dshm") pod "e28674b2-3e97-4240-aeb2-024a332b4caf" (UID: "e28674b2-3e97-4240-aeb2-024a332b4caf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:24.499252 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:08:24.499185    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e28674b2-3e97-4240-aeb2-024a332b4caf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e28674b2-3e97-4240-aeb2-024a332b4caf" (UID: "e28674b2-3e97-4240-aeb2-024a332b4caf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:08:24.499252 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:08:24.499214    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e28674b2-3e97-4240-aeb2-024a332b4caf-kube-api-access-2lvg9" (OuterVolumeSpecName: "kube-api-access-2lvg9") pod "e28674b2-3e97-4240-aeb2-024a332b4caf" (UID: "e28674b2-3e97-4240-aeb2-024a332b4caf"). InnerVolumeSpecName "kube-api-access-2lvg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:08:24.597643 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:08:24.597586    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e28674b2-3e97-4240-aeb2-024a332b4caf-home\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:08:24.597643 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:08:24.597607    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lvg9\" (UniqueName: \"kubernetes.io/projected/e28674b2-3e97-4240-aeb2-024a332b4caf-kube-api-access-2lvg9\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:08:24.597643 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:08:24.597618    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e28674b2-3e97-4240-aeb2-024a332b4caf-tls-certs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:08:24.597643 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:08:24.597628    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e28674b2-3e97-4240-aeb2-024a332b4caf-dshm\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:08:25.306822 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:08:25.306797    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq"
Sep 16 19:08:25.307193 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:08:25.306833    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq" event={"ID":"e28674b2-3e97-4240-aeb2-024a332b4caf","Type":"ContainerDied","Data":"a5f9f096242b07ed7457168e0871b44be1c40706c8a40e191487ce322906d069"}
Sep 16 19:08:25.307193 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:08:25.306893    2577 scope.go:117] "RemoveContainer" containerID="563e9b96cb56c26eb04f3f1bcd48b6c7851ca1070ff04156dd527808adb66049"
Sep 16 19:08:25.335735 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:08:25.335710    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq"]
Sep 16 19:08:25.341014 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:08:25.340991    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444rp9lq"]
Sep 16 19:08:27.154623 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:08:27.154589    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e28674b2-3e97-4240-aeb2-024a332b4caf" path="/var/lib/kubelet/pods/e28674b2-3e97-4240-aeb2-024a332b4caf/volumes"
Sep 16 19:09:38.076901 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.076870    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449"]
Sep 16 19:09:38.077337 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.077198    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e28674b2-3e97-4240-aeb2-024a332b4caf" containerName="main"
Sep 16 19:09:38.077337 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.077210    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e28674b2-3e97-4240-aeb2-024a332b4caf" containerName="main"
Sep 16 19:09:38.077337 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.077279    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e28674b2-3e97-4240-aeb2-024a332b4caf" containerName="main"
Sep 16 19:09:38.080481 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.080465    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449"
Sep 16 19:09:38.083392 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.083365    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:09:38.083554 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.083370    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 16 19:09:38.083554 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.083374    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-zjzwv\""
Sep 16 19:09:38.083554 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.083414    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 16 19:09:38.083554 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.083489    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 16 19:09:38.084714 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.084677    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 16 19:09:38.093305 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.093280    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449"]
Sep 16 19:09:38.219276 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.219252    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d546b01-cf39-465c-b610-8c314ca761cd-model-cache\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-89449\" (UID: \"9d546b01-cf39-465c-b610-8c314ca761cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449"
Sep 16 19:09:38.219381 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.219278    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d546b01-cf39-465c-b610-8c314ca761cd-home\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-89449\" (UID: \"9d546b01-cf39-465c-b610-8c314ca761cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449"
Sep 16 19:09:38.219381 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.219297    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d546b01-cf39-465c-b610-8c314ca761cd-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-89449\" (UID: \"9d546b01-cf39-465c-b610-8c314ca761cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449"
Sep 16 19:09:38.219381 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.219339    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d546b01-cf39-465c-b610-8c314ca761cd-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-89449\" (UID: \"9d546b01-cf39-465c-b610-8c314ca761cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449"
Sep 16 19:09:38.219491 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.219386    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d546b01-cf39-465c-b610-8c314ca761cd-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-89449\" (UID: \"9d546b01-cf39-465c-b610-8c314ca761cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449"
Sep 16 19:09:38.219491 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.219417    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d546b01-cf39-465c-b610-8c314ca761cd-dshm\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-89449\" (UID: \"9d546b01-cf39-465c-b610-8c314ca761cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449"
Sep 16 19:09:38.219491 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.219435    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8c46\" (UniqueName: \"kubernetes.io/projected/9d546b01-cf39-465c-b610-8c314ca761cd-kube-api-access-m8c46\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-89449\" (UID: \"9d546b01-cf39-465c-b610-8c314ca761cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449"
Sep 16 19:09:38.219491 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.219459    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d546b01-cf39-465c-b610-8c314ca761cd-tls-certs\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-89449\" (UID: \"9d546b01-cf39-465c-b610-8c314ca761cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449"
Sep 16 19:09:38.320354 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.320328    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d546b01-cf39-465c-b610-8c314ca761cd-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-89449\" (UID: \"9d546b01-cf39-465c-b610-8c314ca761cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449"
Sep 16 19:09:38.320456 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.320357    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d546b01-cf39-465c-b610-8c314ca761cd-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-89449\" (UID: \"9d546b01-cf39-465c-b610-8c314ca761cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449"
Sep 16 19:09:38.320456 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.320373    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d546b01-cf39-465c-b610-8c314ca761cd-dshm\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-89449\" (UID: \"9d546b01-cf39-465c-b610-8c314ca761cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449"
Sep 16 19:09:38.320456 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.320389    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8c46\" (UniqueName: \"kubernetes.io/projected/9d546b01-cf39-465c-b610-8c314ca761cd-kube-api-access-m8c46\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-89449\" (UID: \"9d546b01-cf39-465c-b610-8c314ca761cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449"
Sep 16 19:09:38.320456 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.320411    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d546b01-cf39-465c-b610-8c314ca761cd-tls-certs\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-89449\" (UID: \"9d546b01-cf39-465c-b610-8c314ca761cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449"
Sep 16 19:09:38.320684 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.320469    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d546b01-cf39-465c-b610-8c314ca761cd-model-cache\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-89449\" (UID: \"9d546b01-cf39-465c-b610-8c314ca761cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449"
Sep 16 19:09:38.320684 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.320496    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d546b01-cf39-465c-b610-8c314ca761cd-home\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-89449\" (UID: \"9d546b01-cf39-465c-b610-8c314ca761cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449"
Sep 16 19:09:38.320684 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.320518    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d546b01-cf39-465c-b610-8c314ca761cd-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-89449\" (UID: \"9d546b01-cf39-465c-b610-8c314ca761cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449"
Sep 16 19:09:38.320854 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.320771    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d546b01-cf39-465c-b610-8c314ca761cd-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-89449\" (UID: \"9d546b01-cf39-465c-b610-8c314ca761cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449"
Sep 16 19:09:38.320919 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.320900    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d546b01-cf39-465c-b610-8c314ca761cd-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-89449\" (UID: \"9d546b01-cf39-465c-b610-8c314ca761cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449"
Sep 16 19:09:38.321004 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.320975    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d546b01-cf39-465c-b610-8c314ca761cd-home\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-89449\" (UID: \"9d546b01-cf39-465c-b610-8c314ca761cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449"
Sep 16 19:09:38.321058 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.321026    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d546b01-cf39-465c-b610-8c314ca761cd-model-cache\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-89449\" (UID: \"9d546b01-cf39-465c-b610-8c314ca761cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449"
Sep 16 19:09:38.321298 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.321279    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d546b01-cf39-465c-b610-8c314ca761cd-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-89449\" (UID: \"9d546b01-cf39-465c-b610-8c314ca761cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449"
Sep 16 19:09:38.322746 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.322729    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d546b01-cf39-465c-b610-8c314ca761cd-dshm\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-89449\" (UID: \"9d546b01-cf39-465c-b610-8c314ca761cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449"
Sep 16 19:09:38.323170 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.323153    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d546b01-cf39-465c-b610-8c314ca761cd-tls-certs\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-89449\" (UID: \"9d546b01-cf39-465c-b610-8c314ca761cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449"
Sep 16 19:09:38.337223 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.337169    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8c46\" (UniqueName: \"kubernetes.io/projected/9d546b01-cf39-465c-b610-8c314ca761cd-kube-api-access-m8c46\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-89449\" (UID: \"9d546b01-cf39-465c-b610-8c314ca761cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449"
Sep 16 19:09:38.385085 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.385059    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67"]
Sep 16 19:09:38.389358 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.389341    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67"
Sep 16 19:09:38.391462 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.391439    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449"
Sep 16 19:09:38.404445 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.404421    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67"]
Sep 16 19:09:38.521722 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.521695    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae084697-44d1-4730-b606-99da1a3d2f02-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67\" (UID: \"ae084697-44d1-4730-b606-99da1a3d2f02\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67"
Sep 16 19:09:38.521874 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.521741    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae084697-44d1-4730-b606-99da1a3d2f02-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67\" (UID: \"ae084697-44d1-4730-b606-99da1a3d2f02\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67"
Sep 16 19:09:38.521874 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.521790    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae084697-44d1-4730-b606-99da1a3d2f02-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67\" (UID: \"ae084697-44d1-4730-b606-99da1a3d2f02\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67"
Sep 16 19:09:38.521874 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.521827    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae084697-44d1-4730-b606-99da1a3d2f02-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67\" (UID: \"ae084697-44d1-4730-b606-99da1a3d2f02\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67"
Sep 16 19:09:38.623228 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.623132    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae084697-44d1-4730-b606-99da1a3d2f02-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67\" (UID: \"ae084697-44d1-4730-b606-99da1a3d2f02\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67"
Sep 16 19:09:38.623228 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.623172    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae084697-44d1-4730-b606-99da1a3d2f02-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67\" (UID: \"ae084697-44d1-4730-b606-99da1a3d2f02\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67"
Sep 16 19:09:38.623228 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.623190    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae084697-44d1-4730-b606-99da1a3d2f02-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67\" (UID: \"ae084697-44d1-4730-b606-99da1a3d2f02\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67"
Sep 16 19:09:38.623228 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.623218    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae084697-44d1-4730-b606-99da1a3d2f02-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67\" (UID: \"ae084697-44d1-4730-b606-99da1a3d2f02\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67"
Sep 16 19:09:38.623623 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.623605    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae084697-44d1-4730-b606-99da1a3d2f02-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67\" (UID: \"ae084697-44d1-4730-b606-99da1a3d2f02\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67"
Sep 16 19:09:38.623694 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.623627    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae084697-44d1-4730-b606-99da1a3d2f02-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67\" (UID: \"ae084697-44d1-4730-b606-99da1a3d2f02\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67"
Sep 16 19:09:38.623887 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.623872    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae084697-44d1-4730-b606-99da1a3d2f02-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67\" (UID: \"ae084697-44d1-4730-b606-99da1a3d2f02\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67"
Sep 16 19:09:38.625719 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.625701    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae084697-44d1-4730-b606-99da1a3d2f02-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67\" (UID: \"ae084697-44d1-4730-b606-99da1a3d2f02\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67"
Sep 16 19:09:38.674676 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.674656    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449"]
Sep 16 19:09:38.676796 ip-10-0-128-247 kubenswrapper[2577]: W0916 19:09:38.676764    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d546b01_cf39_465c_b610_8c314ca761cd.slice/crio-fb680aa93127c4b87d0fd2c40178100571a50201e2a77f53979728a7871c87bb WatchSource:0}: Error finding container fb680aa93127c4b87d0fd2c40178100571a50201e2a77f53979728a7871c87bb: Status 404 returned error can't find the container with id fb680aa93127c4b87d0fd2c40178100571a50201e2a77f53979728a7871c87bb
Sep 16 19:09:38.700398 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.700378    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67"
Sep 16 19:09:38.982615 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:38.982588    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67"]
Sep 16 19:09:38.985703 ip-10-0-128-247 kubenswrapper[2577]: W0916 19:09:38.985662    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae084697_44d1_4730_b606_99da1a3d2f02.slice/crio-65145ddc24291a523dc9faa363aa27c213f7c5544149a21eb546a84fa151e004 WatchSource:0}: Error finding container 65145ddc24291a523dc9faa363aa27c213f7c5544149a21eb546a84fa151e004: Status 404 returned error can't find the container with id 65145ddc24291a523dc9faa363aa27c213f7c5544149a21eb546a84fa151e004
Sep 16 19:09:39.610022 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:39.609977    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67" event={"ID":"ae084697-44d1-4730-b606-99da1a3d2f02","Type":"ContainerStarted","Data":"642482c3176e765a09f485d7e7e046ec478a095bcec612668bf92daf988d720a"}
Sep 16 19:09:39.610522 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:39.610051    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67" event={"ID":"ae084697-44d1-4730-b606-99da1a3d2f02","Type":"ContainerStarted","Data":"65145ddc24291a523dc9faa363aa27c213f7c5544149a21eb546a84fa151e004"}
Sep 16 19:09:39.612493 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:39.612465    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449" event={"ID":"9d546b01-cf39-465c-b610-8c314ca761cd","Type":"ContainerStarted","Data":"114237653fa5ea22f95be8df3797fcb474dc9a4550a1bea93be26d32b170138d"}
Sep 16 19:09:39.612723 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:39.612692    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449" event={"ID":"9d546b01-cf39-465c-b610-8c314ca761cd","Type":"ContainerStarted","Data":"fb680aa93127c4b87d0fd2c40178100571a50201e2a77f53979728a7871c87bb"}
Sep 16 19:09:41.624996 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:41.624962    2577 generic.go:356] "Generic (PLEG): container finished" podID="ae084697-44d1-4730-b606-99da1a3d2f02" containerID="642482c3176e765a09f485d7e7e046ec478a095bcec612668bf92daf988d720a" exitCode=0
Sep 16 19:09:41.625399 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:41.625040    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67" event={"ID":"ae084697-44d1-4730-b606-99da1a3d2f02","Type":"ContainerDied","Data":"642482c3176e765a09f485d7e7e046ec478a095bcec612668bf92daf988d720a"}
Sep 16 19:09:41.626387 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:41.626368    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:09:41.626714 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:41.626691    2577 generic.go:356] "Generic (PLEG): container finished" podID="9d546b01-cf39-465c-b610-8c314ca761cd" containerID="114237653fa5ea22f95be8df3797fcb474dc9a4550a1bea93be26d32b170138d" exitCode=0
Sep 16 19:09:41.626798 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:41.626781    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449" event={"ID":"9d546b01-cf39-465c-b610-8c314ca761cd","Type":"ContainerDied","Data":"114237653fa5ea22f95be8df3797fcb474dc9a4550a1bea93be26d32b170138d"}
Sep 16 19:09:42.633265 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:42.633225    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449" event={"ID":"9d546b01-cf39-465c-b610-8c314ca761cd","Type":"ContainerStarted","Data":"7bfb7ca68d8840f10a047952d5919b35ecbd95c21b01ced66fd5c1e8a80bb153"}
Sep 16 19:09:42.659766 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:42.659720    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449" podStartSLOduration=2.338421253 podStartE2EDuration="4.659703087s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.321281834s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:09:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:09:42.659213344 +0000 UTC m=+1740.042102710" watchObservedRunningTime="2026-09-16 19:09:42.659703087 +0000 UTC m=+1740.042592452"
Sep 16 19:09:43.392369 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:43.392337    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449"
Sep 16 19:09:48.392120 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:48.392082    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449"
Sep 16 19:09:48.411876 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:48.411850    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449"
Sep 16 19:09:48.680615 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:09:48.680440    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449"
Sep 16 19:10:29.874586 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:10:29.874551    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67" event={"ID":"ae084697-44d1-4730-b606-99da1a3d2f02","Type":"ContainerStarted","Data":"482de0631f31d254d99b2d64bf94be7938656ffd4625370314453d494a09b638"}
Sep 16 19:10:29.874990 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:10:29.874753    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67"
Sep 16 19:10:29.901900 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:10:29.901859    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67" podStartSLOduration=1.954693567 podStartE2EDuration="51.901844827s" totalImagesPullingTime="47.598044281s" totalInitContainerRuntime="2.349106979s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:09:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:29.899228491 +0000 UTC m=+1787.282117859" watchObservedRunningTime="2026-09-16 19:10:29.901844827 +0000 UTC m=+1787.284734224"
Sep 16 19:11:00.889896 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:00.889859    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67"
Sep 16 19:11:09.708875 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:09.708841    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67_ae084697-44d1-4730-b606-99da1a3d2f02/main/0.log"
Sep 16 19:11:10.422126 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:10.422093    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67"]
Sep 16 19:11:10.422345 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:10.422308    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67" podUID="ae084697-44d1-4730-b606-99da1a3d2f02" containerName="main" containerID="cri-o://482de0631f31d254d99b2d64bf94be7938656ffd4625370314453d494a09b638" gracePeriod=30
Sep 16 19:11:10.430127 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:10.429599    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449"]
Sep 16 19:11:10.446220 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:11:10.446181    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:11:10.446220 ip-10-0-128-247 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:11:10.446220 ip-10-0-128-247 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 19:11:10.446220 ip-10-0-128-247 kubenswrapper[2577]:  > containerID="7bfb7ca68d8840f10a047952d5919b35ecbd95c21b01ced66fd5c1e8a80bb153" cmd=["/bin/sleep","15"]
Sep 16 19:11:10.446961 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:10.446237    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449" message="" err=<
Sep 16 19:11:10.446961 ip-10-0-128-247 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:11:10.446961 ip-10-0-128-247 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 19:11:10.446961 ip-10-0-128-247 kubenswrapper[2577]:  >
Sep 16 19:11:10.446961 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:11:10.446253    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:11:10.446961 ip-10-0-128-247 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:11:10.446961 ip-10-0-128-247 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 19:11:10.446961 ip-10-0-128-247 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449" podUID="9d546b01-cf39-465c-b610-8c314ca761cd" containerName="main" containerID="cri-o://7bfb7ca68d8840f10a047952d5919b35ecbd95c21b01ced66fd5c1e8a80bb153"
Sep 16 19:11:10.446961 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:10.446324    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449" podUID="9d546b01-cf39-465c-b610-8c314ca761cd" containerName="main" containerID="cri-o://7bfb7ca68d8840f10a047952d5919b35ecbd95c21b01ced66fd5c1e8a80bb153" gracePeriod=60
Sep 16 19:11:10.665935 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:10.665896    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449" podUID="9d546b01-cf39-465c-b610-8c314ca761cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Sep 16 19:11:10.793911 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:10.793884    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449"
Sep 16 19:11:10.879776 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:10.879736    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67" podUID="ae084697-44d1-4730-b606-99da1a3d2f02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Sep 16 19:11:10.926158 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:10.926136    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d546b01-cf39-465c-b610-8c314ca761cd-cabundle-cert\") pod \"9d546b01-cf39-465c-b610-8c314ca761cd\" (UID: \"9d546b01-cf39-465c-b610-8c314ca761cd\") "
Sep 16 19:11:10.926274 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:10.926191    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d546b01-cf39-465c-b610-8c314ca761cd-kserve-provision-location\") pod \"9d546b01-cf39-465c-b610-8c314ca761cd\" (UID: \"9d546b01-cf39-465c-b610-8c314ca761cd\") "
Sep 16 19:11:10.926274 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:10.926214    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d546b01-cf39-465c-b610-8c314ca761cd-tmp-dir\") pod \"9d546b01-cf39-465c-b610-8c314ca761cd\" (UID: \"9d546b01-cf39-465c-b610-8c314ca761cd\") "
Sep 16 19:11:10.926274 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:10.926240    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8c46\" (UniqueName: \"kubernetes.io/projected/9d546b01-cf39-465c-b610-8c314ca761cd-kube-api-access-m8c46\") pod \"9d546b01-cf39-465c-b610-8c314ca761cd\" (UID: \"9d546b01-cf39-465c-b610-8c314ca761cd\") "
Sep 16 19:11:10.926274 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:10.926268    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d546b01-cf39-465c-b610-8c314ca761cd-home\") pod \"9d546b01-cf39-465c-b610-8c314ca761cd\" (UID: \"9d546b01-cf39-465c-b610-8c314ca761cd\") "
Sep 16 19:11:10.926497 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:10.926308    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d546b01-cf39-465c-b610-8c314ca761cd-model-cache\") pod \"9d546b01-cf39-465c-b610-8c314ca761cd\" (UID: \"9d546b01-cf39-465c-b610-8c314ca761cd\") "
Sep 16 19:11:10.926497 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:10.926362    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d546b01-cf39-465c-b610-8c314ca761cd-tls-certs\") pod \"9d546b01-cf39-465c-b610-8c314ca761cd\" (UID: \"9d546b01-cf39-465c-b610-8c314ca761cd\") "
Sep 16 19:11:10.926497 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:10.926393    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d546b01-cf39-465c-b610-8c314ca761cd-dshm\") pod \"9d546b01-cf39-465c-b610-8c314ca761cd\" (UID: \"9d546b01-cf39-465c-b610-8c314ca761cd\") "
Sep 16 19:11:10.926497 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:10.926459    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d546b01-cf39-465c-b610-8c314ca761cd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9d546b01-cf39-465c-b610-8c314ca761cd" (UID: "9d546b01-cf39-465c-b610-8c314ca761cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:10.926747 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:10.926612    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d546b01-cf39-465c-b610-8c314ca761cd-model-cache" (OuterVolumeSpecName: "model-cache") pod "9d546b01-cf39-465c-b610-8c314ca761cd" (UID: "9d546b01-cf39-465c-b610-8c314ca761cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:10.926747 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:10.926651    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d546b01-cf39-465c-b610-8c314ca761cd-home" (OuterVolumeSpecName: "home") pod "9d546b01-cf39-465c-b610-8c314ca761cd" (UID: "9d546b01-cf39-465c-b610-8c314ca761cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:10.926747 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:10.926722    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d546b01-cf39-465c-b610-8c314ca761cd-model-cache\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:11:10.926894 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:10.926766    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d546b01-cf39-465c-b610-8c314ca761cd-tmp-dir\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:11:10.926894 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:10.926780    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d546b01-cf39-465c-b610-8c314ca761cd-home\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:11:10.926983 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:10.926919    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d546b01-cf39-465c-b610-8c314ca761cd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9d546b01-cf39-465c-b610-8c314ca761cd" (UID: "9d546b01-cf39-465c-b610-8c314ca761cd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:10.929031 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:10.929004    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d546b01-cf39-465c-b610-8c314ca761cd-dshm" (OuterVolumeSpecName: "dshm") pod "9d546b01-cf39-465c-b610-8c314ca761cd" (UID: "9d546b01-cf39-465c-b610-8c314ca761cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:10.929145 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:10.929109    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d546b01-cf39-465c-b610-8c314ca761cd-kube-api-access-m8c46" (OuterVolumeSpecName: "kube-api-access-m8c46") pod "9d546b01-cf39-465c-b610-8c314ca761cd" (UID: "9d546b01-cf39-465c-b610-8c314ca761cd"). InnerVolumeSpecName "kube-api-access-m8c46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:10.930667 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:10.930642    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d546b01-cf39-465c-b610-8c314ca761cd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9d546b01-cf39-465c-b610-8c314ca761cd" (UID: "9d546b01-cf39-465c-b610-8c314ca761cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:11:10.949946 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:10.949921    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d546b01-cf39-465c-b610-8c314ca761cd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9d546b01-cf39-465c-b610-8c314ca761cd" (UID: "9d546b01-cf39-465c-b610-8c314ca761cd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:11.027307 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:11.027274    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d546b01-cf39-465c-b610-8c314ca761cd-cabundle-cert\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:11:11.027428 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:11.027313    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d546b01-cf39-465c-b610-8c314ca761cd-kserve-provision-location\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:11:11.027428 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:11.027327    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8c46\" (UniqueName: \"kubernetes.io/projected/9d546b01-cf39-465c-b610-8c314ca761cd-kube-api-access-m8c46\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:11:11.027428 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:11.027342    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d546b01-cf39-465c-b610-8c314ca761cd-tls-certs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:11:11.027428 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:11.027353    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d546b01-cf39-465c-b610-8c314ca761cd-dshm\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:11:11.059696 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:11.059669    2577 generic.go:356] "Generic (PLEG): container finished" podID="ae084697-44d1-4730-b606-99da1a3d2f02" containerID="482de0631f31d254d99b2d64bf94be7938656ffd4625370314453d494a09b638" exitCode=0
Sep 16 19:11:11.059838 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:11.059743    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67" event={"ID":"ae084697-44d1-4730-b606-99da1a3d2f02","Type":"ContainerDied","Data":"482de0631f31d254d99b2d64bf94be7938656ffd4625370314453d494a09b638"}
Sep 16 19:11:11.061376 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:11.061353    2577 generic.go:356] "Generic (PLEG): container finished" podID="9d546b01-cf39-465c-b610-8c314ca761cd" containerID="7bfb7ca68d8840f10a047952d5919b35ecbd95c21b01ced66fd5c1e8a80bb153" exitCode=0
Sep 16 19:11:11.061486 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:11.061404    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449"
Sep 16 19:11:11.061486 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:11.061411    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449" event={"ID":"9d546b01-cf39-465c-b610-8c314ca761cd","Type":"ContainerDied","Data":"7bfb7ca68d8840f10a047952d5919b35ecbd95c21b01ced66fd5c1e8a80bb153"}
Sep 16 19:11:11.061486 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:11.061463    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449" event={"ID":"9d546b01-cf39-465c-b610-8c314ca761cd","Type":"ContainerDied","Data":"fb680aa93127c4b87d0fd2c40178100571a50201e2a77f53979728a7871c87bb"}
Sep 16 19:11:11.061678 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:11.061493    2577 scope.go:117] "RemoveContainer" containerID="7bfb7ca68d8840f10a047952d5919b35ecbd95c21b01ced66fd5c1e8a80bb153"
Sep 16 19:11:11.065074 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:11.065049    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67"
Sep 16 19:11:11.070278 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:11.070264    2577 scope.go:117] "RemoveContainer" containerID="114237653fa5ea22f95be8df3797fcb474dc9a4550a1bea93be26d32b170138d"
Sep 16 19:11:11.077700 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:11.077666    2577 scope.go:117] "RemoveContainer" containerID="7bfb7ca68d8840f10a047952d5919b35ecbd95c21b01ced66fd5c1e8a80bb153"
Sep 16 19:11:11.077927 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:11:11.077909    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bfb7ca68d8840f10a047952d5919b35ecbd95c21b01ced66fd5c1e8a80bb153\": container with ID starting with 7bfb7ca68d8840f10a047952d5919b35ecbd95c21b01ced66fd5c1e8a80bb153 not found: ID does not exist" containerID="7bfb7ca68d8840f10a047952d5919b35ecbd95c21b01ced66fd5c1e8a80bb153"
Sep 16 19:11:11.077973 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:11.077935    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bfb7ca68d8840f10a047952d5919b35ecbd95c21b01ced66fd5c1e8a80bb153"} err="failed to get container status \"7bfb7ca68d8840f10a047952d5919b35ecbd95c21b01ced66fd5c1e8a80bb153\": rpc error: code = NotFound desc = could not find container \"7bfb7ca68d8840f10a047952d5919b35ecbd95c21b01ced66fd5c1e8a80bb153\": container with ID starting with 7bfb7ca68d8840f10a047952d5919b35ecbd95c21b01ced66fd5c1e8a80bb153 not found: ID does not exist"
Sep 16 19:11:11.077973 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:11.077951    2577 scope.go:117] "RemoveContainer" containerID="114237653fa5ea22f95be8df3797fcb474dc9a4550a1bea93be26d32b170138d"
Sep 16 19:11:11.078214 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:11:11.078196    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"114237653fa5ea22f95be8df3797fcb474dc9a4550a1bea93be26d32b170138d\": container with ID starting with 114237653fa5ea22f95be8df3797fcb474dc9a4550a1bea93be26d32b170138d not found: ID does not exist" containerID="114237653fa5ea22f95be8df3797fcb474dc9a4550a1bea93be26d32b170138d"
Sep 16 19:11:11.078271 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:11.078218    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"114237653fa5ea22f95be8df3797fcb474dc9a4550a1bea93be26d32b170138d"} err="failed to get container status \"114237653fa5ea22f95be8df3797fcb474dc9a4550a1bea93be26d32b170138d\": rpc error: code = NotFound desc = could not find container \"114237653fa5ea22f95be8df3797fcb474dc9a4550a1bea93be26d32b170138d\": container with ID starting with 114237653fa5ea22f95be8df3797fcb474dc9a4550a1bea93be26d32b170138d not found: ID does not exist"
Sep 16 19:11:11.107651 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:11.107621    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449"]
Sep 16 19:11:11.115151 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:11.115132    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-89449"]
Sep 16 19:11:11.127718 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:11.127692    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae084697-44d1-4730-b606-99da1a3d2f02-tmp-dir\") pod \"ae084697-44d1-4730-b606-99da1a3d2f02\" (UID: \"ae084697-44d1-4730-b606-99da1a3d2f02\") "
Sep 16 19:11:11.127834 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:11.127731    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae084697-44d1-4730-b606-99da1a3d2f02-tls-certs\") pod \"ae084697-44d1-4730-b606-99da1a3d2f02\" (UID: \"ae084697-44d1-4730-b606-99da1a3d2f02\") "
Sep 16 19:11:11.127834 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:11.127763    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae084697-44d1-4730-b606-99da1a3d2f02-kserve-provision-location\") pod \"ae084697-44d1-4730-b606-99da1a3d2f02\" (UID: \"ae084697-44d1-4730-b606-99da1a3d2f02\") "
Sep 16 19:11:11.127834 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:11.127790    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae084697-44d1-4730-b606-99da1a3d2f02-cabundle-cert\") pod \"ae084697-44d1-4730-b606-99da1a3d2f02\" (UID: \"ae084697-44d1-4730-b606-99da1a3d2f02\") "
Sep 16 19:11:11.128097 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:11.128076    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae084697-44d1-4730-b606-99da1a3d2f02-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ae084697-44d1-4730-b606-99da1a3d2f02" (UID: "ae084697-44d1-4730-b606-99da1a3d2f02"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:11.128203 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:11.128180    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae084697-44d1-4730-b606-99da1a3d2f02-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ae084697-44d1-4730-b606-99da1a3d2f02" (UID: "ae084697-44d1-4730-b606-99da1a3d2f02"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:11.129906 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:11.129884    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae084697-44d1-4730-b606-99da1a3d2f02-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ae084697-44d1-4730-b606-99da1a3d2f02" (UID: "ae084697-44d1-4730-b606-99da1a3d2f02"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:11:11.148332 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:11.148312    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae084697-44d1-4730-b606-99da1a3d2f02-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ae084697-44d1-4730-b606-99da1a3d2f02" (UID: "ae084697-44d1-4730-b606-99da1a3d2f02"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:11.155251 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:11.155229    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d546b01-cf39-465c-b610-8c314ca761cd" path="/var/lib/kubelet/pods/9d546b01-cf39-465c-b610-8c314ca761cd/volumes"
Sep 16 19:11:11.228769 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:11.228733    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae084697-44d1-4730-b606-99da1a3d2f02-tmp-dir\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:11:11.228769 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:11.228759    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae084697-44d1-4730-b606-99da1a3d2f02-tls-certs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:11:11.228769 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:11.228773    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae084697-44d1-4730-b606-99da1a3d2f02-kserve-provision-location\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:11:11.228936 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:11.228787    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae084697-44d1-4730-b606-99da1a3d2f02-cabundle-cert\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:11:12.067979 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:12.067951    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67"
Sep 16 19:11:12.068445 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:12.067949    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67" event={"ID":"ae084697-44d1-4730-b606-99da1a3d2f02","Type":"ContainerDied","Data":"65145ddc24291a523dc9faa363aa27c213f7c5544149a21eb546a84fa151e004"}
Sep 16 19:11:12.068445 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:12.068096    2577 scope.go:117] "RemoveContainer" containerID="482de0631f31d254d99b2d64bf94be7938656ffd4625370314453d494a09b638"
Sep 16 19:11:12.076657 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:12.076641    2577 scope.go:117] "RemoveContainer" containerID="642482c3176e765a09f485d7e7e046ec478a095bcec612668bf92daf988d720a"
Sep 16 19:11:12.090990 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:12.090968    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67"]
Sep 16 19:11:12.099422 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:12.099400    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7cdfcfc67b-h9n67"]
Sep 16 19:11:13.154957 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:13.154927    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae084697-44d1-4730-b606-99da1a3d2f02" path="/var/lib/kubelet/pods/ae084697-44d1-4730-b606-99da1a3d2f02/volumes"
Sep 16 19:11:28.233277 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:28.233249    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-jpmjq"]
Sep 16 19:11:28.233720 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:28.233586    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d546b01-cf39-465c-b610-8c314ca761cd" containerName="main"
Sep 16 19:11:28.233720 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:28.233598    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d546b01-cf39-465c-b610-8c314ca761cd" containerName="main"
Sep 16 19:11:28.233720 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:28.233607    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae084697-44d1-4730-b606-99da1a3d2f02" containerName="main"
Sep 16 19:11:28.233720 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:28.233612    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae084697-44d1-4730-b606-99da1a3d2f02" containerName="main"
Sep 16 19:11:28.233720 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:28.233621    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae084697-44d1-4730-b606-99da1a3d2f02" containerName="storage-initializer"
Sep 16 19:11:28.233720 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:28.233627    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae084697-44d1-4730-b606-99da1a3d2f02" containerName="storage-initializer"
Sep 16 19:11:28.233720 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:28.233640    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d546b01-cf39-465c-b610-8c314ca761cd" containerName="storage-initializer"
Sep 16 19:11:28.233720 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:28.233645    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d546b01-cf39-465c-b610-8c314ca761cd" containerName="storage-initializer"
Sep 16 19:11:28.233720 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:28.233717    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae084697-44d1-4730-b606-99da1a3d2f02" containerName="main"
Sep 16 19:11:28.234038 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:28.233727    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d546b01-cf39-465c-b610-8c314ca761cd" containerName="main"
Sep 16 19:11:28.235655 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:28.235639    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-jpmjq"
Sep 16 19:11:28.238612 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:28.238549    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"kube-root-ca.crt\""
Sep 16 19:11:28.240575 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:28.240516    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"openshift-service-ca.crt\""
Sep 16 19:11:28.240758 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:28.240624    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"default-dockercfg-bggzx\""
Sep 16 19:11:28.240875 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:28.240673    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"stop-v1-kserve-self-signed-certs\""
Sep 16 19:11:28.256284 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:28.256251    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-jpmjq"]
Sep 16 19:11:28.357505 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:28.357473    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/033ee791-2955-49c9-9291-d3ac87fbaf65-dshm\") pod \"stop-v1-kserve-5657b88c95-jpmjq\" (UID: \"033ee791-2955-49c9-9291-d3ac87fbaf65\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-jpmjq"
Sep 16 19:11:28.357505 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:28.357509    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzd6v\" (UniqueName: \"kubernetes.io/projected/033ee791-2955-49c9-9291-d3ac87fbaf65-kube-api-access-mzd6v\") pod \"stop-v1-kserve-5657b88c95-jpmjq\" (UID: \"033ee791-2955-49c9-9291-d3ac87fbaf65\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-jpmjq"
Sep 16 19:11:28.357685 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:28.357546    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/033ee791-2955-49c9-9291-d3ac87fbaf65-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-jpmjq\" (UID: \"033ee791-2955-49c9-9291-d3ac87fbaf65\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-jpmjq"
Sep 16 19:11:28.357685 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:28.357621    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/033ee791-2955-49c9-9291-d3ac87fbaf65-model-cache\") pod \"stop-v1-kserve-5657b88c95-jpmjq\" (UID: \"033ee791-2955-49c9-9291-d3ac87fbaf65\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-jpmjq"
Sep 16 19:11:28.357685 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:28.357647    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/033ee791-2955-49c9-9291-d3ac87fbaf65-tls-certs\") pod \"stop-v1-kserve-5657b88c95-jpmjq\" (UID: \"033ee791-2955-49c9-9291-d3ac87fbaf65\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-jpmjq"
Sep 16 19:11:28.357790 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:28.357709    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/033ee791-2955-49c9-9291-d3ac87fbaf65-home\") pod \"stop-v1-kserve-5657b88c95-jpmjq\" (UID: \"033ee791-2955-49c9-9291-d3ac87fbaf65\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-jpmjq"
Sep 16 19:11:28.458924 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:28.458897    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/033ee791-2955-49c9-9291-d3ac87fbaf65-dshm\") pod \"stop-v1-kserve-5657b88c95-jpmjq\" (UID: \"033ee791-2955-49c9-9291-d3ac87fbaf65\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-jpmjq"
Sep 16 19:11:28.458924 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:28.458923    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzd6v\" (UniqueName: \"kubernetes.io/projected/033ee791-2955-49c9-9291-d3ac87fbaf65-kube-api-access-mzd6v\") pod \"stop-v1-kserve-5657b88c95-jpmjq\" (UID: \"033ee791-2955-49c9-9291-d3ac87fbaf65\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-jpmjq"
Sep 16 19:11:28.459104 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:28.458943    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/033ee791-2955-49c9-9291-d3ac87fbaf65-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-jpmjq\" (UID: \"033ee791-2955-49c9-9291-d3ac87fbaf65\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-jpmjq"
Sep 16 19:11:28.459104 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:28.458963    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/033ee791-2955-49c9-9291-d3ac87fbaf65-model-cache\") pod \"stop-v1-kserve-5657b88c95-jpmjq\" (UID: \"033ee791-2955-49c9-9291-d3ac87fbaf65\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-jpmjq"
Sep 16 19:11:28.459215 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:28.459103    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/033ee791-2955-49c9-9291-d3ac87fbaf65-tls-certs\") pod \"stop-v1-kserve-5657b88c95-jpmjq\" (UID: \"033ee791-2955-49c9-9291-d3ac87fbaf65\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-jpmjq"
Sep 16 19:11:28.459215 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:28.459186    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/033ee791-2955-49c9-9291-d3ac87fbaf65-home\") pod \"stop-v1-kserve-5657b88c95-jpmjq\" (UID: \"033ee791-2955-49c9-9291-d3ac87fbaf65\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-jpmjq"
Sep 16 19:11:28.459341 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:28.459325    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/033ee791-2955-49c9-9291-d3ac87fbaf65-model-cache\") pod \"stop-v1-kserve-5657b88c95-jpmjq\" (UID: \"033ee791-2955-49c9-9291-d3ac87fbaf65\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-jpmjq"
Sep 16 19:11:28.459384 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:28.459354    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/033ee791-2955-49c9-9291-d3ac87fbaf65-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-jpmjq\" (UID: \"033ee791-2955-49c9-9291-d3ac87fbaf65\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-jpmjq"
Sep 16 19:11:28.459478 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:28.459464    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/033ee791-2955-49c9-9291-d3ac87fbaf65-home\") pod \"stop-v1-kserve-5657b88c95-jpmjq\" (UID: \"033ee791-2955-49c9-9291-d3ac87fbaf65\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-jpmjq"
Sep 16 19:11:28.461548 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:28.461509    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/033ee791-2955-49c9-9291-d3ac87fbaf65-dshm\") pod \"stop-v1-kserve-5657b88c95-jpmjq\" (UID: \"033ee791-2955-49c9-9291-d3ac87fbaf65\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-jpmjq"
Sep 16 19:11:28.461698 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:28.461678    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/033ee791-2955-49c9-9291-d3ac87fbaf65-tls-certs\") pod \"stop-v1-kserve-5657b88c95-jpmjq\" (UID: \"033ee791-2955-49c9-9291-d3ac87fbaf65\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-jpmjq"
Sep 16 19:11:28.472643 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:28.472621    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzd6v\" (UniqueName: \"kubernetes.io/projected/033ee791-2955-49c9-9291-d3ac87fbaf65-kube-api-access-mzd6v\") pod \"stop-v1-kserve-5657b88c95-jpmjq\" (UID: \"033ee791-2955-49c9-9291-d3ac87fbaf65\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-jpmjq"
Sep 16 19:11:28.546694 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:28.546639    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-jpmjq"
Sep 16 19:11:28.816982 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:28.816959    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-jpmjq"]
Sep 16 19:11:28.819940 ip-10-0-128-247 kubenswrapper[2577]: W0916 19:11:28.819911    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod033ee791_2955_49c9_9291_d3ac87fbaf65.slice/crio-e6dec71a0b2d14dec7d268f4a73c2afe3e0bf0909847773a8673c9b2da0e7c00 WatchSource:0}: Error finding container e6dec71a0b2d14dec7d268f4a73c2afe3e0bf0909847773a8673c9b2da0e7c00: Status 404 returned error can't find the container with id e6dec71a0b2d14dec7d268f4a73c2afe3e0bf0909847773a8673c9b2da0e7c00
Sep 16 19:11:29.143144 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:29.143105    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-jpmjq" event={"ID":"033ee791-2955-49c9-9291-d3ac87fbaf65","Type":"ContainerStarted","Data":"e6dec71a0b2d14dec7d268f4a73c2afe3e0bf0909847773a8673c9b2da0e7c00"}
Sep 16 19:11:30.149217 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:30.149183    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-jpmjq" event={"ID":"033ee791-2955-49c9-9291-d3ac87fbaf65","Type":"ContainerStarted","Data":"587bb22fe7286a548de1b4a0cc18d80ec300bb79beffead5cc5d73e825422682"}
Sep 16 19:11:30.170742 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:30.170697    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-jpmjq" podStartSLOduration=1.183589027 podStartE2EDuration="2.170680641s" totalImagesPullingTime="987.091614ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:11:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:11:30.168874422 +0000 UTC m=+1847.551763787" watchObservedRunningTime="2026-09-16 19:11:30.170680641 +0000 UTC m=+1847.553570005"
Sep 16 19:11:30.547026 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:30.546951    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-jpmjq"
Sep 16 19:11:38.547152 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:38.547111    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-jpmjq"
Sep 16 19:11:38.559868 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:38.559829    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-jpmjq"
Sep 16 19:11:39.205653 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:11:39.205615    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-jpmjq"
Sep 16 19:12:45.657293 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:45.657255    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs"]
Sep 16 19:12:45.660821 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:45.660804    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs"
Sep 16 19:12:45.663645 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:45.663623    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 16 19:12:45.664797 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:45.664772    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-tl57x\""
Sep 16 19:12:45.664797 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:45.664786    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 16 19:12:45.664967 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:45.664798    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 16 19:12:45.664967 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:45.664803    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:12:45.665072 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:45.665031    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 16 19:12:45.673042 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:45.673020    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs"]
Sep 16 19:12:45.726957 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:45.726936    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-model-cache\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs\" (UID: \"742a3dd2-dce5-4983-a338-1a7a4aa7e23e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs"
Sep 16 19:12:45.727081 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:45.726968    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bggzt\" (UniqueName: \"kubernetes.io/projected/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-kube-api-access-bggzt\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs\" (UID: \"742a3dd2-dce5-4983-a338-1a7a4aa7e23e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs"
Sep 16 19:12:45.727081 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:45.726986    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-home\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs\" (UID: \"742a3dd2-dce5-4983-a338-1a7a4aa7e23e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs"
Sep 16 19:12:45.727081 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:45.727021    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-dshm\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs\" (UID: \"742a3dd2-dce5-4983-a338-1a7a4aa7e23e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs"
Sep 16 19:12:45.727081 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:45.727044    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs\" (UID: \"742a3dd2-dce5-4983-a338-1a7a4aa7e23e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs"
Sep 16 19:12:45.727081 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:45.727073    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-tls-certs\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs\" (UID: \"742a3dd2-dce5-4983-a338-1a7a4aa7e23e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs"
Sep 16 19:12:45.727289 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:45.727091    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs\" (UID: \"742a3dd2-dce5-4983-a338-1a7a4aa7e23e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs"
Sep 16 19:12:45.727289 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:45.727112    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs\" (UID: \"742a3dd2-dce5-4983-a338-1a7a4aa7e23e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs"
Sep 16 19:12:45.828130 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:45.828093    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-dshm\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs\" (UID: \"742a3dd2-dce5-4983-a338-1a7a4aa7e23e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs"
Sep 16 19:12:45.828130 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:45.828131    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs\" (UID: \"742a3dd2-dce5-4983-a338-1a7a4aa7e23e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs"
Sep 16 19:12:45.828278 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:45.828179    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-tls-certs\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs\" (UID: \"742a3dd2-dce5-4983-a338-1a7a4aa7e23e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs"
Sep 16 19:12:45.828278 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:45.828209    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs\" (UID: \"742a3dd2-dce5-4983-a338-1a7a4aa7e23e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs"
Sep 16 19:12:45.828399 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:45.828349    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs\" (UID: \"742a3dd2-dce5-4983-a338-1a7a4aa7e23e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs"
Sep 16 19:12:45.828454 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:45.828425    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-model-cache\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs\" (UID: \"742a3dd2-dce5-4983-a338-1a7a4aa7e23e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs"
Sep 16 19:12:45.828511 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:45.828490    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bggzt\" (UniqueName: \"kubernetes.io/projected/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-kube-api-access-bggzt\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs\" (UID: \"742a3dd2-dce5-4983-a338-1a7a4aa7e23e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs"
Sep 16 19:12:45.828654 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:45.828633    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs\" (UID: \"742a3dd2-dce5-4983-a338-1a7a4aa7e23e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs"
Sep 16 19:12:45.828744 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:45.828690    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs\" (UID: \"742a3dd2-dce5-4983-a338-1a7a4aa7e23e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs"
Sep 16 19:12:45.828744 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:45.828519    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-home\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs\" (UID: \"742a3dd2-dce5-4983-a338-1a7a4aa7e23e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs"
Sep 16 19:12:45.828744 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:45.828734    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-model-cache\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs\" (UID: \"742a3dd2-dce5-4983-a338-1a7a4aa7e23e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs"
Sep 16 19:12:45.829116 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:45.829093    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs\" (UID: \"742a3dd2-dce5-4983-a338-1a7a4aa7e23e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs"
Sep 16 19:12:45.829251 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:45.829231    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-home\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs\" (UID: \"742a3dd2-dce5-4983-a338-1a7a4aa7e23e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs"
Sep 16 19:12:45.830646 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:45.830627    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-dshm\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs\" (UID: \"742a3dd2-dce5-4983-a338-1a7a4aa7e23e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs"
Sep 16 19:12:45.830927 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:45.830913    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-tls-certs\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs\" (UID: \"742a3dd2-dce5-4983-a338-1a7a4aa7e23e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs"
Sep 16 19:12:45.837976 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:45.837958    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bggzt\" (UniqueName: \"kubernetes.io/projected/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-kube-api-access-bggzt\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs\" (UID: \"742a3dd2-dce5-4983-a338-1a7a4aa7e23e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs"
Sep 16 19:12:45.918761 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:45.918678    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7"]
Sep 16 19:12:45.922600 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:45.922578    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7"
Sep 16 19:12:45.933732 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:45.933711    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7"]
Sep 16 19:12:45.971523 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:45.971495    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs"
Sep 16 19:12:46.030122 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:46.030086    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a99ea9d5-1cb5-4e44-9f4e-7c082a354cce-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7\" (UID: \"a99ea9d5-1cb5-4e44-9f4e-7c082a354cce\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7"
Sep 16 19:12:46.030248 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:46.030145    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a99ea9d5-1cb5-4e44-9f4e-7c082a354cce-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7\" (UID: \"a99ea9d5-1cb5-4e44-9f4e-7c082a354cce\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7"
Sep 16 19:12:46.030316 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:46.030291    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a99ea9d5-1cb5-4e44-9f4e-7c082a354cce-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7\" (UID: \"a99ea9d5-1cb5-4e44-9f4e-7c082a354cce\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7"
Sep 16 19:12:46.030420 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:46.030361    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a99ea9d5-1cb5-4e44-9f4e-7c082a354cce-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7\" (UID: \"a99ea9d5-1cb5-4e44-9f4e-7c082a354cce\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7"
Sep 16 19:12:46.131208 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:46.131173    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a99ea9d5-1cb5-4e44-9f4e-7c082a354cce-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7\" (UID: \"a99ea9d5-1cb5-4e44-9f4e-7c082a354cce\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7"
Sep 16 19:12:46.131362 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:46.131228    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a99ea9d5-1cb5-4e44-9f4e-7c082a354cce-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7\" (UID: \"a99ea9d5-1cb5-4e44-9f4e-7c082a354cce\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7"
Sep 16 19:12:46.131362 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:46.131265    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a99ea9d5-1cb5-4e44-9f4e-7c082a354cce-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7\" (UID: \"a99ea9d5-1cb5-4e44-9f4e-7c082a354cce\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7"
Sep 16 19:12:46.131362 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:46.131293    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a99ea9d5-1cb5-4e44-9f4e-7c082a354cce-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7\" (UID: \"a99ea9d5-1cb5-4e44-9f4e-7c082a354cce\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7"
Sep 16 19:12:46.131653 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:46.131632    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a99ea9d5-1cb5-4e44-9f4e-7c082a354cce-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7\" (UID: \"a99ea9d5-1cb5-4e44-9f4e-7c082a354cce\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7"
Sep 16 19:12:46.131741 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:46.131656    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a99ea9d5-1cb5-4e44-9f4e-7c082a354cce-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7\" (UID: \"a99ea9d5-1cb5-4e44-9f4e-7c082a354cce\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7"
Sep 16 19:12:46.131976 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:46.131958    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a99ea9d5-1cb5-4e44-9f4e-7c082a354cce-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7\" (UID: \"a99ea9d5-1cb5-4e44-9f4e-7c082a354cce\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7"
Sep 16 19:12:46.133832 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:46.133816    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a99ea9d5-1cb5-4e44-9f4e-7c082a354cce-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7\" (UID: \"a99ea9d5-1cb5-4e44-9f4e-7c082a354cce\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7"
Sep 16 19:12:46.234594 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:46.234520    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7"
Sep 16 19:12:46.252354 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:46.252327    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs"]
Sep 16 19:12:46.253390 ip-10-0-128-247 kubenswrapper[2577]: W0916 19:12:46.253366    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod742a3dd2_dce5_4983_a338_1a7a4aa7e23e.slice/crio-a9aba3a40a2c7a4eb9a1017a8b135900117dd0b2727f7b243fff561aeb53c369 WatchSource:0}: Error finding container a9aba3a40a2c7a4eb9a1017a8b135900117dd0b2727f7b243fff561aeb53c369: Status 404 returned error can't find the container with id a9aba3a40a2c7a4eb9a1017a8b135900117dd0b2727f7b243fff561aeb53c369
Sep 16 19:12:46.474139 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:46.474109    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs" event={"ID":"742a3dd2-dce5-4983-a338-1a7a4aa7e23e","Type":"ContainerStarted","Data":"72ae8c171f6a01b515c6b768157f3779cc452a955fa53c142d8401ef88455f9e"}
Sep 16 19:12:46.474304 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:46.474165    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs" event={"ID":"742a3dd2-dce5-4983-a338-1a7a4aa7e23e","Type":"ContainerStarted","Data":"a9aba3a40a2c7a4eb9a1017a8b135900117dd0b2727f7b243fff561aeb53c369"}
Sep 16 19:12:46.509829 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:46.509798    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7"]
Sep 16 19:12:46.512590 ip-10-0-128-247 kubenswrapper[2577]: W0916 19:12:46.512566    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda99ea9d5_1cb5_4e44_9f4e_7c082a354cce.slice/crio-64808346373bc329360ab0c83f4dd433229c75e4c52f3c6d91dd56ec5e363d56 WatchSource:0}: Error finding container 64808346373bc329360ab0c83f4dd433229c75e4c52f3c6d91dd56ec5e363d56: Status 404 returned error can't find the container with id 64808346373bc329360ab0c83f4dd433229c75e4c52f3c6d91dd56ec5e363d56
Sep 16 19:12:47.479631 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:47.479592    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7" event={"ID":"a99ea9d5-1cb5-4e44-9f4e-7c082a354cce","Type":"ContainerStarted","Data":"c8df69e2b01afcba05291026fb2b0aec3312fc42bd5aa612263828ff595e1289"}
Sep 16 19:12:47.480107 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:47.479653    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7" event={"ID":"a99ea9d5-1cb5-4e44-9f4e-7c082a354cce","Type":"ContainerStarted","Data":"64808346373bc329360ab0c83f4dd433229c75e4c52f3c6d91dd56ec5e363d56"}
Sep 16 19:12:49.489890 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:49.489860    2577 generic.go:356] "Generic (PLEG): container finished" podID="a99ea9d5-1cb5-4e44-9f4e-7c082a354cce" containerID="c8df69e2b01afcba05291026fb2b0aec3312fc42bd5aa612263828ff595e1289" exitCode=0
Sep 16 19:12:49.490301 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:49.489942    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7" event={"ID":"a99ea9d5-1cb5-4e44-9f4e-7c082a354cce","Type":"ContainerDied","Data":"c8df69e2b01afcba05291026fb2b0aec3312fc42bd5aa612263828ff595e1289"}
Sep 16 19:12:49.491568 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:49.491548    2577 generic.go:356] "Generic (PLEG): container finished" podID="742a3dd2-dce5-4983-a338-1a7a4aa7e23e" containerID="72ae8c171f6a01b515c6b768157f3779cc452a955fa53c142d8401ef88455f9e" exitCode=0
Sep 16 19:12:49.491658 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:49.491604    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs" event={"ID":"742a3dd2-dce5-4983-a338-1a7a4aa7e23e","Type":"ContainerDied","Data":"72ae8c171f6a01b515c6b768157f3779cc452a955fa53c142d8401ef88455f9e"}
Sep 16 19:12:50.497211 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:50.497176    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs" event={"ID":"742a3dd2-dce5-4983-a338-1a7a4aa7e23e","Type":"ContainerStarted","Data":"9bd12f3f1569efb65d8c3d99053329dca58fad9a7d33723f2612688bd9e5cc1c"}
Sep 16 19:12:50.499258 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:50.499236    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7" event={"ID":"a99ea9d5-1cb5-4e44-9f4e-7c082a354cce","Type":"ContainerStarted","Data":"2adb74afc7491b7d73889314d9cb92750bf72cbe7e741f11eee548c1a48813bc"}
Sep 16 19:12:50.499455 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:50.499436    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7"
Sep 16 19:12:50.526227 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:50.526194    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs" podStartSLOduration=3.202850408 podStartE2EDuration="5.526182509s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.323332101s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:12:50.524924835 +0000 UTC m=+1927.907814201" watchObservedRunningTime="2026-09-16 19:12:50.526182509 +0000 UTC m=+1927.909071874"
Sep 16 19:12:50.551708 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:50.551672    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7" podStartSLOduration=3.201412924 podStartE2EDuration="5.551658052s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.350245128s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:12:50.550490501 +0000 UTC m=+1927.933379867" watchObservedRunningTime="2026-09-16 19:12:50.551658052 +0000 UTC m=+1927.934547416"
Sep 16 19:12:50.972248 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:50.972215    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs"
Sep 16 19:12:55.971871 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:55.971825    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs"
Sep 16 19:12:55.988311 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:55.988280    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs"
Sep 16 19:12:56.545851 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:12:56.545818    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs"
Sep 16 19:13:21.515683 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:21.515649    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7"
Sep 16 19:13:28.780343 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:28.780312    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7_a99ea9d5-1cb5-4e44-9f4e-7c082a354cce/main/0.log"
Sep 16 19:13:36.532339 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:36.532255    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7"]
Sep 16 19:13:36.532759 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:36.532576    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7" podUID="a99ea9d5-1cb5-4e44-9f4e-7c082a354cce" containerName="main" containerID="cri-o://2adb74afc7491b7d73889314d9cb92750bf72cbe7e741f11eee548c1a48813bc" gracePeriod=30
Sep 16 19:13:36.536743 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:36.536719    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs"]
Sep 16 19:13:36.546078 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:13:36.546046    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:13:36.546078 ip-10-0-128-247 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:13:36.546078 ip-10-0-128-247 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 19:13:36.546078 ip-10-0-128-247 kubenswrapper[2577]:  > containerID="9bd12f3f1569efb65d8c3d99053329dca58fad9a7d33723f2612688bd9e5cc1c" cmd=["/bin/sleep","15"]
Sep 16 19:13:36.546286 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:36.546094    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs" message="" err=<
Sep 16 19:13:36.546286 ip-10-0-128-247 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:13:36.546286 ip-10-0-128-247 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 19:13:36.546286 ip-10-0-128-247 kubenswrapper[2577]:  >
Sep 16 19:13:36.546286 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:13:36.546105    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:13:36.546286 ip-10-0-128-247 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:13:36.546286 ip-10-0-128-247 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 19:13:36.546286 ip-10-0-128-247 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs" podUID="742a3dd2-dce5-4983-a338-1a7a4aa7e23e" containerName="main" containerID="cri-o://9bd12f3f1569efb65d8c3d99053329dca58fad9a7d33723f2612688bd9e5cc1c"
Sep 16 19:13:36.546286 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:36.546135    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs" podUID="742a3dd2-dce5-4983-a338-1a7a4aa7e23e" containerName="main" containerID="cri-o://9bd12f3f1569efb65d8c3d99053329dca58fad9a7d33723f2612688bd9e5cc1c" gracePeriod=60
Sep 16 19:13:36.899005 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:36.898980    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs"
Sep 16 19:13:36.960777 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:36.960743    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-cabundle-cert\") pod \"742a3dd2-dce5-4983-a338-1a7a4aa7e23e\" (UID: \"742a3dd2-dce5-4983-a338-1a7a4aa7e23e\") "
Sep 16 19:13:36.960965 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:36.960816    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bggzt\" (UniqueName: \"kubernetes.io/projected/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-kube-api-access-bggzt\") pod \"742a3dd2-dce5-4983-a338-1a7a4aa7e23e\" (UID: \"742a3dd2-dce5-4983-a338-1a7a4aa7e23e\") "
Sep 16 19:13:36.960965 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:36.960835    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-dshm\") pod \"742a3dd2-dce5-4983-a338-1a7a4aa7e23e\" (UID: \"742a3dd2-dce5-4983-a338-1a7a4aa7e23e\") "
Sep 16 19:13:36.960965 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:36.960855    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-tmp-dir\") pod \"742a3dd2-dce5-4983-a338-1a7a4aa7e23e\" (UID: \"742a3dd2-dce5-4983-a338-1a7a4aa7e23e\") "
Sep 16 19:13:36.960965 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:36.960879    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-kserve-provision-location\") pod \"742a3dd2-dce5-4983-a338-1a7a4aa7e23e\" (UID: \"742a3dd2-dce5-4983-a338-1a7a4aa7e23e\") "
Sep 16 19:13:36.960965 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:36.960915    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-tls-certs\") pod \"742a3dd2-dce5-4983-a338-1a7a4aa7e23e\" (UID: \"742a3dd2-dce5-4983-a338-1a7a4aa7e23e\") "
Sep 16 19:13:36.960965 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:36.960940    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-home\") pod \"742a3dd2-dce5-4983-a338-1a7a4aa7e23e\" (UID: \"742a3dd2-dce5-4983-a338-1a7a4aa7e23e\") "
Sep 16 19:13:36.961302 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:36.961003    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-model-cache\") pod \"742a3dd2-dce5-4983-a338-1a7a4aa7e23e\" (UID: \"742a3dd2-dce5-4983-a338-1a7a4aa7e23e\") "
Sep 16 19:13:36.961302 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:36.961221    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "742a3dd2-dce5-4983-a338-1a7a4aa7e23e" (UID: "742a3dd2-dce5-4983-a338-1a7a4aa7e23e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:13:36.961423 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:36.961349    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-home" (OuterVolumeSpecName: "home") pod "742a3dd2-dce5-4983-a338-1a7a4aa7e23e" (UID: "742a3dd2-dce5-4983-a338-1a7a4aa7e23e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:36.961475 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:36.961459    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-model-cache" (OuterVolumeSpecName: "model-cache") pod "742a3dd2-dce5-4983-a338-1a7a4aa7e23e" (UID: "742a3dd2-dce5-4983-a338-1a7a4aa7e23e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:36.961707 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:36.961682    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "742a3dd2-dce5-4983-a338-1a7a4aa7e23e" (UID: "742a3dd2-dce5-4983-a338-1a7a4aa7e23e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:36.963399 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:36.963370    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "742a3dd2-dce5-4983-a338-1a7a4aa7e23e" (UID: "742a3dd2-dce5-4983-a338-1a7a4aa7e23e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:13:36.963552 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:36.963503    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-kube-api-access-bggzt" (OuterVolumeSpecName: "kube-api-access-bggzt") pod "742a3dd2-dce5-4983-a338-1a7a4aa7e23e" (UID: "742a3dd2-dce5-4983-a338-1a7a4aa7e23e"). InnerVolumeSpecName "kube-api-access-bggzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:13:36.963552 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:36.963518    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-dshm" (OuterVolumeSpecName: "dshm") pod "742a3dd2-dce5-4983-a338-1a7a4aa7e23e" (UID: "742a3dd2-dce5-4983-a338-1a7a4aa7e23e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:36.984204 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:36.984170    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "742a3dd2-dce5-4983-a338-1a7a4aa7e23e" (UID: "742a3dd2-dce5-4983-a338-1a7a4aa7e23e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:37.062136 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.062077    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-cabundle-cert\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:13:37.062136 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.062103    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bggzt\" (UniqueName: \"kubernetes.io/projected/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-kube-api-access-bggzt\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:13:37.062136 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.062118    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-dshm\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:13:37.062136 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.062132    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-tmp-dir\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:13:37.062342 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.062146    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-kserve-provision-location\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:13:37.062342 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.062157    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-tls-certs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:13:37.062342 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.062168    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-home\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:13:37.062342 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.062179    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/742a3dd2-dce5-4983-a338-1a7a4aa7e23e-model-cache\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:13:37.156184 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.156156    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-jpmjq"]
Sep 16 19:13:37.167124 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.167102    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7"
Sep 16 19:13:37.263184 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.263152    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a99ea9d5-1cb5-4e44-9f4e-7c082a354cce-cabundle-cert\") pod \"a99ea9d5-1cb5-4e44-9f4e-7c082a354cce\" (UID: \"a99ea9d5-1cb5-4e44-9f4e-7c082a354cce\") "
Sep 16 19:13:37.263184 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.263188    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a99ea9d5-1cb5-4e44-9f4e-7c082a354cce-kserve-provision-location\") pod \"a99ea9d5-1cb5-4e44-9f4e-7c082a354cce\" (UID: \"a99ea9d5-1cb5-4e44-9f4e-7c082a354cce\") "
Sep 16 19:13:37.263415 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.263217    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a99ea9d5-1cb5-4e44-9f4e-7c082a354cce-tmp-dir\") pod \"a99ea9d5-1cb5-4e44-9f4e-7c082a354cce\" (UID: \"a99ea9d5-1cb5-4e44-9f4e-7c082a354cce\") "
Sep 16 19:13:37.263415 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.263309    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a99ea9d5-1cb5-4e44-9f4e-7c082a354cce-tls-certs\") pod \"a99ea9d5-1cb5-4e44-9f4e-7c082a354cce\" (UID: \"a99ea9d5-1cb5-4e44-9f4e-7c082a354cce\") "
Sep 16 19:13:37.263631 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.263606    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a99ea9d5-1cb5-4e44-9f4e-7c082a354cce-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a99ea9d5-1cb5-4e44-9f4e-7c082a354cce" (UID: "a99ea9d5-1cb5-4e44-9f4e-7c082a354cce"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:13:37.263710 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.263670    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a99ea9d5-1cb5-4e44-9f4e-7c082a354cce-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a99ea9d5-1cb5-4e44-9f4e-7c082a354cce" (UID: "a99ea9d5-1cb5-4e44-9f4e-7c082a354cce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:37.265503 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.265479    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a99ea9d5-1cb5-4e44-9f4e-7c082a354cce-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a99ea9d5-1cb5-4e44-9f4e-7c082a354cce" (UID: "a99ea9d5-1cb5-4e44-9f4e-7c082a354cce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:13:37.284805 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.284782    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a99ea9d5-1cb5-4e44-9f4e-7c082a354cce-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a99ea9d5-1cb5-4e44-9f4e-7c082a354cce" (UID: "a99ea9d5-1cb5-4e44-9f4e-7c082a354cce"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:37.364155 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.364083    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a99ea9d5-1cb5-4e44-9f4e-7c082a354cce-tls-certs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:13:37.364155 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.364105    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a99ea9d5-1cb5-4e44-9f4e-7c082a354cce-cabundle-cert\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:13:37.364155 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.364118    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a99ea9d5-1cb5-4e44-9f4e-7c082a354cce-kserve-provision-location\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:13:37.364155 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.364132    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a99ea9d5-1cb5-4e44-9f4e-7c082a354cce-tmp-dir\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:13:37.725444 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.725403    2577 generic.go:356] "Generic (PLEG): container finished" podID="742a3dd2-dce5-4983-a338-1a7a4aa7e23e" containerID="9bd12f3f1569efb65d8c3d99053329dca58fad9a7d33723f2612688bd9e5cc1c" exitCode=0
Sep 16 19:13:37.725883 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.725461    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs"
Sep 16 19:13:37.725883 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.725479    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs" event={"ID":"742a3dd2-dce5-4983-a338-1a7a4aa7e23e","Type":"ContainerDied","Data":"9bd12f3f1569efb65d8c3d99053329dca58fad9a7d33723f2612688bd9e5cc1c"}
Sep 16 19:13:37.725883 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.725575    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs" event={"ID":"742a3dd2-dce5-4983-a338-1a7a4aa7e23e","Type":"ContainerDied","Data":"a9aba3a40a2c7a4eb9a1017a8b135900117dd0b2727f7b243fff561aeb53c369"}
Sep 16 19:13:37.725883 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.725641    2577 scope.go:117] "RemoveContainer" containerID="9bd12f3f1569efb65d8c3d99053329dca58fad9a7d33723f2612688bd9e5cc1c"
Sep 16 19:13:37.727683 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.727643    2577 generic.go:356] "Generic (PLEG): container finished" podID="a99ea9d5-1cb5-4e44-9f4e-7c082a354cce" containerID="2adb74afc7491b7d73889314d9cb92750bf72cbe7e741f11eee548c1a48813bc" exitCode=0
Sep 16 19:13:37.727811 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.727704    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7"
Sep 16 19:13:37.727811 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.727705    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7" event={"ID":"a99ea9d5-1cb5-4e44-9f4e-7c082a354cce","Type":"ContainerDied","Data":"2adb74afc7491b7d73889314d9cb92750bf72cbe7e741f11eee548c1a48813bc"}
Sep 16 19:13:37.727948 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.727903    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7" event={"ID":"a99ea9d5-1cb5-4e44-9f4e-7c082a354cce","Type":"ContainerDied","Data":"64808346373bc329360ab0c83f4dd433229c75e4c52f3c6d91dd56ec5e363d56"}
Sep 16 19:13:37.735560 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.735518    2577 scope.go:117] "RemoveContainer" containerID="72ae8c171f6a01b515c6b768157f3779cc452a955fa53c142d8401ef88455f9e"
Sep 16 19:13:37.745147 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.745132    2577 scope.go:117] "RemoveContainer" containerID="9bd12f3f1569efb65d8c3d99053329dca58fad9a7d33723f2612688bd9e5cc1c"
Sep 16 19:13:37.745403 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:13:37.745384    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bd12f3f1569efb65d8c3d99053329dca58fad9a7d33723f2612688bd9e5cc1c\": container with ID starting with 9bd12f3f1569efb65d8c3d99053329dca58fad9a7d33723f2612688bd9e5cc1c not found: ID does not exist" containerID="9bd12f3f1569efb65d8c3d99053329dca58fad9a7d33723f2612688bd9e5cc1c"
Sep 16 19:13:37.745466 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.745409    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bd12f3f1569efb65d8c3d99053329dca58fad9a7d33723f2612688bd9e5cc1c"} err="failed to get container status \"9bd12f3f1569efb65d8c3d99053329dca58fad9a7d33723f2612688bd9e5cc1c\": rpc error: code = NotFound desc = could not find container \"9bd12f3f1569efb65d8c3d99053329dca58fad9a7d33723f2612688bd9e5cc1c\": container with ID starting with 9bd12f3f1569efb65d8c3d99053329dca58fad9a7d33723f2612688bd9e5cc1c not found: ID does not exist"
Sep 16 19:13:37.745466 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.745424    2577 scope.go:117] "RemoveContainer" containerID="72ae8c171f6a01b515c6b768157f3779cc452a955fa53c142d8401ef88455f9e"
Sep 16 19:13:37.745686 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:13:37.745670    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72ae8c171f6a01b515c6b768157f3779cc452a955fa53c142d8401ef88455f9e\": container with ID starting with 72ae8c171f6a01b515c6b768157f3779cc452a955fa53c142d8401ef88455f9e not found: ID does not exist" containerID="72ae8c171f6a01b515c6b768157f3779cc452a955fa53c142d8401ef88455f9e"
Sep 16 19:13:37.745728 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.745692    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72ae8c171f6a01b515c6b768157f3779cc452a955fa53c142d8401ef88455f9e"} err="failed to get container status \"72ae8c171f6a01b515c6b768157f3779cc452a955fa53c142d8401ef88455f9e\": rpc error: code = NotFound desc = could not find container \"72ae8c171f6a01b515c6b768157f3779cc452a955fa53c142d8401ef88455f9e\": container with ID starting with 72ae8c171f6a01b515c6b768157f3779cc452a955fa53c142d8401ef88455f9e not found: ID does not exist"
Sep 16 19:13:37.745728 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.745708    2577 scope.go:117] "RemoveContainer" containerID="2adb74afc7491b7d73889314d9cb92750bf72cbe7e741f11eee548c1a48813bc"
Sep 16 19:13:37.752887 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.752867    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs"]
Sep 16 19:13:37.753049 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.753036    2577 scope.go:117] "RemoveContainer" containerID="c8df69e2b01afcba05291026fb2b0aec3312fc42bd5aa612263828ff595e1289"
Sep 16 19:13:37.757989 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.757970    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-pvfcs"]
Sep 16 19:13:37.762781 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.762715    2577 scope.go:117] "RemoveContainer" containerID="2adb74afc7491b7d73889314d9cb92750bf72cbe7e741f11eee548c1a48813bc"
Sep 16 19:13:37.763338 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:13:37.763315    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2adb74afc7491b7d73889314d9cb92750bf72cbe7e741f11eee548c1a48813bc\": container with ID starting with 2adb74afc7491b7d73889314d9cb92750bf72cbe7e741f11eee548c1a48813bc not found: ID does not exist" containerID="2adb74afc7491b7d73889314d9cb92750bf72cbe7e741f11eee548c1a48813bc"
Sep 16 19:13:37.763413 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.763348    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2adb74afc7491b7d73889314d9cb92750bf72cbe7e741f11eee548c1a48813bc"} err="failed to get container status \"2adb74afc7491b7d73889314d9cb92750bf72cbe7e741f11eee548c1a48813bc\": rpc error: code = NotFound desc = could not find container \"2adb74afc7491b7d73889314d9cb92750bf72cbe7e741f11eee548c1a48813bc\": container with ID starting with 2adb74afc7491b7d73889314d9cb92750bf72cbe7e741f11eee548c1a48813bc not found: ID does not exist"
Sep 16 19:13:37.763413 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.763370    2577 scope.go:117] "RemoveContainer" containerID="c8df69e2b01afcba05291026fb2b0aec3312fc42bd5aa612263828ff595e1289"
Sep 16 19:13:37.763722 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:13:37.763700    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8df69e2b01afcba05291026fb2b0aec3312fc42bd5aa612263828ff595e1289\": container with ID starting with c8df69e2b01afcba05291026fb2b0aec3312fc42bd5aa612263828ff595e1289 not found: ID does not exist" containerID="c8df69e2b01afcba05291026fb2b0aec3312fc42bd5aa612263828ff595e1289"
Sep 16 19:13:37.763786 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.763731    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8df69e2b01afcba05291026fb2b0aec3312fc42bd5aa612263828ff595e1289"} err="failed to get container status \"c8df69e2b01afcba05291026fb2b0aec3312fc42bd5aa612263828ff595e1289\": rpc error: code = NotFound desc = could not find container \"c8df69e2b01afcba05291026fb2b0aec3312fc42bd5aa612263828ff595e1289\": container with ID starting with c8df69e2b01afcba05291026fb2b0aec3312fc42bd5aa612263828ff595e1289 not found: ID does not exist"
Sep 16 19:13:37.774568 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.774546    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7"]
Sep 16 19:13:37.779205 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:37.779186    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-cbfff58fd-br5r7"]
Sep 16 19:13:39.155101 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:39.155067    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="742a3dd2-dce5-4983-a338-1a7a4aa7e23e" path="/var/lib/kubelet/pods/742a3dd2-dce5-4983-a338-1a7a4aa7e23e/volumes"
Sep 16 19:13:39.155670 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:39.155650    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a99ea9d5-1cb5-4e44-9f4e-7c082a354cce" path="/var/lib/kubelet/pods/a99ea9d5-1cb5-4e44-9f4e-7c082a354cce/volumes"
Sep 16 19:13:45.746662 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:45.746630    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-hdtd5"]
Sep 16 19:13:45.747022 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:45.746982    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a99ea9d5-1cb5-4e44-9f4e-7c082a354cce" containerName="storage-initializer"
Sep 16 19:13:45.747022 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:45.746994    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a99ea9d5-1cb5-4e44-9f4e-7c082a354cce" containerName="storage-initializer"
Sep 16 19:13:45.747022 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:45.747005    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a99ea9d5-1cb5-4e44-9f4e-7c082a354cce" containerName="main"
Sep 16 19:13:45.747022 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:45.747011    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a99ea9d5-1cb5-4e44-9f4e-7c082a354cce" containerName="main"
Sep 16 19:13:45.747022 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:45.747022    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="742a3dd2-dce5-4983-a338-1a7a4aa7e23e" containerName="storage-initializer"
Sep 16 19:13:45.747197 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:45.747028    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="742a3dd2-dce5-4983-a338-1a7a4aa7e23e" containerName="storage-initializer"
Sep 16 19:13:45.747197 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:45.747039    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="742a3dd2-dce5-4983-a338-1a7a4aa7e23e" containerName="main"
Sep 16 19:13:45.747197 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:45.747044    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="742a3dd2-dce5-4983-a338-1a7a4aa7e23e" containerName="main"
Sep 16 19:13:45.747197 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:45.747103    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="742a3dd2-dce5-4983-a338-1a7a4aa7e23e" containerName="main"
Sep 16 19:13:45.747197 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:45.747112    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a99ea9d5-1cb5-4e44-9f4e-7c082a354cce" containerName="main"
Sep 16 19:13:45.749921 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:45.749905    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-hdtd5"
Sep 16 19:13:45.754005 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:45.753981    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"decom-v1-kserve-self-signed-certs\""
Sep 16 19:13:45.754005 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:45.753998    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"kube-root-ca.crt\""
Sep 16 19:13:45.754218 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:45.754062    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"openshift-service-ca.crt\""
Sep 16 19:13:45.754218 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:45.754065    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"default-dockercfg-2r677\""
Sep 16 19:13:45.758956 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:45.758936    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-hdtd5"]
Sep 16 19:13:45.827735 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:45.827712    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfa344f3-edfc-4a3d-ade5-d63353234636-dshm\") pod \"decom-v1-kserve-6cb797fb75-hdtd5\" (UID: \"dfa344f3-edfc-4a3d-ade5-d63353234636\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-hdtd5"
Sep 16 19:13:45.827835 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:45.827748    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfa344f3-edfc-4a3d-ade5-d63353234636-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-hdtd5\" (UID: \"dfa344f3-edfc-4a3d-ade5-d63353234636\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-hdtd5"
Sep 16 19:13:45.827835 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:45.827770    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cxx9\" (UniqueName: \"kubernetes.io/projected/dfa344f3-edfc-4a3d-ade5-d63353234636-kube-api-access-9cxx9\") pod \"decom-v1-kserve-6cb797fb75-hdtd5\" (UID: \"dfa344f3-edfc-4a3d-ade5-d63353234636\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-hdtd5"
Sep 16 19:13:45.827918 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:45.827832    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfa344f3-edfc-4a3d-ade5-d63353234636-model-cache\") pod \"decom-v1-kserve-6cb797fb75-hdtd5\" (UID: \"dfa344f3-edfc-4a3d-ade5-d63353234636\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-hdtd5"
Sep 16 19:13:45.827918 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:45.827902    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfa344f3-edfc-4a3d-ade5-d63353234636-home\") pod \"decom-v1-kserve-6cb797fb75-hdtd5\" (UID: \"dfa344f3-edfc-4a3d-ade5-d63353234636\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-hdtd5"
Sep 16 19:13:45.827985 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:45.827938    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfa344f3-edfc-4a3d-ade5-d63353234636-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-hdtd5\" (UID: \"dfa344f3-edfc-4a3d-ade5-d63353234636\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-hdtd5"
Sep 16 19:13:45.928941 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:45.928919    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfa344f3-edfc-4a3d-ade5-d63353234636-dshm\") pod \"decom-v1-kserve-6cb797fb75-hdtd5\" (UID: \"dfa344f3-edfc-4a3d-ade5-d63353234636\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-hdtd5"
Sep 16 19:13:45.929036 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:45.928949    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfa344f3-edfc-4a3d-ade5-d63353234636-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-hdtd5\" (UID: \"dfa344f3-edfc-4a3d-ade5-d63353234636\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-hdtd5"
Sep 16 19:13:45.929036 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:45.928967    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cxx9\" (UniqueName: \"kubernetes.io/projected/dfa344f3-edfc-4a3d-ade5-d63353234636-kube-api-access-9cxx9\") pod \"decom-v1-kserve-6cb797fb75-hdtd5\" (UID: \"dfa344f3-edfc-4a3d-ade5-d63353234636\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-hdtd5"
Sep 16 19:13:45.929036 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:45.928986    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfa344f3-edfc-4a3d-ade5-d63353234636-model-cache\") pod \"decom-v1-kserve-6cb797fb75-hdtd5\" (UID: \"dfa344f3-edfc-4a3d-ade5-d63353234636\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-hdtd5"
Sep 16 19:13:45.929036 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:45.929015    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfa344f3-edfc-4a3d-ade5-d63353234636-home\") pod \"decom-v1-kserve-6cb797fb75-hdtd5\" (UID: \"dfa344f3-edfc-4a3d-ade5-d63353234636\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-hdtd5"
Sep 16 19:13:45.929247 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:45.929141    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfa344f3-edfc-4a3d-ade5-d63353234636-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-hdtd5\" (UID: \"dfa344f3-edfc-4a3d-ade5-d63353234636\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-hdtd5"
Sep 16 19:13:45.929443 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:45.929424    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfa344f3-edfc-4a3d-ade5-d63353234636-home\") pod \"decom-v1-kserve-6cb797fb75-hdtd5\" (UID: \"dfa344f3-edfc-4a3d-ade5-d63353234636\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-hdtd5"
Sep 16 19:13:45.929507 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:45.929459    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfa344f3-edfc-4a3d-ade5-d63353234636-model-cache\") pod \"decom-v1-kserve-6cb797fb75-hdtd5\" (UID: \"dfa344f3-edfc-4a3d-ade5-d63353234636\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-hdtd5"
Sep 16 19:13:45.929507 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:45.929482    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfa344f3-edfc-4a3d-ade5-d63353234636-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-hdtd5\" (UID: \"dfa344f3-edfc-4a3d-ade5-d63353234636\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-hdtd5"
Sep 16 19:13:45.931297 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:45.931275    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfa344f3-edfc-4a3d-ade5-d63353234636-dshm\") pod \"decom-v1-kserve-6cb797fb75-hdtd5\" (UID: \"dfa344f3-edfc-4a3d-ade5-d63353234636\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-hdtd5"
Sep 16 19:13:45.931654 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:45.931638    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfa344f3-edfc-4a3d-ade5-d63353234636-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-hdtd5\" (UID: \"dfa344f3-edfc-4a3d-ade5-d63353234636\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-hdtd5"
Sep 16 19:13:45.938194 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:45.938174    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cxx9\" (UniqueName: \"kubernetes.io/projected/dfa344f3-edfc-4a3d-ade5-d63353234636-kube-api-access-9cxx9\") pod \"decom-v1-kserve-6cb797fb75-hdtd5\" (UID: \"dfa344f3-edfc-4a3d-ade5-d63353234636\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-hdtd5"
Sep 16 19:13:46.061710 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:46.061654    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-hdtd5"
Sep 16 19:13:46.334503 ip-10-0-128-247 kubenswrapper[2577]: W0916 19:13:46.334463    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfa344f3_edfc_4a3d_ade5_d63353234636.slice/crio-0abc5ff8bdf9c3182de55e7a9ba7c149649958759f665bd5fab2fe464a8c1151 WatchSource:0}: Error finding container 0abc5ff8bdf9c3182de55e7a9ba7c149649958759f665bd5fab2fe464a8c1151: Status 404 returned error can't find the container with id 0abc5ff8bdf9c3182de55e7a9ba7c149649958759f665bd5fab2fe464a8c1151
Sep 16 19:13:46.335191 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:46.335168    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-hdtd5"]
Sep 16 19:13:46.771133 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:46.771094    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-hdtd5" event={"ID":"dfa344f3-edfc-4a3d-ade5-d63353234636","Type":"ContainerStarted","Data":"8992654ce0090b7b42fee401712089a562779baa269be8ad82805b59501589d3"}
Sep 16 19:13:46.771638 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:46.771166    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-hdtd5" event={"ID":"dfa344f3-edfc-4a3d-ade5-d63353234636","Type":"ContainerStarted","Data":"0abc5ff8bdf9c3182de55e7a9ba7c149649958759f665bd5fab2fe464a8c1151"}
Sep 16 19:13:46.793837 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:46.793789    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-hdtd5" podStartSLOduration=1.793773936 podStartE2EDuration="1.793773936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:13:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:13:46.792956336 +0000 UTC m=+1984.175845702" watchObservedRunningTime="2026-09-16 19:13:46.793773936 +0000 UTC m=+1984.176663300"
Sep 16 19:13:47.062039 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:47.061968    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-hdtd5"
Sep 16 19:13:52.176801 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:52.176745    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-jpmjq" podUID="033ee791-2955-49c9-9291-d3ac87fbaf65" containerName="main" containerID="cri-o://587bb22fe7286a548de1b4a0cc18d80ec300bb79beffead5cc5d73e825422682" gracePeriod=45
Sep 16 19:13:52.194637 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:52.194607    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-jpmjq" podUID="033ee791-2955-49c9-9291-d3ac87fbaf65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 16 19:13:52.418681 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:52.418657    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-jpmjq"
Sep 16 19:13:52.478516 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:52.478450    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/033ee791-2955-49c9-9291-d3ac87fbaf65-tmp-dir\") pod \"033ee791-2955-49c9-9291-d3ac87fbaf65\" (UID: \"033ee791-2955-49c9-9291-d3ac87fbaf65\") "
Sep 16 19:13:52.478516 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:52.478496    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/033ee791-2955-49c9-9291-d3ac87fbaf65-home\") pod \"033ee791-2955-49c9-9291-d3ac87fbaf65\" (UID: \"033ee791-2955-49c9-9291-d3ac87fbaf65\") "
Sep 16 19:13:52.478705 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:52.478561    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzd6v\" (UniqueName: \"kubernetes.io/projected/033ee791-2955-49c9-9291-d3ac87fbaf65-kube-api-access-mzd6v\") pod \"033ee791-2955-49c9-9291-d3ac87fbaf65\" (UID: \"033ee791-2955-49c9-9291-d3ac87fbaf65\") "
Sep 16 19:13:52.478705 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:52.478585    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/033ee791-2955-49c9-9291-d3ac87fbaf65-tls-certs\") pod \"033ee791-2955-49c9-9291-d3ac87fbaf65\" (UID: \"033ee791-2955-49c9-9291-d3ac87fbaf65\") "
Sep 16 19:13:52.478705 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:52.478662    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/033ee791-2955-49c9-9291-d3ac87fbaf65-dshm\") pod \"033ee791-2955-49c9-9291-d3ac87fbaf65\" (UID: \"033ee791-2955-49c9-9291-d3ac87fbaf65\") "
Sep 16 19:13:52.478866 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:52.478710    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/033ee791-2955-49c9-9291-d3ac87fbaf65-model-cache\") pod \"033ee791-2955-49c9-9291-d3ac87fbaf65\" (UID: \"033ee791-2955-49c9-9291-d3ac87fbaf65\") "
Sep 16 19:13:52.478866 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:52.478744    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/033ee791-2955-49c9-9291-d3ac87fbaf65-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "033ee791-2955-49c9-9291-d3ac87fbaf65" (UID: "033ee791-2955-49c9-9291-d3ac87fbaf65"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:52.478866 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:52.478792    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/033ee791-2955-49c9-9291-d3ac87fbaf65-home" (OuterVolumeSpecName: "home") pod "033ee791-2955-49c9-9291-d3ac87fbaf65" (UID: "033ee791-2955-49c9-9291-d3ac87fbaf65"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:52.479025 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:52.479007    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/033ee791-2955-49c9-9291-d3ac87fbaf65-tmp-dir\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:13:52.479078 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:52.479031    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/033ee791-2955-49c9-9291-d3ac87fbaf65-home\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:13:52.479078 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:52.479059    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/033ee791-2955-49c9-9291-d3ac87fbaf65-model-cache" (OuterVolumeSpecName: "model-cache") pod "033ee791-2955-49c9-9291-d3ac87fbaf65" (UID: "033ee791-2955-49c9-9291-d3ac87fbaf65"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:52.480927 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:52.480902    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/033ee791-2955-49c9-9291-d3ac87fbaf65-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "033ee791-2955-49c9-9291-d3ac87fbaf65" (UID: "033ee791-2955-49c9-9291-d3ac87fbaf65"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:13:52.481340 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:52.481325    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/033ee791-2955-49c9-9291-d3ac87fbaf65-kube-api-access-mzd6v" (OuterVolumeSpecName: "kube-api-access-mzd6v") pod "033ee791-2955-49c9-9291-d3ac87fbaf65" (UID: "033ee791-2955-49c9-9291-d3ac87fbaf65"). InnerVolumeSpecName "kube-api-access-mzd6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:13:52.481388 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:52.481335    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/033ee791-2955-49c9-9291-d3ac87fbaf65-dshm" (OuterVolumeSpecName: "dshm") pod "033ee791-2955-49c9-9291-d3ac87fbaf65" (UID: "033ee791-2955-49c9-9291-d3ac87fbaf65"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:52.579398 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:52.579372    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/033ee791-2955-49c9-9291-d3ac87fbaf65-model-cache\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:13:52.579398 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:52.579393    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzd6v\" (UniqueName: \"kubernetes.io/projected/033ee791-2955-49c9-9291-d3ac87fbaf65-kube-api-access-mzd6v\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:13:52.579398 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:52.579403    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/033ee791-2955-49c9-9291-d3ac87fbaf65-tls-certs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:13:52.579609 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:52.579411    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/033ee791-2955-49c9-9291-d3ac87fbaf65-dshm\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:13:52.796795 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:52.796713    2577 generic.go:356] "Generic (PLEG): container finished" podID="033ee791-2955-49c9-9291-d3ac87fbaf65" containerID="587bb22fe7286a548de1b4a0cc18d80ec300bb79beffead5cc5d73e825422682" exitCode=0
Sep 16 19:13:52.796795 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:52.796769    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-jpmjq"
Sep 16 19:13:52.796952 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:52.796772    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-jpmjq" event={"ID":"033ee791-2955-49c9-9291-d3ac87fbaf65","Type":"ContainerDied","Data":"587bb22fe7286a548de1b4a0cc18d80ec300bb79beffead5cc5d73e825422682"}
Sep 16 19:13:52.796952 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:52.796915    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-jpmjq" event={"ID":"033ee791-2955-49c9-9291-d3ac87fbaf65","Type":"ContainerDied","Data":"e6dec71a0b2d14dec7d268f4a73c2afe3e0bf0909847773a8673c9b2da0e7c00"}
Sep 16 19:13:52.796952 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:52.796943    2577 scope.go:117] "RemoveContainer" containerID="587bb22fe7286a548de1b4a0cc18d80ec300bb79beffead5cc5d73e825422682"
Sep 16 19:13:52.806219 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:52.806202    2577 scope.go:117] "RemoveContainer" containerID="587bb22fe7286a548de1b4a0cc18d80ec300bb79beffead5cc5d73e825422682"
Sep 16 19:13:52.806501 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:13:52.806473    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"587bb22fe7286a548de1b4a0cc18d80ec300bb79beffead5cc5d73e825422682\": container with ID starting with 587bb22fe7286a548de1b4a0cc18d80ec300bb79beffead5cc5d73e825422682 not found: ID does not exist" containerID="587bb22fe7286a548de1b4a0cc18d80ec300bb79beffead5cc5d73e825422682"
Sep 16 19:13:52.806670 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:52.806505    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"587bb22fe7286a548de1b4a0cc18d80ec300bb79beffead5cc5d73e825422682"} err="failed to get container status \"587bb22fe7286a548de1b4a0cc18d80ec300bb79beffead5cc5d73e825422682\": rpc error: code = NotFound desc = could not find container \"587bb22fe7286a548de1b4a0cc18d80ec300bb79beffead5cc5d73e825422682\": container with ID starting with 587bb22fe7286a548de1b4a0cc18d80ec300bb79beffead5cc5d73e825422682 not found: ID does not exist"
Sep 16 19:13:52.822727 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:52.822699    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-jpmjq"]
Sep 16 19:13:52.827131 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:52.827111    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-jpmjq"]
Sep 16 19:13:53.155152 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:53.155118    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="033ee791-2955-49c9-9291-d3ac87fbaf65" path="/var/lib/kubelet/pods/033ee791-2955-49c9-9291-d3ac87fbaf65/volumes"
Sep 16 19:13:56.062748 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:56.062714    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-hdtd5"
Sep 16 19:13:56.075436 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:56.075414    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-hdtd5"
Sep 16 19:13:56.827023 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:13:56.826994    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-hdtd5"
Sep 16 19:14:00.759622 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:00.759583    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-nznv7"]
Sep 16 19:14:00.760246 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:00.759942    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="033ee791-2955-49c9-9291-d3ac87fbaf65" containerName="main"
Sep 16 19:14:00.760246 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:00.759954    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="033ee791-2955-49c9-9291-d3ac87fbaf65" containerName="main"
Sep 16 19:14:00.760246 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:00.760023    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="033ee791-2955-49c9-9291-d3ac87fbaf65" containerName="main"
Sep 16 19:14:00.764566 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:00.764543    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-nznv7"
Sep 16 19:14:00.768196 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:00.768174    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"decom-v2-kserve-self-signed-certs\""
Sep 16 19:14:00.778116 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:00.778093    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-nznv7"]
Sep 16 19:14:00.839649 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:00.839622    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa65e5ac-7582-4580-acf8-9819039bab77-dshm\") pod \"decom-v2-kserve-77f867d787-nznv7\" (UID: \"fa65e5ac-7582-4580-acf8-9819039bab77\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-nznv7"
Sep 16 19:14:00.839774 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:00.839683    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znqd6\" (UniqueName: \"kubernetes.io/projected/fa65e5ac-7582-4580-acf8-9819039bab77-kube-api-access-znqd6\") pod \"decom-v2-kserve-77f867d787-nznv7\" (UID: \"fa65e5ac-7582-4580-acf8-9819039bab77\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-nznv7"
Sep 16 19:14:00.839774 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:00.839705    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa65e5ac-7582-4580-acf8-9819039bab77-home\") pod \"decom-v2-kserve-77f867d787-nznv7\" (UID: \"fa65e5ac-7582-4580-acf8-9819039bab77\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-nznv7"
Sep 16 19:14:00.839774 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:00.839729    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa65e5ac-7582-4580-acf8-9819039bab77-tls-certs\") pod \"decom-v2-kserve-77f867d787-nznv7\" (UID: \"fa65e5ac-7582-4580-acf8-9819039bab77\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-nznv7"
Sep 16 19:14:00.839774 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:00.839751    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa65e5ac-7582-4580-acf8-9819039bab77-tmp-dir\") pod \"decom-v2-kserve-77f867d787-nznv7\" (UID: \"fa65e5ac-7582-4580-acf8-9819039bab77\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-nznv7"
Sep 16 19:14:00.839920 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:00.839806    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa65e5ac-7582-4580-acf8-9819039bab77-model-cache\") pod \"decom-v2-kserve-77f867d787-nznv7\" (UID: \"fa65e5ac-7582-4580-acf8-9819039bab77\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-nznv7"
Sep 16 19:14:00.940881 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:00.940853    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znqd6\" (UniqueName: \"kubernetes.io/projected/fa65e5ac-7582-4580-acf8-9819039bab77-kube-api-access-znqd6\") pod \"decom-v2-kserve-77f867d787-nznv7\" (UID: \"fa65e5ac-7582-4580-acf8-9819039bab77\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-nznv7"
Sep 16 19:14:00.941010 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:00.940884    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa65e5ac-7582-4580-acf8-9819039bab77-home\") pod \"decom-v2-kserve-77f867d787-nznv7\" (UID: \"fa65e5ac-7582-4580-acf8-9819039bab77\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-nznv7"
Sep 16 19:14:00.941010 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:00.940909    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa65e5ac-7582-4580-acf8-9819039bab77-tls-certs\") pod \"decom-v2-kserve-77f867d787-nznv7\" (UID: \"fa65e5ac-7582-4580-acf8-9819039bab77\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-nznv7"
Sep 16 19:14:00.941010 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:00.940980    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa65e5ac-7582-4580-acf8-9819039bab77-tmp-dir\") pod \"decom-v2-kserve-77f867d787-nznv7\" (UID: \"fa65e5ac-7582-4580-acf8-9819039bab77\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-nznv7"
Sep 16 19:14:00.941193 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:00.941016    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa65e5ac-7582-4580-acf8-9819039bab77-model-cache\") pod \"decom-v2-kserve-77f867d787-nznv7\" (UID: \"fa65e5ac-7582-4580-acf8-9819039bab77\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-nznv7"
Sep 16 19:14:00.941193 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:00.941046    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa65e5ac-7582-4580-acf8-9819039bab77-dshm\") pod \"decom-v2-kserve-77f867d787-nznv7\" (UID: \"fa65e5ac-7582-4580-acf8-9819039bab77\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-nznv7"
Sep 16 19:14:00.941335 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:00.941313    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa65e5ac-7582-4580-acf8-9819039bab77-home\") pod \"decom-v2-kserve-77f867d787-nznv7\" (UID: \"fa65e5ac-7582-4580-acf8-9819039bab77\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-nznv7"
Sep 16 19:14:00.941396 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:00.941344    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa65e5ac-7582-4580-acf8-9819039bab77-tmp-dir\") pod \"decom-v2-kserve-77f867d787-nznv7\" (UID: \"fa65e5ac-7582-4580-acf8-9819039bab77\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-nznv7"
Sep 16 19:14:00.941441 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:00.941424    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa65e5ac-7582-4580-acf8-9819039bab77-model-cache\") pod \"decom-v2-kserve-77f867d787-nznv7\" (UID: \"fa65e5ac-7582-4580-acf8-9819039bab77\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-nznv7"
Sep 16 19:14:00.943358 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:00.943341    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa65e5ac-7582-4580-acf8-9819039bab77-dshm\") pod \"decom-v2-kserve-77f867d787-nznv7\" (UID: \"fa65e5ac-7582-4580-acf8-9819039bab77\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-nznv7"
Sep 16 19:14:00.943563 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:00.943548    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa65e5ac-7582-4580-acf8-9819039bab77-tls-certs\") pod \"decom-v2-kserve-77f867d787-nznv7\" (UID: \"fa65e5ac-7582-4580-acf8-9819039bab77\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-nznv7"
Sep 16 19:14:00.951459 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:00.951435    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znqd6\" (UniqueName: \"kubernetes.io/projected/fa65e5ac-7582-4580-acf8-9819039bab77-kube-api-access-znqd6\") pod \"decom-v2-kserve-77f867d787-nznv7\" (UID: \"fa65e5ac-7582-4580-acf8-9819039bab77\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-nznv7"
Sep 16 19:14:01.074598 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:01.074550    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-nznv7"
Sep 16 19:14:01.355967 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:01.355905    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-nznv7"]
Sep 16 19:14:01.358153 ip-10-0-128-247 kubenswrapper[2577]: W0916 19:14:01.358123    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa65e5ac_7582_4580_acf8_9819039bab77.slice/crio-166d0893cbfe723e6dfbadf92eaeb69c9ca2d1e507f6452a35c3e177151095c4 WatchSource:0}: Error finding container 166d0893cbfe723e6dfbadf92eaeb69c9ca2d1e507f6452a35c3e177151095c4: Status 404 returned error can't find the container with id 166d0893cbfe723e6dfbadf92eaeb69c9ca2d1e507f6452a35c3e177151095c4
Sep 16 19:14:01.844193 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:01.844157    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-nznv7" event={"ID":"fa65e5ac-7582-4580-acf8-9819039bab77","Type":"ContainerStarted","Data":"07163841bd4dda89dd1d0fa9ec4deee072b8f38daf5d38cae4e48f009c370db5"}
Sep 16 19:14:01.844609 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:01.844212    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-nznv7" event={"ID":"fa65e5ac-7582-4580-acf8-9819039bab77","Type":"ContainerStarted","Data":"166d0893cbfe723e6dfbadf92eaeb69c9ca2d1e507f6452a35c3e177151095c4"}
Sep 16 19:14:01.874656 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:01.874618    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-nznv7" podStartSLOduration=1.87460278 podStartE2EDuration="1.87460278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:01.87097903 +0000 UTC m=+1999.253868394" watchObservedRunningTime="2026-09-16 19:14:01.87460278 +0000 UTC m=+1999.257492145"
Sep 16 19:14:02.075072 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:02.075043    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-nznv7"
Sep 16 19:14:11.074783 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:11.074750    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-nznv7"
Sep 16 19:14:11.086911 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:11.086884    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-nznv7"
Sep 16 19:14:11.901297 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:11.901264    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-nznv7"
Sep 16 19:14:36.152034 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.152000    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn"]
Sep 16 19:14:36.156193 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.156172    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn"
Sep 16 19:14:36.159682 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.159660    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 16 19:14:36.159812 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.159733    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 16 19:14:36.159812 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.159776    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-x7z49\""
Sep 16 19:14:36.160912 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.160892    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:14:36.161096 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.161060    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 16 19:14:36.161188 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.161142    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 16 19:14:36.171436 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.171393    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn"]
Sep 16 19:14:36.199515 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.199485    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d68d796-ec11-41a8-a910-50a31086dff7-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-58487b448-rkhzn\" (UID: \"2d68d796-ec11-41a8-a910-50a31086dff7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn"
Sep 16 19:14:36.199515 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.199520    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d68d796-ec11-41a8-a910-50a31086dff7-tls-certs\") pod \"tokenizer-template-args-test-kserve-58487b448-rkhzn\" (UID: \"2d68d796-ec11-41a8-a910-50a31086dff7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn"
Sep 16 19:14:36.199692 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.199566    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d68d796-ec11-41a8-a910-50a31086dff7-model-cache\") pod \"tokenizer-template-args-test-kserve-58487b448-rkhzn\" (UID: \"2d68d796-ec11-41a8-a910-50a31086dff7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn"
Sep 16 19:14:36.199692 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.199605    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d68d796-ec11-41a8-a910-50a31086dff7-dshm\") pod \"tokenizer-template-args-test-kserve-58487b448-rkhzn\" (UID: \"2d68d796-ec11-41a8-a910-50a31086dff7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn"
Sep 16 19:14:36.199692 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.199662    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d68d796-ec11-41a8-a910-50a31086dff7-tmp-dir\") pod \"tokenizer-template-args-test-kserve-58487b448-rkhzn\" (UID: \"2d68d796-ec11-41a8-a910-50a31086dff7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn"
Sep 16 19:14:36.199692 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.199683    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d68d796-ec11-41a8-a910-50a31086dff7-home\") pod \"tokenizer-template-args-test-kserve-58487b448-rkhzn\" (UID: \"2d68d796-ec11-41a8-a910-50a31086dff7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn"
Sep 16 19:14:36.199860 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.199700    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8jmz\" (UniqueName: \"kubernetes.io/projected/2d68d796-ec11-41a8-a910-50a31086dff7-kube-api-access-g8jmz\") pod \"tokenizer-template-args-test-kserve-58487b448-rkhzn\" (UID: \"2d68d796-ec11-41a8-a910-50a31086dff7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn"
Sep 16 19:14:36.199860 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.199729    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d68d796-ec11-41a8-a910-50a31086dff7-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-58487b448-rkhzn\" (UID: \"2d68d796-ec11-41a8-a910-50a31086dff7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn"
Sep 16 19:14:36.300810 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.300784    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d68d796-ec11-41a8-a910-50a31086dff7-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-58487b448-rkhzn\" (UID: \"2d68d796-ec11-41a8-a910-50a31086dff7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn"
Sep 16 19:14:36.300909 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.300818    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d68d796-ec11-41a8-a910-50a31086dff7-tls-certs\") pod \"tokenizer-template-args-test-kserve-58487b448-rkhzn\" (UID: \"2d68d796-ec11-41a8-a910-50a31086dff7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn"
Sep 16 19:14:36.300909 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.300854    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d68d796-ec11-41a8-a910-50a31086dff7-model-cache\") pod \"tokenizer-template-args-test-kserve-58487b448-rkhzn\" (UID: \"2d68d796-ec11-41a8-a910-50a31086dff7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn"
Sep 16 19:14:36.300909 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.300883    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d68d796-ec11-41a8-a910-50a31086dff7-dshm\") pod \"tokenizer-template-args-test-kserve-58487b448-rkhzn\" (UID: \"2d68d796-ec11-41a8-a910-50a31086dff7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn"
Sep 16 19:14:36.301052 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.300912    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d68d796-ec11-41a8-a910-50a31086dff7-tmp-dir\") pod \"tokenizer-template-args-test-kserve-58487b448-rkhzn\" (UID: \"2d68d796-ec11-41a8-a910-50a31086dff7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn"
Sep 16 19:14:36.301052 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.300940    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d68d796-ec11-41a8-a910-50a31086dff7-home\") pod \"tokenizer-template-args-test-kserve-58487b448-rkhzn\" (UID: \"2d68d796-ec11-41a8-a910-50a31086dff7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn"
Sep 16 19:14:36.301052 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.300965    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8jmz\" (UniqueName: \"kubernetes.io/projected/2d68d796-ec11-41a8-a910-50a31086dff7-kube-api-access-g8jmz\") pod \"tokenizer-template-args-test-kserve-58487b448-rkhzn\" (UID: \"2d68d796-ec11-41a8-a910-50a31086dff7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn"
Sep 16 19:14:36.301052 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.300995    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d68d796-ec11-41a8-a910-50a31086dff7-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-58487b448-rkhzn\" (UID: \"2d68d796-ec11-41a8-a910-50a31086dff7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn"
Sep 16 19:14:36.301590 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.301568    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d68d796-ec11-41a8-a910-50a31086dff7-home\") pod \"tokenizer-template-args-test-kserve-58487b448-rkhzn\" (UID: \"2d68d796-ec11-41a8-a910-50a31086dff7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn"
Sep 16 19:14:36.301676 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.301586    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d68d796-ec11-41a8-a910-50a31086dff7-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-58487b448-rkhzn\" (UID: \"2d68d796-ec11-41a8-a910-50a31086dff7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn"
Sep 16 19:14:36.301676 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.301599    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d68d796-ec11-41a8-a910-50a31086dff7-tmp-dir\") pod \"tokenizer-template-args-test-kserve-58487b448-rkhzn\" (UID: \"2d68d796-ec11-41a8-a910-50a31086dff7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn"
Sep 16 19:14:36.301676 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.301630    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d68d796-ec11-41a8-a910-50a31086dff7-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-58487b448-rkhzn\" (UID: \"2d68d796-ec11-41a8-a910-50a31086dff7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn"
Sep 16 19:14:36.301676 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.301663    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d68d796-ec11-41a8-a910-50a31086dff7-model-cache\") pod \"tokenizer-template-args-test-kserve-58487b448-rkhzn\" (UID: \"2d68d796-ec11-41a8-a910-50a31086dff7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn"
Sep 16 19:14:36.303484 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.303464    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d68d796-ec11-41a8-a910-50a31086dff7-tls-certs\") pod \"tokenizer-template-args-test-kserve-58487b448-rkhzn\" (UID: \"2d68d796-ec11-41a8-a910-50a31086dff7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn"
Sep 16 19:14:36.303634 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.303616    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d68d796-ec11-41a8-a910-50a31086dff7-dshm\") pod \"tokenizer-template-args-test-kserve-58487b448-rkhzn\" (UID: \"2d68d796-ec11-41a8-a910-50a31086dff7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn"
Sep 16 19:14:36.311024 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.311006    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8jmz\" (UniqueName: \"kubernetes.io/projected/2d68d796-ec11-41a8-a910-50a31086dff7-kube-api-access-g8jmz\") pod \"tokenizer-template-args-test-kserve-58487b448-rkhzn\" (UID: \"2d68d796-ec11-41a8-a910-50a31086dff7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn"
Sep 16 19:14:36.393258 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.393221    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85495646bb-prqsh"]
Sep 16 19:14:36.398015 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.397997    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85495646bb-prqsh"
Sep 16 19:14:36.402241 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.402185    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21309066-3633-41aa-8cba-7dd583a84a43-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-85495646bb-prqsh\" (UID: \"21309066-3633-41aa-8cba-7dd583a84a43\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85495646bb-prqsh"
Sep 16 19:14:36.402339 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.402242    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21309066-3633-41aa-8cba-7dd583a84a43-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-85495646bb-prqsh\" (UID: \"21309066-3633-41aa-8cba-7dd583a84a43\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85495646bb-prqsh"
Sep 16 19:14:36.402339 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.402271    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21309066-3633-41aa-8cba-7dd583a84a43-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-85495646bb-prqsh\" (UID: \"21309066-3633-41aa-8cba-7dd583a84a43\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85495646bb-prqsh"
Sep 16 19:14:36.402339 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.402293    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21309066-3633-41aa-8cba-7dd583a84a43-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-85495646bb-prqsh\" (UID: \"21309066-3633-41aa-8cba-7dd583a84a43\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85495646bb-prqsh"
Sep 16 19:14:36.409009 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.408986    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85495646bb-prqsh"]
Sep 16 19:14:36.466449 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.466412    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn"
Sep 16 19:14:36.502781 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.502752    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21309066-3633-41aa-8cba-7dd583a84a43-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-85495646bb-prqsh\" (UID: \"21309066-3633-41aa-8cba-7dd583a84a43\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85495646bb-prqsh"
Sep 16 19:14:36.502900 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.502802    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21309066-3633-41aa-8cba-7dd583a84a43-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-85495646bb-prqsh\" (UID: \"21309066-3633-41aa-8cba-7dd583a84a43\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85495646bb-prqsh"
Sep 16 19:14:36.502900 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.502822    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21309066-3633-41aa-8cba-7dd583a84a43-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-85495646bb-prqsh\" (UID: \"21309066-3633-41aa-8cba-7dd583a84a43\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85495646bb-prqsh"
Sep 16 19:14:36.502900 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.502846    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21309066-3633-41aa-8cba-7dd583a84a43-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-85495646bb-prqsh\" (UID: \"21309066-3633-41aa-8cba-7dd583a84a43\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85495646bb-prqsh"
Sep 16 19:14:36.503244 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.503195    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21309066-3633-41aa-8cba-7dd583a84a43-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-85495646bb-prqsh\" (UID: \"21309066-3633-41aa-8cba-7dd583a84a43\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85495646bb-prqsh"
Sep 16 19:14:36.503244 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.503236    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21309066-3633-41aa-8cba-7dd583a84a43-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-85495646bb-prqsh\" (UID: \"21309066-3633-41aa-8cba-7dd583a84a43\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85495646bb-prqsh"
Sep 16 19:14:36.503670 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.503614    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21309066-3633-41aa-8cba-7dd583a84a43-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-85495646bb-prqsh\" (UID: \"21309066-3633-41aa-8cba-7dd583a84a43\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85495646bb-prqsh"
Sep 16 19:14:36.505653 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.505631    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21309066-3633-41aa-8cba-7dd583a84a43-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-85495646bb-prqsh\" (UID: \"21309066-3633-41aa-8cba-7dd583a84a43\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85495646bb-prqsh"
Sep 16 19:14:36.710884 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.710821    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85495646bb-prqsh"
Sep 16 19:14:36.745079 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.745053    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn"]
Sep 16 19:14:36.746561 ip-10-0-128-247 kubenswrapper[2577]: W0916 19:14:36.746504    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d68d796_ec11_41a8_a910_50a31086dff7.slice/crio-fd32ea10462f1568de714618ce360cadce4d211860ea5788e7586638b0ba3f97 WatchSource:0}: Error finding container fd32ea10462f1568de714618ce360cadce4d211860ea5788e7586638b0ba3f97: Status 404 returned error can't find the container with id fd32ea10462f1568de714618ce360cadce4d211860ea5788e7586638b0ba3f97
Sep 16 19:14:36.986655 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.986632    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85495646bb-prqsh"]
Sep 16 19:14:36.989023 ip-10-0-128-247 kubenswrapper[2577]: W0916 19:14:36.988997    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21309066_3633_41aa_8cba_7dd583a84a43.slice/crio-64730047309bcaf9e447f7f0ebaa23732ae7653692fe8634d93eeb72c3ba313f WatchSource:0}: Error finding container 64730047309bcaf9e447f7f0ebaa23732ae7653692fe8634d93eeb72c3ba313f: Status 404 returned error can't find the container with id 64730047309bcaf9e447f7f0ebaa23732ae7653692fe8634d93eeb72c3ba313f
Sep 16 19:14:36.998591 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:36.998558    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85495646bb-prqsh" event={"ID":"21309066-3633-41aa-8cba-7dd583a84a43","Type":"ContainerStarted","Data":"64730047309bcaf9e447f7f0ebaa23732ae7653692fe8634d93eeb72c3ba313f"}
Sep 16 19:14:37.000220 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:37.000190    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn" event={"ID":"2d68d796-ec11-41a8-a910-50a31086dff7","Type":"ContainerStarted","Data":"3a27ed979fcee948f396aff915db83c950e3bc8b13ac78afe3dc0a04bd709625"}
Sep 16 19:14:37.000332 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:37.000246    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn" event={"ID":"2d68d796-ec11-41a8-a910-50a31086dff7","Type":"ContainerStarted","Data":"fd32ea10462f1568de714618ce360cadce4d211860ea5788e7586638b0ba3f97"}
Sep 16 19:14:38.011015 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:38.010973    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85495646bb-prqsh" event={"ID":"21309066-3633-41aa-8cba-7dd583a84a43","Type":"ContainerStarted","Data":"05543afabb3b40eb42d535758422ee7b8e6cea64c49e13b9a06440a14f6e4d4f"}
Sep 16 19:14:40.022909 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:40.022876    2577 generic.go:356] "Generic (PLEG): container finished" podID="21309066-3633-41aa-8cba-7dd583a84a43" containerID="05543afabb3b40eb42d535758422ee7b8e6cea64c49e13b9a06440a14f6e4d4f" exitCode=0
Sep 16 19:14:40.023307 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:40.022959    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85495646bb-prqsh" event={"ID":"21309066-3633-41aa-8cba-7dd583a84a43","Type":"ContainerDied","Data":"05543afabb3b40eb42d535758422ee7b8e6cea64c49e13b9a06440a14f6e4d4f"}
Sep 16 19:14:40.024679 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:40.024656    2577 generic.go:356] "Generic (PLEG): container finished" podID="2d68d796-ec11-41a8-a910-50a31086dff7" containerID="3a27ed979fcee948f396aff915db83c950e3bc8b13ac78afe3dc0a04bd709625" exitCode=0
Sep 16 19:14:40.024784 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:40.024728    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn" event={"ID":"2d68d796-ec11-41a8-a910-50a31086dff7","Type":"ContainerDied","Data":"3a27ed979fcee948f396aff915db83c950e3bc8b13ac78afe3dc0a04bd709625"}
Sep 16 19:14:40.569105 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:40.569073    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-nznv7"]
Sep 16 19:14:41.031856 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:41.031811    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85495646bb-prqsh" event={"ID":"21309066-3633-41aa-8cba-7dd583a84a43","Type":"ContainerStarted","Data":"641b2dd0686a7a6c09859e54ee0a76020b487d0145c52f03296673e1284be778"}
Sep 16 19:14:41.032299 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:41.032059    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85495646bb-prqsh"
Sep 16 19:14:41.034076 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:41.034054    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn" event={"ID":"2d68d796-ec11-41a8-a910-50a31086dff7","Type":"ContainerStarted","Data":"b89d51470373537e713ec015e0ae3a50bf471a9c4fd68697ac47030dd1ff9323"}
Sep 16 19:14:41.057443 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:41.057386    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85495646bb-prqsh" podStartSLOduration=2.66557277 podStartE2EDuration="5.057367354s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.391794584s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:41.055098517 +0000 UTC m=+2038.437987903" watchObservedRunningTime="2026-09-16 19:14:41.057367354 +0000 UTC m=+2038.440256721"
Sep 16 19:14:41.079399 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:41.079350    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn" podStartSLOduration=2.677505154 podStartE2EDuration="5.079333498s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.401828344s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:41.078221219 +0000 UTC m=+2038.461110586" watchObservedRunningTime="2026-09-16 19:14:41.079333498 +0000 UTC m=+2038.462222864"
Sep 16 19:14:41.467192 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:41.467157    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn"
Sep 16 19:14:46.467291 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:46.467244    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn"
Sep 16 19:14:46.479891 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:46.479867    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn"
Sep 16 19:14:47.075351 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:47.075323    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn"
Sep 16 19:14:55.588281 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:55.588214    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-nznv7" podUID="fa65e5ac-7582-4580-acf8-9819039bab77" containerName="main" containerID="cri-o://07163841bd4dda89dd1d0fa9ec4deee072b8f38daf5d38cae4e48f009c370db5" gracePeriod=45
Sep 16 19:14:55.847372 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:55.847318    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-nznv7"
Sep 16 19:14:55.974707 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:55.974674    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znqd6\" (UniqueName: \"kubernetes.io/projected/fa65e5ac-7582-4580-acf8-9819039bab77-kube-api-access-znqd6\") pod \"fa65e5ac-7582-4580-acf8-9819039bab77\" (UID: \"fa65e5ac-7582-4580-acf8-9819039bab77\") "
Sep 16 19:14:55.974867 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:55.974723    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa65e5ac-7582-4580-acf8-9819039bab77-tmp-dir\") pod \"fa65e5ac-7582-4580-acf8-9819039bab77\" (UID: \"fa65e5ac-7582-4580-acf8-9819039bab77\") "
Sep 16 19:14:55.974867 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:55.974800    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa65e5ac-7582-4580-acf8-9819039bab77-model-cache\") pod \"fa65e5ac-7582-4580-acf8-9819039bab77\" (UID: \"fa65e5ac-7582-4580-acf8-9819039bab77\") "
Sep 16 19:14:55.974867 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:55.974842    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa65e5ac-7582-4580-acf8-9819039bab77-tls-certs\") pod \"fa65e5ac-7582-4580-acf8-9819039bab77\" (UID: \"fa65e5ac-7582-4580-acf8-9819039bab77\") "
Sep 16 19:14:55.975038 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:55.974886    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa65e5ac-7582-4580-acf8-9819039bab77-dshm\") pod \"fa65e5ac-7582-4580-acf8-9819039bab77\" (UID: \"fa65e5ac-7582-4580-acf8-9819039bab77\") "
Sep 16 19:14:55.975038 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:55.974914    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa65e5ac-7582-4580-acf8-9819039bab77-home\") pod \"fa65e5ac-7582-4580-acf8-9819039bab77\" (UID: \"fa65e5ac-7582-4580-acf8-9819039bab77\") "
Sep 16 19:14:55.975130 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:55.975061    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa65e5ac-7582-4580-acf8-9819039bab77-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fa65e5ac-7582-4580-acf8-9819039bab77" (UID: "fa65e5ac-7582-4580-acf8-9819039bab77"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:55.975195 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:55.975126    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa65e5ac-7582-4580-acf8-9819039bab77-model-cache" (OuterVolumeSpecName: "model-cache") pod "fa65e5ac-7582-4580-acf8-9819039bab77" (UID: "fa65e5ac-7582-4580-acf8-9819039bab77"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:55.975251 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:55.975195    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa65e5ac-7582-4580-acf8-9819039bab77-tmp-dir\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:14:55.975306 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:55.975264    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa65e5ac-7582-4580-acf8-9819039bab77-home" (OuterVolumeSpecName: "home") pod "fa65e5ac-7582-4580-acf8-9819039bab77" (UID: "fa65e5ac-7582-4580-acf8-9819039bab77"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:55.977284 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:55.977250    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa65e5ac-7582-4580-acf8-9819039bab77-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fa65e5ac-7582-4580-acf8-9819039bab77" (UID: "fa65e5ac-7582-4580-acf8-9819039bab77"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:14:55.977284 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:55.977268    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa65e5ac-7582-4580-acf8-9819039bab77-dshm" (OuterVolumeSpecName: "dshm") pod "fa65e5ac-7582-4580-acf8-9819039bab77" (UID: "fa65e5ac-7582-4580-acf8-9819039bab77"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:55.977444 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:55.977308    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa65e5ac-7582-4580-acf8-9819039bab77-kube-api-access-znqd6" (OuterVolumeSpecName: "kube-api-access-znqd6") pod "fa65e5ac-7582-4580-acf8-9819039bab77" (UID: "fa65e5ac-7582-4580-acf8-9819039bab77"). InnerVolumeSpecName "kube-api-access-znqd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:14:56.076200 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:56.076170    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa65e5ac-7582-4580-acf8-9819039bab77-home\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:14:56.076200 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:56.076193    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znqd6\" (UniqueName: \"kubernetes.io/projected/fa65e5ac-7582-4580-acf8-9819039bab77-kube-api-access-znqd6\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:14:56.076200 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:56.076203    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa65e5ac-7582-4580-acf8-9819039bab77-model-cache\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:14:56.076429 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:56.076212    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa65e5ac-7582-4580-acf8-9819039bab77-tls-certs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:14:56.076429 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:56.076219    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa65e5ac-7582-4580-acf8-9819039bab77-dshm\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:14:56.107288 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:56.107205    2577 generic.go:356] "Generic (PLEG): container finished" podID="fa65e5ac-7582-4580-acf8-9819039bab77" containerID="07163841bd4dda89dd1d0fa9ec4deee072b8f38daf5d38cae4e48f009c370db5" exitCode=0
Sep 16 19:14:56.107288 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:56.107254    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-nznv7"
Sep 16 19:14:56.107453 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:56.107250    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-nznv7" event={"ID":"fa65e5ac-7582-4580-acf8-9819039bab77","Type":"ContainerDied","Data":"07163841bd4dda89dd1d0fa9ec4deee072b8f38daf5d38cae4e48f009c370db5"}
Sep 16 19:14:56.107453 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:56.107392    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-nznv7" event={"ID":"fa65e5ac-7582-4580-acf8-9819039bab77","Type":"ContainerDied","Data":"166d0893cbfe723e6dfbadf92eaeb69c9ca2d1e507f6452a35c3e177151095c4"}
Sep 16 19:14:56.107453 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:56.107423    2577 scope.go:117] "RemoveContainer" containerID="07163841bd4dda89dd1d0fa9ec4deee072b8f38daf5d38cae4e48f009c370db5"
Sep 16 19:14:56.116665 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:56.116621    2577 scope.go:117] "RemoveContainer" containerID="07163841bd4dda89dd1d0fa9ec4deee072b8f38daf5d38cae4e48f009c370db5"
Sep 16 19:14:56.116906 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:14:56.116887    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07163841bd4dda89dd1d0fa9ec4deee072b8f38daf5d38cae4e48f009c370db5\": container with ID starting with 07163841bd4dda89dd1d0fa9ec4deee072b8f38daf5d38cae4e48f009c370db5 not found: ID does not exist" containerID="07163841bd4dda89dd1d0fa9ec4deee072b8f38daf5d38cae4e48f009c370db5"
Sep 16 19:14:56.116961 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:56.116914    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07163841bd4dda89dd1d0fa9ec4deee072b8f38daf5d38cae4e48f009c370db5"} err="failed to get container status \"07163841bd4dda89dd1d0fa9ec4deee072b8f38daf5d38cae4e48f009c370db5\": rpc error: code = NotFound desc = could not find container \"07163841bd4dda89dd1d0fa9ec4deee072b8f38daf5d38cae4e48f009c370db5\": container with ID starting with 07163841bd4dda89dd1d0fa9ec4deee072b8f38daf5d38cae4e48f009c370db5 not found: ID does not exist"
Sep 16 19:14:56.133746 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:56.133720    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-nznv7"]
Sep 16 19:14:56.136154 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:56.136121    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-nznv7"]
Sep 16 19:14:57.156884 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:57.156842    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa65e5ac-7582-4580-acf8-9819039bab77" path="/var/lib/kubelet/pods/fa65e5ac-7582-4580-acf8-9819039bab77/volumes"
Sep 16 19:14:57.712394 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:14:57.712360    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-hdtd5"]
Sep 16 19:15:12.049083 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:12.049005    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85495646bb-prqsh"
Sep 16 19:15:12.732133 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:12.732090    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-hdtd5" podUID="dfa344f3-edfc-4a3d-ade5-d63353234636" containerName="main" containerID="cri-o://8992654ce0090b7b42fee401712089a562779baa269be8ad82805b59501589d3" gracePeriod=45
Sep 16 19:15:12.815997 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:12.815961    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-hdtd5" podUID="dfa344f3-edfc-4a3d-ade5-d63353234636" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 16 19:15:12.979724 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:12.979702    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-hdtd5"
Sep 16 19:15:13.115342 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:13.115281    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfa344f3-edfc-4a3d-ade5-d63353234636-dshm\") pod \"dfa344f3-edfc-4a3d-ade5-d63353234636\" (UID: \"dfa344f3-edfc-4a3d-ade5-d63353234636\") "
Sep 16 19:15:13.115776 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:13.115343    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfa344f3-edfc-4a3d-ade5-d63353234636-tls-certs\") pod \"dfa344f3-edfc-4a3d-ade5-d63353234636\" (UID: \"dfa344f3-edfc-4a3d-ade5-d63353234636\") "
Sep 16 19:15:13.115776 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:13.115369    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfa344f3-edfc-4a3d-ade5-d63353234636-tmp-dir\") pod \"dfa344f3-edfc-4a3d-ade5-d63353234636\" (UID: \"dfa344f3-edfc-4a3d-ade5-d63353234636\") "
Sep 16 19:15:13.115776 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:13.115397    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfa344f3-edfc-4a3d-ade5-d63353234636-home\") pod \"dfa344f3-edfc-4a3d-ade5-d63353234636\" (UID: \"dfa344f3-edfc-4a3d-ade5-d63353234636\") "
Sep 16 19:15:13.115776 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:13.115449    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cxx9\" (UniqueName: \"kubernetes.io/projected/dfa344f3-edfc-4a3d-ade5-d63353234636-kube-api-access-9cxx9\") pod \"dfa344f3-edfc-4a3d-ade5-d63353234636\" (UID: \"dfa344f3-edfc-4a3d-ade5-d63353234636\") "
Sep 16 19:15:13.115776 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:13.115483    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfa344f3-edfc-4a3d-ade5-d63353234636-model-cache\") pod \"dfa344f3-edfc-4a3d-ade5-d63353234636\" (UID: \"dfa344f3-edfc-4a3d-ade5-d63353234636\") "
Sep 16 19:15:13.115776 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:13.115704    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfa344f3-edfc-4a3d-ade5-d63353234636-home" (OuterVolumeSpecName: "home") pod "dfa344f3-edfc-4a3d-ade5-d63353234636" (UID: "dfa344f3-edfc-4a3d-ade5-d63353234636"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:13.115776 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:13.115719    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfa344f3-edfc-4a3d-ade5-d63353234636-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dfa344f3-edfc-4a3d-ade5-d63353234636" (UID: "dfa344f3-edfc-4a3d-ade5-d63353234636"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:13.116058 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:13.115794    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfa344f3-edfc-4a3d-ade5-d63353234636-model-cache" (OuterVolumeSpecName: "model-cache") pod "dfa344f3-edfc-4a3d-ade5-d63353234636" (UID: "dfa344f3-edfc-4a3d-ade5-d63353234636"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:13.116058 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:13.115816    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfa344f3-edfc-4a3d-ade5-d63353234636-tmp-dir\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:15:13.116058 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:13.115835    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfa344f3-edfc-4a3d-ade5-d63353234636-home\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:15:13.117609 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:13.117575    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfa344f3-edfc-4a3d-ade5-d63353234636-dshm" (OuterVolumeSpecName: "dshm") pod "dfa344f3-edfc-4a3d-ade5-d63353234636" (UID: "dfa344f3-edfc-4a3d-ade5-d63353234636"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:13.117721 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:13.117660    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfa344f3-edfc-4a3d-ade5-d63353234636-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dfa344f3-edfc-4a3d-ade5-d63353234636" (UID: "dfa344f3-edfc-4a3d-ade5-d63353234636"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:15:13.117769 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:13.117719    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfa344f3-edfc-4a3d-ade5-d63353234636-kube-api-access-9cxx9" (OuterVolumeSpecName: "kube-api-access-9cxx9") pod "dfa344f3-edfc-4a3d-ade5-d63353234636" (UID: "dfa344f3-edfc-4a3d-ade5-d63353234636"). InnerVolumeSpecName "kube-api-access-9cxx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:15:13.187699 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:13.187677    2577 generic.go:356] "Generic (PLEG): container finished" podID="dfa344f3-edfc-4a3d-ade5-d63353234636" containerID="8992654ce0090b7b42fee401712089a562779baa269be8ad82805b59501589d3" exitCode=0
Sep 16 19:15:13.187806 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:13.187718    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-hdtd5"
Sep 16 19:15:13.187806 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:13.187759    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-hdtd5" event={"ID":"dfa344f3-edfc-4a3d-ade5-d63353234636","Type":"ContainerDied","Data":"8992654ce0090b7b42fee401712089a562779baa269be8ad82805b59501589d3"}
Sep 16 19:15:13.187907 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:13.187807    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-hdtd5" event={"ID":"dfa344f3-edfc-4a3d-ade5-d63353234636","Type":"ContainerDied","Data":"0abc5ff8bdf9c3182de55e7a9ba7c149649958759f665bd5fab2fe464a8c1151"}
Sep 16 19:15:13.187907 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:13.187828    2577 scope.go:117] "RemoveContainer" containerID="8992654ce0090b7b42fee401712089a562779baa269be8ad82805b59501589d3"
Sep 16 19:15:13.195979 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:13.195964    2577 scope.go:117] "RemoveContainer" containerID="8992654ce0090b7b42fee401712089a562779baa269be8ad82805b59501589d3"
Sep 16 19:15:13.196271 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:15:13.196251    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8992654ce0090b7b42fee401712089a562779baa269be8ad82805b59501589d3\": container with ID starting with 8992654ce0090b7b42fee401712089a562779baa269be8ad82805b59501589d3 not found: ID does not exist" containerID="8992654ce0090b7b42fee401712089a562779baa269be8ad82805b59501589d3"
Sep 16 19:15:13.196357 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:13.196275    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8992654ce0090b7b42fee401712089a562779baa269be8ad82805b59501589d3"} err="failed to get container status \"8992654ce0090b7b42fee401712089a562779baa269be8ad82805b59501589d3\": rpc error: code = NotFound desc = could not find container \"8992654ce0090b7b42fee401712089a562779baa269be8ad82805b59501589d3\": container with ID starting with 8992654ce0090b7b42fee401712089a562779baa269be8ad82805b59501589d3 not found: ID does not exist"
Sep 16 19:15:13.207200 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:13.207176    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-hdtd5"]
Sep 16 19:15:13.213322 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:13.213301    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-hdtd5"]
Sep 16 19:15:13.217048 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:13.217032    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfa344f3-edfc-4a3d-ade5-d63353234636-model-cache\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:15:13.217099 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:13.217051    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfa344f3-edfc-4a3d-ade5-d63353234636-dshm\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:15:13.217099 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:13.217060    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfa344f3-edfc-4a3d-ade5-d63353234636-tls-certs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:15:13.217099 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:13.217068    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cxx9\" (UniqueName: \"kubernetes.io/projected/dfa344f3-edfc-4a3d-ade5-d63353234636-kube-api-access-9cxx9\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:15:15.155070 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:15.155037    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfa344f3-edfc-4a3d-ade5-d63353234636" path="/var/lib/kubelet/pods/dfa344f3-edfc-4a3d-ade5-d63353234636/volumes"
Sep 16 19:15:37.498241 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:37.498211    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-0eb24714_tokenizer-template-args-test-tokenizer-85495646bb-prqsh_21309066-3633-41aa-8cba-7dd583a84a43/main/0.log"
Sep 16 19:15:50.102811 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.102779    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85495646bb-prqsh"]
Sep 16 19:15:50.103222 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.102987    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85495646bb-prqsh" podUID="21309066-3633-41aa-8cba-7dd583a84a43" containerName="main" containerID="cri-o://641b2dd0686a7a6c09859e54ee0a76020b487d0145c52f03296673e1284be778" gracePeriod=30
Sep 16 19:15:50.114418 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.114390    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn"]
Sep 16 19:15:50.123624 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:15:50.123588    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:15:50.123624 ip-10-0-128-247 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:15:50.123624 ip-10-0-128-247 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 19:15:50.123624 ip-10-0-128-247 kubenswrapper[2577]:  > containerID="b89d51470373537e713ec015e0ae3a50bf471a9c4fd68697ac47030dd1ff9323" cmd=["/bin/sleep","15"]
Sep 16 19:15:50.123864 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.123640    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn" message="" err=<
Sep 16 19:15:50.123864 ip-10-0-128-247 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:15:50.123864 ip-10-0-128-247 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 19:15:50.123864 ip-10-0-128-247 kubenswrapper[2577]:  >
Sep 16 19:15:50.123864 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:15:50.123655    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:15:50.123864 ip-10-0-128-247 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:15:50.123864 ip-10-0-128-247 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 19:15:50.123864 ip-10-0-128-247 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn" podUID="2d68d796-ec11-41a8-a910-50a31086dff7" containerName="main" containerID="cri-o://b89d51470373537e713ec015e0ae3a50bf471a9c4fd68697ac47030dd1ff9323"
Sep 16 19:15:50.123864 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.123695    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn" podUID="2d68d796-ec11-41a8-a910-50a31086dff7" containerName="main" containerID="cri-o://b89d51470373537e713ec015e0ae3a50bf471a9c4fd68697ac47030dd1ff9323" gracePeriod=60
Sep 16 19:15:50.350607 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.350567    2577 generic.go:356] "Generic (PLEG): container finished" podID="2d68d796-ec11-41a8-a910-50a31086dff7" containerID="b89d51470373537e713ec015e0ae3a50bf471a9c4fd68697ac47030dd1ff9323" exitCode=0
Sep 16 19:15:50.350764 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.350638    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn" event={"ID":"2d68d796-ec11-41a8-a910-50a31086dff7","Type":"ContainerDied","Data":"b89d51470373537e713ec015e0ae3a50bf471a9c4fd68697ac47030dd1ff9323"}
Sep 16 19:15:50.471835 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.471805    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn"
Sep 16 19:15:50.485961 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.485937    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d68d796-ec11-41a8-a910-50a31086dff7-tls-certs\") pod \"2d68d796-ec11-41a8-a910-50a31086dff7\" (UID: \"2d68d796-ec11-41a8-a910-50a31086dff7\") "
Sep 16 19:15:50.486066 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.486016    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d68d796-ec11-41a8-a910-50a31086dff7-kserve-provision-location\") pod \"2d68d796-ec11-41a8-a910-50a31086dff7\" (UID: \"2d68d796-ec11-41a8-a910-50a31086dff7\") "
Sep 16 19:15:50.486066 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.486049    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d68d796-ec11-41a8-a910-50a31086dff7-dshm\") pod \"2d68d796-ec11-41a8-a910-50a31086dff7\" (UID: \"2d68d796-ec11-41a8-a910-50a31086dff7\") "
Sep 16 19:15:50.486204 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.486105    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d68d796-ec11-41a8-a910-50a31086dff7-cabundle-cert\") pod \"2d68d796-ec11-41a8-a910-50a31086dff7\" (UID: \"2d68d796-ec11-41a8-a910-50a31086dff7\") "
Sep 16 19:15:50.486204 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.486149    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d68d796-ec11-41a8-a910-50a31086dff7-home\") pod \"2d68d796-ec11-41a8-a910-50a31086dff7\" (UID: \"2d68d796-ec11-41a8-a910-50a31086dff7\") "
Sep 16 19:15:50.486204 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.486176    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8jmz\" (UniqueName: \"kubernetes.io/projected/2d68d796-ec11-41a8-a910-50a31086dff7-kube-api-access-g8jmz\") pod \"2d68d796-ec11-41a8-a910-50a31086dff7\" (UID: \"2d68d796-ec11-41a8-a910-50a31086dff7\") "
Sep 16 19:15:50.486363 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.486234    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d68d796-ec11-41a8-a910-50a31086dff7-tmp-dir\") pod \"2d68d796-ec11-41a8-a910-50a31086dff7\" (UID: \"2d68d796-ec11-41a8-a910-50a31086dff7\") "
Sep 16 19:15:50.486363 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.486301    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d68d796-ec11-41a8-a910-50a31086dff7-model-cache\") pod \"2d68d796-ec11-41a8-a910-50a31086dff7\" (UID: \"2d68d796-ec11-41a8-a910-50a31086dff7\") "
Sep 16 19:15:50.486459 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.486402    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d68d796-ec11-41a8-a910-50a31086dff7-home" (OuterVolumeSpecName: "home") pod "2d68d796-ec11-41a8-a910-50a31086dff7" (UID: "2d68d796-ec11-41a8-a910-50a31086dff7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:50.486459 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.486430    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d68d796-ec11-41a8-a910-50a31086dff7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2d68d796-ec11-41a8-a910-50a31086dff7" (UID: "2d68d796-ec11-41a8-a910-50a31086dff7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:15:50.486613 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.486569    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d68d796-ec11-41a8-a910-50a31086dff7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2d68d796-ec11-41a8-a910-50a31086dff7" (UID: "2d68d796-ec11-41a8-a910-50a31086dff7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:50.486613 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.486588    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d68d796-ec11-41a8-a910-50a31086dff7-cabundle-cert\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:15:50.486613 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.486611    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d68d796-ec11-41a8-a910-50a31086dff7-home\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:15:50.486762 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.486687    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d68d796-ec11-41a8-a910-50a31086dff7-model-cache" (OuterVolumeSpecName: "model-cache") pod "2d68d796-ec11-41a8-a910-50a31086dff7" (UID: "2d68d796-ec11-41a8-a910-50a31086dff7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:50.498120 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.498090    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d68d796-ec11-41a8-a910-50a31086dff7-dshm" (OuterVolumeSpecName: "dshm") pod "2d68d796-ec11-41a8-a910-50a31086dff7" (UID: "2d68d796-ec11-41a8-a910-50a31086dff7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:50.509657 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.509632    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d68d796-ec11-41a8-a910-50a31086dff7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2d68d796-ec11-41a8-a910-50a31086dff7" (UID: "2d68d796-ec11-41a8-a910-50a31086dff7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:50.510957 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.510906    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d68d796-ec11-41a8-a910-50a31086dff7-kube-api-access-g8jmz" (OuterVolumeSpecName: "kube-api-access-g8jmz") pod "2d68d796-ec11-41a8-a910-50a31086dff7" (UID: "2d68d796-ec11-41a8-a910-50a31086dff7"). InnerVolumeSpecName "kube-api-access-g8jmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:15:50.510957 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.510919    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d68d796-ec11-41a8-a910-50a31086dff7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2d68d796-ec11-41a8-a910-50a31086dff7" (UID: "2d68d796-ec11-41a8-a910-50a31086dff7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:15:50.587216 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.587192    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d68d796-ec11-41a8-a910-50a31086dff7-dshm\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:15:50.587216 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.587217    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8jmz\" (UniqueName: \"kubernetes.io/projected/2d68d796-ec11-41a8-a910-50a31086dff7-kube-api-access-g8jmz\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:15:50.587370 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.587229    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d68d796-ec11-41a8-a910-50a31086dff7-tmp-dir\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:15:50.587370 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.587240    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d68d796-ec11-41a8-a910-50a31086dff7-model-cache\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:15:50.587370 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.587247    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d68d796-ec11-41a8-a910-50a31086dff7-tls-certs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:15:50.587370 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.587255    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d68d796-ec11-41a8-a910-50a31086dff7-kserve-provision-location\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:15:50.643499 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.643447    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85495646bb-prqsh"
Sep 16 19:15:50.688462 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.688438    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21309066-3633-41aa-8cba-7dd583a84a43-kserve-provision-location\") pod \"21309066-3633-41aa-8cba-7dd583a84a43\" (UID: \"21309066-3633-41aa-8cba-7dd583a84a43\") "
Sep 16 19:15:50.688600 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.688484    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21309066-3633-41aa-8cba-7dd583a84a43-cabundle-cert\") pod \"21309066-3633-41aa-8cba-7dd583a84a43\" (UID: \"21309066-3633-41aa-8cba-7dd583a84a43\") "
Sep 16 19:15:50.688600 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.688546    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21309066-3633-41aa-8cba-7dd583a84a43-tls-certs\") pod \"21309066-3633-41aa-8cba-7dd583a84a43\" (UID: \"21309066-3633-41aa-8cba-7dd583a84a43\") "
Sep 16 19:15:50.688737 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.688606    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21309066-3633-41aa-8cba-7dd583a84a43-tmp-dir\") pod \"21309066-3633-41aa-8cba-7dd583a84a43\" (UID: \"21309066-3633-41aa-8cba-7dd583a84a43\") "
Sep 16 19:15:50.688925 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.688902    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21309066-3633-41aa-8cba-7dd583a84a43-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "21309066-3633-41aa-8cba-7dd583a84a43" (UID: "21309066-3633-41aa-8cba-7dd583a84a43"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:15:50.689123 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.689097    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21309066-3633-41aa-8cba-7dd583a84a43-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "21309066-3633-41aa-8cba-7dd583a84a43" (UID: "21309066-3633-41aa-8cba-7dd583a84a43"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:50.690846 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.690827    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21309066-3633-41aa-8cba-7dd583a84a43-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "21309066-3633-41aa-8cba-7dd583a84a43" (UID: "21309066-3633-41aa-8cba-7dd583a84a43"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:15:50.710381 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.710357    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21309066-3633-41aa-8cba-7dd583a84a43-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "21309066-3633-41aa-8cba-7dd583a84a43" (UID: "21309066-3633-41aa-8cba-7dd583a84a43"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:50.790098 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.790075    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21309066-3633-41aa-8cba-7dd583a84a43-cabundle-cert\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:15:50.790098 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.790096    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21309066-3633-41aa-8cba-7dd583a84a43-tls-certs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:15:50.790227 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.790104    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21309066-3633-41aa-8cba-7dd583a84a43-tmp-dir\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:15:50.790227 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:50.790113    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21309066-3633-41aa-8cba-7dd583a84a43-kserve-provision-location\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:15:51.356120 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:51.356095    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn"
Sep 16 19:15:51.356460 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:51.356115    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn" event={"ID":"2d68d796-ec11-41a8-a910-50a31086dff7","Type":"ContainerDied","Data":"fd32ea10462f1568de714618ce360cadce4d211860ea5788e7586638b0ba3f97"}
Sep 16 19:15:51.356460 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:51.356179    2577 scope.go:117] "RemoveContainer" containerID="b89d51470373537e713ec015e0ae3a50bf471a9c4fd68697ac47030dd1ff9323"
Sep 16 19:15:51.357906 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:51.357883    2577 generic.go:356] "Generic (PLEG): container finished" podID="21309066-3633-41aa-8cba-7dd583a84a43" containerID="641b2dd0686a7a6c09859e54ee0a76020b487d0145c52f03296673e1284be778" exitCode=0
Sep 16 19:15:51.358052 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:51.357969    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85495646bb-prqsh" event={"ID":"21309066-3633-41aa-8cba-7dd583a84a43","Type":"ContainerDied","Data":"641b2dd0686a7a6c09859e54ee0a76020b487d0145c52f03296673e1284be778"}
Sep 16 19:15:51.358170 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:51.358084    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85495646bb-prqsh" event={"ID":"21309066-3633-41aa-8cba-7dd583a84a43","Type":"ContainerDied","Data":"64730047309bcaf9e447f7f0ebaa23732ae7653692fe8634d93eeb72c3ba313f"}
Sep 16 19:15:51.358170 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:51.358036    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85495646bb-prqsh"
Sep 16 19:15:51.365205 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:51.365186    2577 scope.go:117] "RemoveContainer" containerID="3a27ed979fcee948f396aff915db83c950e3bc8b13ac78afe3dc0a04bd709625"
Sep 16 19:15:51.372666 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:51.372649    2577 scope.go:117] "RemoveContainer" containerID="641b2dd0686a7a6c09859e54ee0a76020b487d0145c52f03296673e1284be778"
Sep 16 19:15:51.379994 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:51.379972    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn"]
Sep 16 19:15:51.381212 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:51.381196    2577 scope.go:117] "RemoveContainer" containerID="05543afabb3b40eb42d535758422ee7b8e6cea64c49e13b9a06440a14f6e4d4f"
Sep 16 19:15:51.384620 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:51.384600    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-rkhzn"]
Sep 16 19:15:51.388583 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:51.388566    2577 scope.go:117] "RemoveContainer" containerID="641b2dd0686a7a6c09859e54ee0a76020b487d0145c52f03296673e1284be778"
Sep 16 19:15:51.388849 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:15:51.388830    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"641b2dd0686a7a6c09859e54ee0a76020b487d0145c52f03296673e1284be778\": container with ID starting with 641b2dd0686a7a6c09859e54ee0a76020b487d0145c52f03296673e1284be778 not found: ID does not exist" containerID="641b2dd0686a7a6c09859e54ee0a76020b487d0145c52f03296673e1284be778"
Sep 16 19:15:51.388909 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:51.388854    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"641b2dd0686a7a6c09859e54ee0a76020b487d0145c52f03296673e1284be778"} err="failed to get container status \"641b2dd0686a7a6c09859e54ee0a76020b487d0145c52f03296673e1284be778\": rpc error: code = NotFound desc = could not find container \"641b2dd0686a7a6c09859e54ee0a76020b487d0145c52f03296673e1284be778\": container with ID starting with 641b2dd0686a7a6c09859e54ee0a76020b487d0145c52f03296673e1284be778 not found: ID does not exist"
Sep 16 19:15:51.388909 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:51.388869    2577 scope.go:117] "RemoveContainer" containerID="05543afabb3b40eb42d535758422ee7b8e6cea64c49e13b9a06440a14f6e4d4f"
Sep 16 19:15:51.389131 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:15:51.389112    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05543afabb3b40eb42d535758422ee7b8e6cea64c49e13b9a06440a14f6e4d4f\": container with ID starting with 05543afabb3b40eb42d535758422ee7b8e6cea64c49e13b9a06440a14f6e4d4f not found: ID does not exist" containerID="05543afabb3b40eb42d535758422ee7b8e6cea64c49e13b9a06440a14f6e4d4f"
Sep 16 19:15:51.389169 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:51.389136    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05543afabb3b40eb42d535758422ee7b8e6cea64c49e13b9a06440a14f6e4d4f"} err="failed to get container status \"05543afabb3b40eb42d535758422ee7b8e6cea64c49e13b9a06440a14f6e4d4f\": rpc error: code = NotFound desc = could not find container \"05543afabb3b40eb42d535758422ee7b8e6cea64c49e13b9a06440a14f6e4d4f\": container with ID starting with 05543afabb3b40eb42d535758422ee7b8e6cea64c49e13b9a06440a14f6e4d4f not found: ID does not exist"
Sep 16 19:15:51.402284 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:51.402265    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85495646bb-prqsh"]
Sep 16 19:15:51.407436 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:51.407413    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-85495646bb-prqsh"]
Sep 16 19:15:53.158422 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:53.158382    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21309066-3633-41aa-8cba-7dd583a84a43" path="/var/lib/kubelet/pods/21309066-3633-41aa-8cba-7dd583a84a43/volumes"
Sep 16 19:15:53.159047 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:15:53.159029    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d68d796-ec11-41a8-a910-50a31086dff7" path="/var/lib/kubelet/pods/2d68d796-ec11-41a8-a910-50a31086dff7/volumes"
Sep 16 19:16:06.463036 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:06.463004    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-58r8s"]
Sep 16 19:16:06.463425 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:06.463347    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfa344f3-edfc-4a3d-ade5-d63353234636" containerName="main"
Sep 16 19:16:06.463425 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:06.463357    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfa344f3-edfc-4a3d-ade5-d63353234636" containerName="main"
Sep 16 19:16:06.463425 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:06.463365    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d68d796-ec11-41a8-a910-50a31086dff7" containerName="main"
Sep 16 19:16:06.463425 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:06.463370    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d68d796-ec11-41a8-a910-50a31086dff7" containerName="main"
Sep 16 19:16:06.463425 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:06.463377    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d68d796-ec11-41a8-a910-50a31086dff7" containerName="storage-initializer"
Sep 16 19:16:06.463425 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:06.463383    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d68d796-ec11-41a8-a910-50a31086dff7" containerName="storage-initializer"
Sep 16 19:16:06.463425 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:06.463395    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa65e5ac-7582-4580-acf8-9819039bab77" containerName="main"
Sep 16 19:16:06.463425 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:06.463400    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa65e5ac-7582-4580-acf8-9819039bab77" containerName="main"
Sep 16 19:16:06.463425 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:06.463409    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21309066-3633-41aa-8cba-7dd583a84a43" containerName="storage-initializer"
Sep 16 19:16:06.463425 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:06.463414    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="21309066-3633-41aa-8cba-7dd583a84a43" containerName="storage-initializer"
Sep 16 19:16:06.463425 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:06.463420    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21309066-3633-41aa-8cba-7dd583a84a43" containerName="main"
Sep 16 19:16:06.463425 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:06.463425    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="21309066-3633-41aa-8cba-7dd583a84a43" containerName="main"
Sep 16 19:16:06.463901 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:06.463473    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d68d796-ec11-41a8-a910-50a31086dff7" containerName="main"
Sep 16 19:16:06.463901 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:06.463481    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa65e5ac-7582-4580-acf8-9819039bab77" containerName="main"
Sep 16 19:16:06.463901 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:06.463489    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfa344f3-edfc-4a3d-ade5-d63353234636" containerName="main"
Sep 16 19:16:06.463901 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:06.463496    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="21309066-3633-41aa-8cba-7dd583a84a43" containerName="main"
Sep 16 19:16:06.468154 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:06.468133    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-58r8s"
Sep 16 19:16:06.471302 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:06.471280    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v1-kserve-self-signed-certs\""
Sep 16 19:16:06.472476 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:06.472457    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"default-dockercfg-92568\""
Sep 16 19:16:06.472600 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:06.472492    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"kube-root-ca.crt\""
Sep 16 19:16:06.472600 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:06.472561    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"openshift-service-ca.crt\""
Sep 16 19:16:06.487939 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:06.487915    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-58r8s"]
Sep 16 19:16:06.506002 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:06.505980    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34f8b194-16b7-465d-b3b2-5d369bc33b55-home\") pod \"tri-v1-kserve-79f8f98bc5-58r8s\" (UID: \"34f8b194-16b7-465d-b3b2-5d369bc33b55\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-58r8s"
Sep 16 19:16:06.506098 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:06.506023    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34f8b194-16b7-465d-b3b2-5d369bc33b55-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-58r8s\" (UID: \"34f8b194-16b7-465d-b3b2-5d369bc33b55\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-58r8s"
Sep 16 19:16:06.506340 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:06.506096    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34f8b194-16b7-465d-b3b2-5d369bc33b55-dshm\") pod \"tri-v1-kserve-79f8f98bc5-58r8s\" (UID: \"34f8b194-16b7-465d-b3b2-5d369bc33b55\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-58r8s"
Sep 16 19:16:06.506340 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:06.506156    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34f8b194-16b7-465d-b3b2-5d369bc33b55-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-58r8s\" (UID: \"34f8b194-16b7-465d-b3b2-5d369bc33b55\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-58r8s"
Sep 16 19:16:06.506340 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:06.506196    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34f8b194-16b7-465d-b3b2-5d369bc33b55-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-58r8s\" (UID: \"34f8b194-16b7-465d-b3b2-5d369bc33b55\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-58r8s"
Sep 16 19:16:06.506340 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:06.506233    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqn9g\" (UniqueName: \"kubernetes.io/projected/34f8b194-16b7-465d-b3b2-5d369bc33b55-kube-api-access-gqn9g\") pod \"tri-v1-kserve-79f8f98bc5-58r8s\" (UID: \"34f8b194-16b7-465d-b3b2-5d369bc33b55\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-58r8s"
Sep 16 19:16:06.607015 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:06.606990    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34f8b194-16b7-465d-b3b2-5d369bc33b55-dshm\") pod \"tri-v1-kserve-79f8f98bc5-58r8s\" (UID: \"34f8b194-16b7-465d-b3b2-5d369bc33b55\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-58r8s"
Sep 16 19:16:06.607117 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:06.607025    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34f8b194-16b7-465d-b3b2-5d369bc33b55-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-58r8s\" (UID: \"34f8b194-16b7-465d-b3b2-5d369bc33b55\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-58r8s"
Sep 16 19:16:06.607117 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:06.607060    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34f8b194-16b7-465d-b3b2-5d369bc33b55-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-58r8s\" (UID: \"34f8b194-16b7-465d-b3b2-5d369bc33b55\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-58r8s"
Sep 16 19:16:06.607117 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:06.607089    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqn9g\" (UniqueName: \"kubernetes.io/projected/34f8b194-16b7-465d-b3b2-5d369bc33b55-kube-api-access-gqn9g\") pod \"tri-v1-kserve-79f8f98bc5-58r8s\" (UID: \"34f8b194-16b7-465d-b3b2-5d369bc33b55\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-58r8s"
Sep 16 19:16:06.607237 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:06.607132    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34f8b194-16b7-465d-b3b2-5d369bc33b55-home\") pod \"tri-v1-kserve-79f8f98bc5-58r8s\" (UID: \"34f8b194-16b7-465d-b3b2-5d369bc33b55\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-58r8s"
Sep 16 19:16:06.607237 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:06.607166    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34f8b194-16b7-465d-b3b2-5d369bc33b55-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-58r8s\" (UID: \"34f8b194-16b7-465d-b3b2-5d369bc33b55\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-58r8s"
Sep 16 19:16:06.607441 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:06.607419    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34f8b194-16b7-465d-b3b2-5d369bc33b55-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-58r8s\" (UID: \"34f8b194-16b7-465d-b3b2-5d369bc33b55\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-58r8s"
Sep 16 19:16:06.607505 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:06.607477    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34f8b194-16b7-465d-b3b2-5d369bc33b55-home\") pod \"tri-v1-kserve-79f8f98bc5-58r8s\" (UID: \"34f8b194-16b7-465d-b3b2-5d369bc33b55\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-58r8s"
Sep 16 19:16:06.607589 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:06.607522    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34f8b194-16b7-465d-b3b2-5d369bc33b55-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-58r8s\" (UID: \"34f8b194-16b7-465d-b3b2-5d369bc33b55\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-58r8s"
Sep 16 19:16:06.609511 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:06.609490    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34f8b194-16b7-465d-b3b2-5d369bc33b55-dshm\") pod \"tri-v1-kserve-79f8f98bc5-58r8s\" (UID: \"34f8b194-16b7-465d-b3b2-5d369bc33b55\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-58r8s"
Sep 16 19:16:06.609911 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:06.609892    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34f8b194-16b7-465d-b3b2-5d369bc33b55-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-58r8s\" (UID: \"34f8b194-16b7-465d-b3b2-5d369bc33b55\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-58r8s"
Sep 16 19:16:06.617023 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:06.617005    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqn9g\" (UniqueName: \"kubernetes.io/projected/34f8b194-16b7-465d-b3b2-5d369bc33b55-kube-api-access-gqn9g\") pod \"tri-v1-kserve-79f8f98bc5-58r8s\" (UID: \"34f8b194-16b7-465d-b3b2-5d369bc33b55\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-58r8s"
Sep 16 19:16:06.778642 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:06.778586    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-58r8s"
Sep 16 19:16:07.053029 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:07.053006    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-58r8s"]
Sep 16 19:16:07.054727 ip-10-0-128-247 kubenswrapper[2577]: W0916 19:16:07.054700    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34f8b194_16b7_465d_b3b2_5d369bc33b55.slice/crio-a49cb7dfb5fb43f74a3f53e1833ab5e2776b0a9cda9799ed281ab8d4a45e1577 WatchSource:0}: Error finding container a49cb7dfb5fb43f74a3f53e1833ab5e2776b0a9cda9799ed281ab8d4a45e1577: Status 404 returned error can't find the container with id a49cb7dfb5fb43f74a3f53e1833ab5e2776b0a9cda9799ed281ab8d4a45e1577
Sep 16 19:16:07.431674 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:07.431640    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-58r8s" event={"ID":"34f8b194-16b7-465d-b3b2-5d369bc33b55","Type":"ContainerStarted","Data":"b6c04ab12e2655df35c737cd9489a2a4c5dbe62236bd0a15c0888004cb8bc2e6"}
Sep 16 19:16:07.431878 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:07.431698    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-58r8s" event={"ID":"34f8b194-16b7-465d-b3b2-5d369bc33b55","Type":"ContainerStarted","Data":"a49cb7dfb5fb43f74a3f53e1833ab5e2776b0a9cda9799ed281ab8d4a45e1577"}
Sep 16 19:16:07.452385 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:07.452337    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-58r8s" podStartSLOduration=1.452322293 podStartE2EDuration="1.452322293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:16:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:16:07.451608997 +0000 UTC m=+2124.834498363" watchObservedRunningTime="2026-09-16 19:16:07.452322293 +0000 UTC m=+2124.835211660"
Sep 16 19:16:07.779314 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:07.779239    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-58r8s"
Sep 16 19:16:16.779045 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:16.779003    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-58r8s"
Sep 16 19:16:16.791796 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:16.791769    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-58r8s"
Sep 16 19:16:17.488190 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:17.488161    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-58r8s"
Sep 16 19:16:21.029569 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:21.029518    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-vcffz"]
Sep 16 19:16:21.032880 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:21.032862    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-vcffz"
Sep 16 19:16:21.035671 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:21.035650    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v2-kserve-self-signed-certs\""
Sep 16 19:16:21.044681 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:21.044656    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-vcffz"]
Sep 16 19:16:21.114264 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:21.114237    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/082e4754-ea62-459c-89d2-47b2cfa63906-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-vcffz\" (UID: \"082e4754-ea62-459c-89d2-47b2cfa63906\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-vcffz"
Sep 16 19:16:21.114409 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:21.114275    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/082e4754-ea62-459c-89d2-47b2cfa63906-dshm\") pod \"tri-v2-kserve-dcb554bf5-vcffz\" (UID: \"082e4754-ea62-459c-89d2-47b2cfa63906\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-vcffz"
Sep 16 19:16:21.114409 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:21.114309    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/082e4754-ea62-459c-89d2-47b2cfa63906-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-vcffz\" (UID: \"082e4754-ea62-459c-89d2-47b2cfa63906\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-vcffz"
Sep 16 19:16:21.114409 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:21.114367    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/082e4754-ea62-459c-89d2-47b2cfa63906-home\") pod \"tri-v2-kserve-dcb554bf5-vcffz\" (UID: \"082e4754-ea62-459c-89d2-47b2cfa63906\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-vcffz"
Sep 16 19:16:21.114409 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:21.114389    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/082e4754-ea62-459c-89d2-47b2cfa63906-model-cache\") pod \"tri-v2-kserve-dcb554bf5-vcffz\" (UID: \"082e4754-ea62-459c-89d2-47b2cfa63906\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-vcffz"
Sep 16 19:16:21.114613 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:21.114410    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c69j\" (UniqueName: \"kubernetes.io/projected/082e4754-ea62-459c-89d2-47b2cfa63906-kube-api-access-2c69j\") pod \"tri-v2-kserve-dcb554bf5-vcffz\" (UID: \"082e4754-ea62-459c-89d2-47b2cfa63906\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-vcffz"
Sep 16 19:16:21.215196 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:21.215167    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/082e4754-ea62-459c-89d2-47b2cfa63906-dshm\") pod \"tri-v2-kserve-dcb554bf5-vcffz\" (UID: \"082e4754-ea62-459c-89d2-47b2cfa63906\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-vcffz"
Sep 16 19:16:21.215338 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:21.215217    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/082e4754-ea62-459c-89d2-47b2cfa63906-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-vcffz\" (UID: \"082e4754-ea62-459c-89d2-47b2cfa63906\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-vcffz"
Sep 16 19:16:21.215338 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:21.215263    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/082e4754-ea62-459c-89d2-47b2cfa63906-home\") pod \"tri-v2-kserve-dcb554bf5-vcffz\" (UID: \"082e4754-ea62-459c-89d2-47b2cfa63906\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-vcffz"
Sep 16 19:16:21.215436 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:21.215407    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/082e4754-ea62-459c-89d2-47b2cfa63906-model-cache\") pod \"tri-v2-kserve-dcb554bf5-vcffz\" (UID: \"082e4754-ea62-459c-89d2-47b2cfa63906\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-vcffz"
Sep 16 19:16:21.215491 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:21.215446    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c69j\" (UniqueName: \"kubernetes.io/projected/082e4754-ea62-459c-89d2-47b2cfa63906-kube-api-access-2c69j\") pod \"tri-v2-kserve-dcb554bf5-vcffz\" (UID: \"082e4754-ea62-459c-89d2-47b2cfa63906\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-vcffz"
Sep 16 19:16:21.215589 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:21.215572    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/082e4754-ea62-459c-89d2-47b2cfa63906-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-vcffz\" (UID: \"082e4754-ea62-459c-89d2-47b2cfa63906\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-vcffz"
Sep 16 19:16:21.215702 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:21.215674    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/082e4754-ea62-459c-89d2-47b2cfa63906-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-vcffz\" (UID: \"082e4754-ea62-459c-89d2-47b2cfa63906\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-vcffz"
Sep 16 19:16:21.215765 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:21.215711    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/082e4754-ea62-459c-89d2-47b2cfa63906-home\") pod \"tri-v2-kserve-dcb554bf5-vcffz\" (UID: \"082e4754-ea62-459c-89d2-47b2cfa63906\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-vcffz"
Sep 16 19:16:21.215765 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:21.215732    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/082e4754-ea62-459c-89d2-47b2cfa63906-model-cache\") pod \"tri-v2-kserve-dcb554bf5-vcffz\" (UID: \"082e4754-ea62-459c-89d2-47b2cfa63906\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-vcffz"
Sep 16 19:16:21.217728 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:21.217708    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/082e4754-ea62-459c-89d2-47b2cfa63906-dshm\") pod \"tri-v2-kserve-dcb554bf5-vcffz\" (UID: \"082e4754-ea62-459c-89d2-47b2cfa63906\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-vcffz"
Sep 16 19:16:21.218062 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:21.218047    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/082e4754-ea62-459c-89d2-47b2cfa63906-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-vcffz\" (UID: \"082e4754-ea62-459c-89d2-47b2cfa63906\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-vcffz"
Sep 16 19:16:21.225814 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:21.225796    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c69j\" (UniqueName: \"kubernetes.io/projected/082e4754-ea62-459c-89d2-47b2cfa63906-kube-api-access-2c69j\") pod \"tri-v2-kserve-dcb554bf5-vcffz\" (UID: \"082e4754-ea62-459c-89d2-47b2cfa63906\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-vcffz"
Sep 16 19:16:21.344134 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:21.344079    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-vcffz"
Sep 16 19:16:21.622299 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:21.622228    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-vcffz"]
Sep 16 19:16:21.625081 ip-10-0-128-247 kubenswrapper[2577]: W0916 19:16:21.625055    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod082e4754_ea62_459c_89d2_47b2cfa63906.slice/crio-45b2f9eeb16bf8431f59a643c406fc44d270869751e9cc2c5792d1cfbc2366d0 WatchSource:0}: Error finding container 45b2f9eeb16bf8431f59a643c406fc44d270869751e9cc2c5792d1cfbc2366d0: Status 404 returned error can't find the container with id 45b2f9eeb16bf8431f59a643c406fc44d270869751e9cc2c5792d1cfbc2366d0
Sep 16 19:16:22.502073 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:22.502037    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-vcffz" event={"ID":"082e4754-ea62-459c-89d2-47b2cfa63906","Type":"ContainerStarted","Data":"53277cece43a841735f704c9fd775460fe36c877a3ff27ed1da53ad6e2cd73cd"}
Sep 16 19:16:22.502436 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:22.502086    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-vcffz" event={"ID":"082e4754-ea62-459c-89d2-47b2cfa63906","Type":"ContainerStarted","Data":"45b2f9eeb16bf8431f59a643c406fc44d270869751e9cc2c5792d1cfbc2366d0"}
Sep 16 19:16:22.522649 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:22.522603    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-vcffz" podStartSLOduration=1.5225881060000002 podStartE2EDuration="1.522588106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:16:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:16:22.521673553 +0000 UTC m=+2139.904562919" watchObservedRunningTime="2026-09-16 19:16:22.522588106 +0000 UTC m=+2139.905477515"
Sep 16 19:16:23.344452 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:23.344418    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-vcffz"
Sep 16 19:16:28.871469 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:28.871429    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-w67hb"]
Sep 16 19:16:28.875013 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:28.874992    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-w67hb"
Sep 16 19:16:28.878766 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:28.878746    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v3-kserve-self-signed-certs\""
Sep 16 19:16:28.898287 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:28.898264    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-w67hb"]
Sep 16 19:16:28.979842 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:28.979816    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33216803-17d0-45f5-9f05-621d649b92be-dshm\") pod \"tri-v3-kserve-798c88c66c-w67hb\" (UID: \"33216803-17d0-45f5-9f05-621d649b92be\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-w67hb"
Sep 16 19:16:28.979966 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:28.979842    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33216803-17d0-45f5-9f05-621d649b92be-tls-certs\") pod \"tri-v3-kserve-798c88c66c-w67hb\" (UID: \"33216803-17d0-45f5-9f05-621d649b92be\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-w67hb"
Sep 16 19:16:28.979966 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:28.979890    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33216803-17d0-45f5-9f05-621d649b92be-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-w67hb\" (UID: \"33216803-17d0-45f5-9f05-621d649b92be\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-w67hb"
Sep 16 19:16:28.980048 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:28.979961    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33216803-17d0-45f5-9f05-621d649b92be-home\") pod \"tri-v3-kserve-798c88c66c-w67hb\" (UID: \"33216803-17d0-45f5-9f05-621d649b92be\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-w67hb"
Sep 16 19:16:28.980048 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:28.980011    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkffl\" (UniqueName: \"kubernetes.io/projected/33216803-17d0-45f5-9f05-621d649b92be-kube-api-access-vkffl\") pod \"tri-v3-kserve-798c88c66c-w67hb\" (UID: \"33216803-17d0-45f5-9f05-621d649b92be\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-w67hb"
Sep 16 19:16:28.980048 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:28.980045    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33216803-17d0-45f5-9f05-621d649b92be-model-cache\") pod \"tri-v3-kserve-798c88c66c-w67hb\" (UID: \"33216803-17d0-45f5-9f05-621d649b92be\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-w67hb"
Sep 16 19:16:29.081199 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:29.081171    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33216803-17d0-45f5-9f05-621d649b92be-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-w67hb\" (UID: \"33216803-17d0-45f5-9f05-621d649b92be\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-w67hb"
Sep 16 19:16:29.081287 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:29.081208    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33216803-17d0-45f5-9f05-621d649b92be-home\") pod \"tri-v3-kserve-798c88c66c-w67hb\" (UID: \"33216803-17d0-45f5-9f05-621d649b92be\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-w67hb"
Sep 16 19:16:29.081287 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:29.081237    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkffl\" (UniqueName: \"kubernetes.io/projected/33216803-17d0-45f5-9f05-621d649b92be-kube-api-access-vkffl\") pod \"tri-v3-kserve-798c88c66c-w67hb\" (UID: \"33216803-17d0-45f5-9f05-621d649b92be\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-w67hb"
Sep 16 19:16:29.081287 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:29.081261    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33216803-17d0-45f5-9f05-621d649b92be-model-cache\") pod \"tri-v3-kserve-798c88c66c-w67hb\" (UID: \"33216803-17d0-45f5-9f05-621d649b92be\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-w67hb"
Sep 16 19:16:29.081405 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:29.081288    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33216803-17d0-45f5-9f05-621d649b92be-dshm\") pod \"tri-v3-kserve-798c88c66c-w67hb\" (UID: \"33216803-17d0-45f5-9f05-621d649b92be\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-w67hb"
Sep 16 19:16:29.081405 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:29.081303    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33216803-17d0-45f5-9f05-621d649b92be-tls-certs\") pod \"tri-v3-kserve-798c88c66c-w67hb\" (UID: \"33216803-17d0-45f5-9f05-621d649b92be\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-w67hb"
Sep 16 19:16:29.081664 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:29.081642    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33216803-17d0-45f5-9f05-621d649b92be-home\") pod \"tri-v3-kserve-798c88c66c-w67hb\" (UID: \"33216803-17d0-45f5-9f05-621d649b92be\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-w67hb"
Sep 16 19:16:29.081745 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:29.081659    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33216803-17d0-45f5-9f05-621d649b92be-model-cache\") pod \"tri-v3-kserve-798c88c66c-w67hb\" (UID: \"33216803-17d0-45f5-9f05-621d649b92be\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-w67hb"
Sep 16 19:16:29.081745 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:29.081699    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33216803-17d0-45f5-9f05-621d649b92be-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-w67hb\" (UID: \"33216803-17d0-45f5-9f05-621d649b92be\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-w67hb"
Sep 16 19:16:29.090009 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:29.084131    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33216803-17d0-45f5-9f05-621d649b92be-dshm\") pod \"tri-v3-kserve-798c88c66c-w67hb\" (UID: \"33216803-17d0-45f5-9f05-621d649b92be\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-w67hb"
Sep 16 19:16:29.090009 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:29.084417    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33216803-17d0-45f5-9f05-621d649b92be-tls-certs\") pod \"tri-v3-kserve-798c88c66c-w67hb\" (UID: \"33216803-17d0-45f5-9f05-621d649b92be\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-w67hb"
Sep 16 19:16:29.093216 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:29.093195    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkffl\" (UniqueName: \"kubernetes.io/projected/33216803-17d0-45f5-9f05-621d649b92be-kube-api-access-vkffl\") pod \"tri-v3-kserve-798c88c66c-w67hb\" (UID: \"33216803-17d0-45f5-9f05-621d649b92be\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-w67hb"
Sep 16 19:16:29.186475 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:29.186454    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-w67hb"
Sep 16 19:16:29.462547 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:29.462509    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-w67hb"]
Sep 16 19:16:29.464582 ip-10-0-128-247 kubenswrapper[2577]: W0916 19:16:29.464554    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33216803_17d0_45f5_9f05_621d649b92be.slice/crio-6cd2e42c36526ee736b4e6701de97ed08619e2283306e4535931cf0bf0c8a511 WatchSource:0}: Error finding container 6cd2e42c36526ee736b4e6701de97ed08619e2283306e4535931cf0bf0c8a511: Status 404 returned error can't find the container with id 6cd2e42c36526ee736b4e6701de97ed08619e2283306e4535931cf0bf0c8a511
Sep 16 19:16:29.535460 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:29.535432    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-w67hb" event={"ID":"33216803-17d0-45f5-9f05-621d649b92be","Type":"ContainerStarted","Data":"1433c718c45eda83dd4a38c189986bbe981c9a57746187bbd47b5b1a49a832a0"}
Sep 16 19:16:29.535594 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:29.535495    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-w67hb" event={"ID":"33216803-17d0-45f5-9f05-621d649b92be","Type":"ContainerStarted","Data":"6cd2e42c36526ee736b4e6701de97ed08619e2283306e4535931cf0bf0c8a511"}
Sep 16 19:16:29.554824 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:29.554780    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-w67hb" podStartSLOduration=1.554766305 podStartE2EDuration="1.554766305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:16:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:16:29.553364661 +0000 UTC m=+2146.936254041" watchObservedRunningTime="2026-09-16 19:16:29.554766305 +0000 UTC m=+2146.937655705"
Sep 16 19:16:30.186770 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:30.186737    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-w67hb"
Sep 16 19:16:31.344831 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:31.344801    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-vcffz"
Sep 16 19:16:31.357551 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:31.357507    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-vcffz"
Sep 16 19:16:31.557105 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:31.557072    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-vcffz"
Sep 16 19:16:39.187330 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:39.187256    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-w67hb"
Sep 16 19:16:39.199384 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:39.199362    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-w67hb"
Sep 16 19:16:39.591188 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:16:39.591118    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-w67hb"
Sep 16 19:18:48.163723 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:18:48.163683    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-58r8s"]
Sep 16 19:18:48.169402 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:18:48.169374    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-vcffz"]
Sep 16 19:18:48.180306 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:18:48.180268    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-w67hb"]
Sep 16 19:19:03.184731 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.184696    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-58r8s" podUID="34f8b194-16b7-465d-b3b2-5d369bc33b55" containerName="main" containerID="cri-o://b6c04ab12e2655df35c737cd9489a2a4c5dbe62236bd0a15c0888004cb8bc2e6" gracePeriod=45
Sep 16 19:19:03.189182 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.189142    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-vcffz" podUID="082e4754-ea62-459c-89d2-47b2cfa63906" containerName="main" containerID="cri-o://53277cece43a841735f704c9fd775460fe36c877a3ff27ed1da53ad6e2cd73cd" gracePeriod=45
Sep 16 19:19:03.199565 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.199517    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-w67hb" podUID="33216803-17d0-45f5-9f05-621d649b92be" containerName="main" containerID="cri-o://1433c718c45eda83dd4a38c189986bbe981c9a57746187bbd47b5b1a49a832a0" gracePeriod=45
Sep 16 19:19:03.467518 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.467493    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-58r8s"
Sep 16 19:19:03.476020 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.476000    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-vcffz"
Sep 16 19:19:03.488802 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.488783    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-w67hb"
Sep 16 19:19:03.552423 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.552379    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34f8b194-16b7-465d-b3b2-5d369bc33b55-tmp-dir\") pod \"34f8b194-16b7-465d-b3b2-5d369bc33b55\" (UID: \"34f8b194-16b7-465d-b3b2-5d369bc33b55\") "
Sep 16 19:19:03.552423 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.552421    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/082e4754-ea62-459c-89d2-47b2cfa63906-tmp-dir\") pod \"082e4754-ea62-459c-89d2-47b2cfa63906\" (UID: \"082e4754-ea62-459c-89d2-47b2cfa63906\") "
Sep 16 19:19:03.552423 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.552437    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33216803-17d0-45f5-9f05-621d649b92be-home\") pod \"33216803-17d0-45f5-9f05-621d649b92be\" (UID: \"33216803-17d0-45f5-9f05-621d649b92be\") "
Sep 16 19:19:03.552758 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.552471    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33216803-17d0-45f5-9f05-621d649b92be-model-cache\") pod \"33216803-17d0-45f5-9f05-621d649b92be\" (UID: \"33216803-17d0-45f5-9f05-621d649b92be\") "
Sep 16 19:19:03.552758 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.552504    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2c69j\" (UniqueName: \"kubernetes.io/projected/082e4754-ea62-459c-89d2-47b2cfa63906-kube-api-access-2c69j\") pod \"082e4754-ea62-459c-89d2-47b2cfa63906\" (UID: \"082e4754-ea62-459c-89d2-47b2cfa63906\") "
Sep 16 19:19:03.552758 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.552551    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33216803-17d0-45f5-9f05-621d649b92be-dshm\") pod \"33216803-17d0-45f5-9f05-621d649b92be\" (UID: \"33216803-17d0-45f5-9f05-621d649b92be\") "
Sep 16 19:19:03.552758 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.552580    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkffl\" (UniqueName: \"kubernetes.io/projected/33216803-17d0-45f5-9f05-621d649b92be-kube-api-access-vkffl\") pod \"33216803-17d0-45f5-9f05-621d649b92be\" (UID: \"33216803-17d0-45f5-9f05-621d649b92be\") "
Sep 16 19:19:03.552758 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.552610    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34f8b194-16b7-465d-b3b2-5d369bc33b55-tls-certs\") pod \"34f8b194-16b7-465d-b3b2-5d369bc33b55\" (UID: \"34f8b194-16b7-465d-b3b2-5d369bc33b55\") "
Sep 16 19:19:03.552758 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.552636    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34f8b194-16b7-465d-b3b2-5d369bc33b55-model-cache\") pod \"34f8b194-16b7-465d-b3b2-5d369bc33b55\" (UID: \"34f8b194-16b7-465d-b3b2-5d369bc33b55\") "
Sep 16 19:19:03.552758 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.552662    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33216803-17d0-45f5-9f05-621d649b92be-tls-certs\") pod \"33216803-17d0-45f5-9f05-621d649b92be\" (UID: \"33216803-17d0-45f5-9f05-621d649b92be\") "
Sep 16 19:19:03.552758 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.552674    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34f8b194-16b7-465d-b3b2-5d369bc33b55-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "34f8b194-16b7-465d-b3b2-5d369bc33b55" (UID: "34f8b194-16b7-465d-b3b2-5d369bc33b55"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:03.552758 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.552691    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/082e4754-ea62-459c-89d2-47b2cfa63906-model-cache\") pod \"082e4754-ea62-459c-89d2-47b2cfa63906\" (UID: \"082e4754-ea62-459c-89d2-47b2cfa63906\") "
Sep 16 19:19:03.552758 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.552700    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33216803-17d0-45f5-9f05-621d649b92be-home" (OuterVolumeSpecName: "home") pod "33216803-17d0-45f5-9f05-621d649b92be" (UID: "33216803-17d0-45f5-9f05-621d649b92be"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:03.552758 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.552715    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/082e4754-ea62-459c-89d2-47b2cfa63906-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "082e4754-ea62-459c-89d2-47b2cfa63906" (UID: "082e4754-ea62-459c-89d2-47b2cfa63906"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:03.552758 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.552723    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34f8b194-16b7-465d-b3b2-5d369bc33b55-dshm\") pod \"34f8b194-16b7-465d-b3b2-5d369bc33b55\" (UID: \"34f8b194-16b7-465d-b3b2-5d369bc33b55\") "
Sep 16 19:19:03.553415 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.552770    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34f8b194-16b7-465d-b3b2-5d369bc33b55-home\") pod \"34f8b194-16b7-465d-b3b2-5d369bc33b55\" (UID: \"34f8b194-16b7-465d-b3b2-5d369bc33b55\") "
Sep 16 19:19:03.553415 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.552794    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/082e4754-ea62-459c-89d2-47b2cfa63906-home\") pod \"082e4754-ea62-459c-89d2-47b2cfa63906\" (UID: \"082e4754-ea62-459c-89d2-47b2cfa63906\") "
Sep 16 19:19:03.553415 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.552844    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqn9g\" (UniqueName: \"kubernetes.io/projected/34f8b194-16b7-465d-b3b2-5d369bc33b55-kube-api-access-gqn9g\") pod \"34f8b194-16b7-465d-b3b2-5d369bc33b55\" (UID: \"34f8b194-16b7-465d-b3b2-5d369bc33b55\") "
Sep 16 19:19:03.553415 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.552856    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34f8b194-16b7-465d-b3b2-5d369bc33b55-model-cache" (OuterVolumeSpecName: "model-cache") pod "34f8b194-16b7-465d-b3b2-5d369bc33b55" (UID: "34f8b194-16b7-465d-b3b2-5d369bc33b55"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:03.553415 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.552913    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33216803-17d0-45f5-9f05-621d649b92be-tmp-dir\") pod \"33216803-17d0-45f5-9f05-621d649b92be\" (UID: \"33216803-17d0-45f5-9f05-621d649b92be\") "
Sep 16 19:19:03.553415 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.552944    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/082e4754-ea62-459c-89d2-47b2cfa63906-tls-certs\") pod \"082e4754-ea62-459c-89d2-47b2cfa63906\" (UID: \"082e4754-ea62-459c-89d2-47b2cfa63906\") "
Sep 16 19:19:03.553415 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.552969    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/082e4754-ea62-459c-89d2-47b2cfa63906-dshm\") pod \"082e4754-ea62-459c-89d2-47b2cfa63906\" (UID: \"082e4754-ea62-459c-89d2-47b2cfa63906\") "
Sep 16 19:19:03.553415 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.553281    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34f8b194-16b7-465d-b3b2-5d369bc33b55-tmp-dir\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:19:03.553415 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.553299    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/082e4754-ea62-459c-89d2-47b2cfa63906-tmp-dir\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:19:03.553415 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.553312    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33216803-17d0-45f5-9f05-621d649b92be-home\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:19:03.553415 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.553325    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34f8b194-16b7-465d-b3b2-5d369bc33b55-model-cache\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:19:03.554028 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.553492    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33216803-17d0-45f5-9f05-621d649b92be-model-cache" (OuterVolumeSpecName: "model-cache") pod "33216803-17d0-45f5-9f05-621d649b92be" (UID: "33216803-17d0-45f5-9f05-621d649b92be"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:03.554396 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.554092    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/082e4754-ea62-459c-89d2-47b2cfa63906-model-cache" (OuterVolumeSpecName: "model-cache") pod "082e4754-ea62-459c-89d2-47b2cfa63906" (UID: "082e4754-ea62-459c-89d2-47b2cfa63906"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:03.554396 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.554183    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33216803-17d0-45f5-9f05-621d649b92be-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "33216803-17d0-45f5-9f05-621d649b92be" (UID: "33216803-17d0-45f5-9f05-621d649b92be"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:03.554396 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.554330    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34f8b194-16b7-465d-b3b2-5d369bc33b55-home" (OuterVolumeSpecName: "home") pod "34f8b194-16b7-465d-b3b2-5d369bc33b55" (UID: "34f8b194-16b7-465d-b3b2-5d369bc33b55"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:03.554674 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.554402    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/082e4754-ea62-459c-89d2-47b2cfa63906-home" (OuterVolumeSpecName: "home") pod "082e4754-ea62-459c-89d2-47b2cfa63906" (UID: "082e4754-ea62-459c-89d2-47b2cfa63906"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:03.555934 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.555898    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33216803-17d0-45f5-9f05-621d649b92be-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "33216803-17d0-45f5-9f05-621d649b92be" (UID: "33216803-17d0-45f5-9f05-621d649b92be"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:19:03.556129 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.555983    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34f8b194-16b7-465d-b3b2-5d369bc33b55-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "34f8b194-16b7-465d-b3b2-5d369bc33b55" (UID: "34f8b194-16b7-465d-b3b2-5d369bc33b55"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:19:03.556129 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.556020    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/082e4754-ea62-459c-89d2-47b2cfa63906-dshm" (OuterVolumeSpecName: "dshm") pod "082e4754-ea62-459c-89d2-47b2cfa63906" (UID: "082e4754-ea62-459c-89d2-47b2cfa63906"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:03.556335 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.556290    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34f8b194-16b7-465d-b3b2-5d369bc33b55-kube-api-access-gqn9g" (OuterVolumeSpecName: "kube-api-access-gqn9g") pod "34f8b194-16b7-465d-b3b2-5d369bc33b55" (UID: "34f8b194-16b7-465d-b3b2-5d369bc33b55"). InnerVolumeSpecName "kube-api-access-gqn9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:19:03.556430 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.556391    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/082e4754-ea62-459c-89d2-47b2cfa63906-kube-api-access-2c69j" (OuterVolumeSpecName: "kube-api-access-2c69j") pod "082e4754-ea62-459c-89d2-47b2cfa63906" (UID: "082e4754-ea62-459c-89d2-47b2cfa63906"). InnerVolumeSpecName "kube-api-access-2c69j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:19:03.556960 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.556939    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33216803-17d0-45f5-9f05-621d649b92be-dshm" (OuterVolumeSpecName: "dshm") pod "33216803-17d0-45f5-9f05-621d649b92be" (UID: "33216803-17d0-45f5-9f05-621d649b92be"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:03.557227 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.557208    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/082e4754-ea62-459c-89d2-47b2cfa63906-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "082e4754-ea62-459c-89d2-47b2cfa63906" (UID: "082e4754-ea62-459c-89d2-47b2cfa63906"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:19:03.557408 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.557391    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33216803-17d0-45f5-9f05-621d649b92be-kube-api-access-vkffl" (OuterVolumeSpecName: "kube-api-access-vkffl") pod "33216803-17d0-45f5-9f05-621d649b92be" (UID: "33216803-17d0-45f5-9f05-621d649b92be"). InnerVolumeSpecName "kube-api-access-vkffl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:19:03.558198 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.558179    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34f8b194-16b7-465d-b3b2-5d369bc33b55-dshm" (OuterVolumeSpecName: "dshm") pod "34f8b194-16b7-465d-b3b2-5d369bc33b55" (UID: "34f8b194-16b7-465d-b3b2-5d369bc33b55"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:03.654315 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.654290    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33216803-17d0-45f5-9f05-621d649b92be-model-cache\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:19:03.654315 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.654311    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c69j\" (UniqueName: \"kubernetes.io/projected/082e4754-ea62-459c-89d2-47b2cfa63906-kube-api-access-2c69j\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:19:03.654445 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.654320    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33216803-17d0-45f5-9f05-621d649b92be-dshm\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:19:03.654445 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.654329    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkffl\" (UniqueName: \"kubernetes.io/projected/33216803-17d0-45f5-9f05-621d649b92be-kube-api-access-vkffl\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:19:03.654445 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.654338    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34f8b194-16b7-465d-b3b2-5d369bc33b55-tls-certs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:19:03.654445 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.654346    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33216803-17d0-45f5-9f05-621d649b92be-tls-certs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:19:03.654445 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.654353    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/082e4754-ea62-459c-89d2-47b2cfa63906-model-cache\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:19:03.654445 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.654360    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34f8b194-16b7-465d-b3b2-5d369bc33b55-dshm\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:19:03.654445 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.654367    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34f8b194-16b7-465d-b3b2-5d369bc33b55-home\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:19:03.654445 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.654375    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/082e4754-ea62-459c-89d2-47b2cfa63906-home\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:19:03.654445 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.654383    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqn9g\" (UniqueName: \"kubernetes.io/projected/34f8b194-16b7-465d-b3b2-5d369bc33b55-kube-api-access-gqn9g\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:19:03.654445 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.654390    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33216803-17d0-45f5-9f05-621d649b92be-tmp-dir\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:19:03.654445 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.654398    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/082e4754-ea62-459c-89d2-47b2cfa63906-tls-certs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:19:03.654445 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:03.654405    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/082e4754-ea62-459c-89d2-47b2cfa63906-dshm\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:19:04.240487 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:04.240456    2577 generic.go:356] "Generic (PLEG): container finished" podID="34f8b194-16b7-465d-b3b2-5d369bc33b55" containerID="b6c04ab12e2655df35c737cd9489a2a4c5dbe62236bd0a15c0888004cb8bc2e6" exitCode=0
Sep 16 19:19:04.240880 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:04.240511    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-58r8s"
Sep 16 19:19:04.240880 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:04.240548    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-58r8s" event={"ID":"34f8b194-16b7-465d-b3b2-5d369bc33b55","Type":"ContainerDied","Data":"b6c04ab12e2655df35c737cd9489a2a4c5dbe62236bd0a15c0888004cb8bc2e6"}
Sep 16 19:19:04.240880 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:04.240599    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-58r8s" event={"ID":"34f8b194-16b7-465d-b3b2-5d369bc33b55","Type":"ContainerDied","Data":"a49cb7dfb5fb43f74a3f53e1833ab5e2776b0a9cda9799ed281ab8d4a45e1577"}
Sep 16 19:19:04.240880 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:04.240622    2577 scope.go:117] "RemoveContainer" containerID="b6c04ab12e2655df35c737cd9489a2a4c5dbe62236bd0a15c0888004cb8bc2e6"
Sep 16 19:19:04.242031 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:04.242004    2577 generic.go:356] "Generic (PLEG): container finished" podID="082e4754-ea62-459c-89d2-47b2cfa63906" containerID="53277cece43a841735f704c9fd775460fe36c877a3ff27ed1da53ad6e2cd73cd" exitCode=0
Sep 16 19:19:04.242140 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:04.242038    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-vcffz" event={"ID":"082e4754-ea62-459c-89d2-47b2cfa63906","Type":"ContainerDied","Data":"53277cece43a841735f704c9fd775460fe36c877a3ff27ed1da53ad6e2cd73cd"}
Sep 16 19:19:04.242140 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:04.242075    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-vcffz"
Sep 16 19:19:04.242140 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:04.242093    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-vcffz" event={"ID":"082e4754-ea62-459c-89d2-47b2cfa63906","Type":"ContainerDied","Data":"45b2f9eeb16bf8431f59a643c406fc44d270869751e9cc2c5792d1cfbc2366d0"}
Sep 16 19:19:04.243677 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:04.243650    2577 generic.go:356] "Generic (PLEG): container finished" podID="33216803-17d0-45f5-9f05-621d649b92be" containerID="1433c718c45eda83dd4a38c189986bbe981c9a57746187bbd47b5b1a49a832a0" exitCode=0
Sep 16 19:19:04.243785 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:04.243714    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-w67hb"
Sep 16 19:19:04.243785 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:04.243757    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-w67hb" event={"ID":"33216803-17d0-45f5-9f05-621d649b92be","Type":"ContainerDied","Data":"1433c718c45eda83dd4a38c189986bbe981c9a57746187bbd47b5b1a49a832a0"}
Sep 16 19:19:04.243908 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:04.243794    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-w67hb" event={"ID":"33216803-17d0-45f5-9f05-621d649b92be","Type":"ContainerDied","Data":"6cd2e42c36526ee736b4e6701de97ed08619e2283306e4535931cf0bf0c8a511"}
Sep 16 19:19:04.250256 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:04.250239    2577 scope.go:117] "RemoveContainer" containerID="b6c04ab12e2655df35c737cd9489a2a4c5dbe62236bd0a15c0888004cb8bc2e6"
Sep 16 19:19:04.250601 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:19:04.250482    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6c04ab12e2655df35c737cd9489a2a4c5dbe62236bd0a15c0888004cb8bc2e6\": container with ID starting with b6c04ab12e2655df35c737cd9489a2a4c5dbe62236bd0a15c0888004cb8bc2e6 not found: ID does not exist" containerID="b6c04ab12e2655df35c737cd9489a2a4c5dbe62236bd0a15c0888004cb8bc2e6"
Sep 16 19:19:04.250601 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:04.250515    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6c04ab12e2655df35c737cd9489a2a4c5dbe62236bd0a15c0888004cb8bc2e6"} err="failed to get container status \"b6c04ab12e2655df35c737cd9489a2a4c5dbe62236bd0a15c0888004cb8bc2e6\": rpc error: code = NotFound desc = could not find container \"b6c04ab12e2655df35c737cd9489a2a4c5dbe62236bd0a15c0888004cb8bc2e6\": container with ID starting with b6c04ab12e2655df35c737cd9489a2a4c5dbe62236bd0a15c0888004cb8bc2e6 not found: ID does not exist"
Sep 16 19:19:04.250601 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:04.250554    2577 scope.go:117] "RemoveContainer" containerID="53277cece43a841735f704c9fd775460fe36c877a3ff27ed1da53ad6e2cd73cd"
Sep 16 19:19:04.259611 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:04.259583    2577 scope.go:117] "RemoveContainer" containerID="53277cece43a841735f704c9fd775460fe36c877a3ff27ed1da53ad6e2cd73cd"
Sep 16 19:19:04.259932 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:19:04.259914    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53277cece43a841735f704c9fd775460fe36c877a3ff27ed1da53ad6e2cd73cd\": container with ID starting with 53277cece43a841735f704c9fd775460fe36c877a3ff27ed1da53ad6e2cd73cd not found: ID does not exist" containerID="53277cece43a841735f704c9fd775460fe36c877a3ff27ed1da53ad6e2cd73cd"
Sep 16 19:19:04.259992 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:04.259938    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53277cece43a841735f704c9fd775460fe36c877a3ff27ed1da53ad6e2cd73cd"} err="failed to get container status \"53277cece43a841735f704c9fd775460fe36c877a3ff27ed1da53ad6e2cd73cd\": rpc error: code = NotFound desc = could not find container \"53277cece43a841735f704c9fd775460fe36c877a3ff27ed1da53ad6e2cd73cd\": container with ID starting with 53277cece43a841735f704c9fd775460fe36c877a3ff27ed1da53ad6e2cd73cd not found: ID does not exist"
Sep 16 19:19:04.259992 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:04.259953    2577 scope.go:117] "RemoveContainer" containerID="1433c718c45eda83dd4a38c189986bbe981c9a57746187bbd47b5b1a49a832a0"
Sep 16 19:19:04.267194 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:04.267176    2577 scope.go:117] "RemoveContainer" containerID="1433c718c45eda83dd4a38c189986bbe981c9a57746187bbd47b5b1a49a832a0"
Sep 16 19:19:04.267445 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:19:04.267429    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1433c718c45eda83dd4a38c189986bbe981c9a57746187bbd47b5b1a49a832a0\": container with ID starting with 1433c718c45eda83dd4a38c189986bbe981c9a57746187bbd47b5b1a49a832a0 not found: ID does not exist" containerID="1433c718c45eda83dd4a38c189986bbe981c9a57746187bbd47b5b1a49a832a0"
Sep 16 19:19:04.267495 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:04.267450    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1433c718c45eda83dd4a38c189986bbe981c9a57746187bbd47b5b1a49a832a0"} err="failed to get container status \"1433c718c45eda83dd4a38c189986bbe981c9a57746187bbd47b5b1a49a832a0\": rpc error: code = NotFound desc = could not find container \"1433c718c45eda83dd4a38c189986bbe981c9a57746187bbd47b5b1a49a832a0\": container with ID starting with 1433c718c45eda83dd4a38c189986bbe981c9a57746187bbd47b5b1a49a832a0 not found: ID does not exist"
Sep 16 19:19:04.272377 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:04.272356    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-58r8s"]
Sep 16 19:19:04.278900 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:04.278882    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-58r8s"]
Sep 16 19:19:04.292438 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:04.292415    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-vcffz"]
Sep 16 19:19:04.301358 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:04.301339    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-vcffz"]
Sep 16 19:19:04.321253 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:04.321227    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-w67hb"]
Sep 16 19:19:04.326909 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:04.326887    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-w67hb"]
Sep 16 19:19:05.156218 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:05.156183    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="082e4754-ea62-459c-89d2-47b2cfa63906" path="/var/lib/kubelet/pods/082e4754-ea62-459c-89d2-47b2cfa63906/volumes"
Sep 16 19:19:05.156600 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:05.156583    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33216803-17d0-45f5-9f05-621d649b92be" path="/var/lib/kubelet/pods/33216803-17d0-45f5-9f05-621d649b92be/volumes"
Sep 16 19:19:05.156948 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:05.156936    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34f8b194-16b7-465d-b3b2-5d369bc33b55" path="/var/lib/kubelet/pods/34f8b194-16b7-465d-b3b2-5d369bc33b55/volumes"
Sep 16 19:19:12.151231 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:12.151194    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-dxvk7"]
Sep 16 19:19:12.151591 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:12.151504    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33216803-17d0-45f5-9f05-621d649b92be" containerName="main"
Sep 16 19:19:12.151591 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:12.151514    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="33216803-17d0-45f5-9f05-621d649b92be" containerName="main"
Sep 16 19:19:12.151591 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:12.151551    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="082e4754-ea62-459c-89d2-47b2cfa63906" containerName="main"
Sep 16 19:19:12.151591 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:12.151558    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="082e4754-ea62-459c-89d2-47b2cfa63906" containerName="main"
Sep 16 19:19:12.151591 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:12.151570    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34f8b194-16b7-465d-b3b2-5d369bc33b55" containerName="main"
Sep 16 19:19:12.151591 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:12.151575    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="34f8b194-16b7-465d-b3b2-5d369bc33b55" containerName="main"
Sep 16 19:19:12.151801 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:12.151623    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="33216803-17d0-45f5-9f05-621d649b92be" containerName="main"
Sep 16 19:19:12.151801 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:12.151634    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="082e4754-ea62-459c-89d2-47b2cfa63906" containerName="main"
Sep 16 19:19:12.151801 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:12.151641    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="34f8b194-16b7-465d-b3b2-5d369bc33b55" containerName="main"
Sep 16 19:19:12.156508 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:12.156484    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-dxvk7"
Sep 16 19:19:12.160998 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:12.160980    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"kube-root-ca.crt\""
Sep 16 19:19:12.161109 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:12.160997    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"late-v1-kserve-self-signed-certs\""
Sep 16 19:19:12.161109 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:12.161007    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"openshift-service-ca.crt\""
Sep 16 19:19:12.161109 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:12.160997    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"default-dockercfg-rjgp5\""
Sep 16 19:19:12.164673 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:12.164652    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-dxvk7"]
Sep 16 19:19:12.220449 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:12.220426    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e27102e-9bdd-474d-8039-9865a63f7fa2-home\") pod \"late-v1-kserve-56cbc8456c-dxvk7\" (UID: \"1e27102e-9bdd-474d-8039-9865a63f7fa2\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-dxvk7"
Sep 16 19:19:12.220594 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:12.220464    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e27102e-9bdd-474d-8039-9865a63f7fa2-model-cache\") pod \"late-v1-kserve-56cbc8456c-dxvk7\" (UID: \"1e27102e-9bdd-474d-8039-9865a63f7fa2\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-dxvk7"
Sep 16 19:19:12.220594 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:12.220484    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e27102e-9bdd-474d-8039-9865a63f7fa2-dshm\") pod \"late-v1-kserve-56cbc8456c-dxvk7\" (UID: \"1e27102e-9bdd-474d-8039-9865a63f7fa2\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-dxvk7"
Sep 16 19:19:12.220723 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:12.220610    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e27102e-9bdd-474d-8039-9865a63f7fa2-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-dxvk7\" (UID: \"1e27102e-9bdd-474d-8039-9865a63f7fa2\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-dxvk7"
Sep 16 19:19:12.220723 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:12.220650    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-656z8\" (UniqueName: \"kubernetes.io/projected/1e27102e-9bdd-474d-8039-9865a63f7fa2-kube-api-access-656z8\") pod \"late-v1-kserve-56cbc8456c-dxvk7\" (UID: \"1e27102e-9bdd-474d-8039-9865a63f7fa2\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-dxvk7"
Sep 16 19:19:12.220723 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:12.220699    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e27102e-9bdd-474d-8039-9865a63f7fa2-tls-certs\") pod \"late-v1-kserve-56cbc8456c-dxvk7\" (UID: \"1e27102e-9bdd-474d-8039-9865a63f7fa2\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-dxvk7"
Sep 16 19:19:12.321271 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:12.321241    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e27102e-9bdd-474d-8039-9865a63f7fa2-model-cache\") pod \"late-v1-kserve-56cbc8456c-dxvk7\" (UID: \"1e27102e-9bdd-474d-8039-9865a63f7fa2\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-dxvk7"
Sep 16 19:19:12.321271 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:12.321271    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e27102e-9bdd-474d-8039-9865a63f7fa2-dshm\") pod \"late-v1-kserve-56cbc8456c-dxvk7\" (UID: \"1e27102e-9bdd-474d-8039-9865a63f7fa2\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-dxvk7"
Sep 16 19:19:12.321446 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:12.321310    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e27102e-9bdd-474d-8039-9865a63f7fa2-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-dxvk7\" (UID: \"1e27102e-9bdd-474d-8039-9865a63f7fa2\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-dxvk7"
Sep 16 19:19:12.321446 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:12.321337    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-656z8\" (UniqueName: \"kubernetes.io/projected/1e27102e-9bdd-474d-8039-9865a63f7fa2-kube-api-access-656z8\") pod \"late-v1-kserve-56cbc8456c-dxvk7\" (UID: \"1e27102e-9bdd-474d-8039-9865a63f7fa2\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-dxvk7"
Sep 16 19:19:12.321446 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:12.321366    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e27102e-9bdd-474d-8039-9865a63f7fa2-tls-certs\") pod \"late-v1-kserve-56cbc8456c-dxvk7\" (UID: \"1e27102e-9bdd-474d-8039-9865a63f7fa2\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-dxvk7"
Sep 16 19:19:12.321446 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:12.321416    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e27102e-9bdd-474d-8039-9865a63f7fa2-home\") pod \"late-v1-kserve-56cbc8456c-dxvk7\" (UID: \"1e27102e-9bdd-474d-8039-9865a63f7fa2\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-dxvk7"
Sep 16 19:19:12.321732 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:12.321713    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e27102e-9bdd-474d-8039-9865a63f7fa2-model-cache\") pod \"late-v1-kserve-56cbc8456c-dxvk7\" (UID: \"1e27102e-9bdd-474d-8039-9865a63f7fa2\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-dxvk7"
Sep 16 19:19:12.321797 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:12.321750    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e27102e-9bdd-474d-8039-9865a63f7fa2-home\") pod \"late-v1-kserve-56cbc8456c-dxvk7\" (UID: \"1e27102e-9bdd-474d-8039-9865a63f7fa2\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-dxvk7"
Sep 16 19:19:12.321846 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:12.321820    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e27102e-9bdd-474d-8039-9865a63f7fa2-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-dxvk7\" (UID: \"1e27102e-9bdd-474d-8039-9865a63f7fa2\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-dxvk7"
Sep 16 19:19:12.323801 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:12.323777    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e27102e-9bdd-474d-8039-9865a63f7fa2-dshm\") pod \"late-v1-kserve-56cbc8456c-dxvk7\" (UID: \"1e27102e-9bdd-474d-8039-9865a63f7fa2\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-dxvk7"
Sep 16 19:19:12.324074 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:12.324056    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e27102e-9bdd-474d-8039-9865a63f7fa2-tls-certs\") pod \"late-v1-kserve-56cbc8456c-dxvk7\" (UID: \"1e27102e-9bdd-474d-8039-9865a63f7fa2\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-dxvk7"
Sep 16 19:19:12.333422 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:12.333399    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-656z8\" (UniqueName: \"kubernetes.io/projected/1e27102e-9bdd-474d-8039-9865a63f7fa2-kube-api-access-656z8\") pod \"late-v1-kserve-56cbc8456c-dxvk7\" (UID: \"1e27102e-9bdd-474d-8039-9865a63f7fa2\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-dxvk7"
Sep 16 19:19:12.467741 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:12.467683    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-dxvk7"
Sep 16 19:19:12.743773 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:12.743748    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-dxvk7"]
Sep 16 19:19:12.745629 ip-10-0-128-247 kubenswrapper[2577]: W0916 19:19:12.745599    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e27102e_9bdd_474d_8039_9865a63f7fa2.slice/crio-9400dc92c377178147322bb89ba2a4d51ed7e3239b2af9ac2e2b704868f08726 WatchSource:0}: Error finding container 9400dc92c377178147322bb89ba2a4d51ed7e3239b2af9ac2e2b704868f08726: Status 404 returned error can't find the container with id 9400dc92c377178147322bb89ba2a4d51ed7e3239b2af9ac2e2b704868f08726
Sep 16 19:19:13.286908 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:13.286865    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-dxvk7" event={"ID":"1e27102e-9bdd-474d-8039-9865a63f7fa2","Type":"ContainerStarted","Data":"085c8e89569f99a69326aa9a522566e72bdfadb87e336a0dfc95e39cc0ae28e3"}
Sep 16 19:19:13.287360 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:13.286923    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-dxvk7" event={"ID":"1e27102e-9bdd-474d-8039-9865a63f7fa2","Type":"ContainerStarted","Data":"9400dc92c377178147322bb89ba2a4d51ed7e3239b2af9ac2e2b704868f08726"}
Sep 16 19:19:13.308294 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:13.308246    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-dxvk7" podStartSLOduration=1.308232112 podStartE2EDuration="1.308232112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:19:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:19:13.306761439 +0000 UTC m=+2310.689650803" watchObservedRunningTime="2026-09-16 19:19:13.308232112 +0000 UTC m=+2310.691121477"
Sep 16 19:19:13.468446 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:13.468418    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-dxvk7"
Sep 16 19:19:22.468783 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:22.468746    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-dxvk7"
Sep 16 19:19:22.482440 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:22.482418    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-dxvk7"
Sep 16 19:19:23.342962 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:23.342926    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-dxvk7"
Sep 16 19:19:37.451348 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:37.451277    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7"]
Sep 16 19:19:37.454840 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:37.454823    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7"
Sep 16 19:19:37.458852 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:37.458831    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 16 19:19:37.458993 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:37.458833    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 16 19:19:37.458993 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:37.458862    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 16 19:19:37.458993 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:37.458834    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-bjrrr\""
Sep 16 19:19:37.467716 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:37.467695    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7"]
Sep 16 19:19:37.605066 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:37.605037    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfkkd\" (UniqueName: \"kubernetes.io/projected/06d33010-86ad-4c67-a364-c5f265b03d72-kube-api-access-vfkkd\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7\" (UID: \"06d33010-86ad-4c67-a364-c5f265b03d72\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7"
Sep 16 19:19:37.605214 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:37.605069    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06d33010-86ad-4c67-a364-c5f265b03d72-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7\" (UID: \"06d33010-86ad-4c67-a364-c5f265b03d72\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7"
Sep 16 19:19:37.605214 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:37.605096    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06d33010-86ad-4c67-a364-c5f265b03d72-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7\" (UID: \"06d33010-86ad-4c67-a364-c5f265b03d72\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7"
Sep 16 19:19:37.605214 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:37.605123    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06d33010-86ad-4c67-a364-c5f265b03d72-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7\" (UID: \"06d33010-86ad-4c67-a364-c5f265b03d72\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7"
Sep 16 19:19:37.605214 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:37.605196    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06d33010-86ad-4c67-a364-c5f265b03d72-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7\" (UID: \"06d33010-86ad-4c67-a364-c5f265b03d72\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7"
Sep 16 19:19:37.605413 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:37.605272    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06d33010-86ad-4c67-a364-c5f265b03d72-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7\" (UID: \"06d33010-86ad-4c67-a364-c5f265b03d72\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7"
Sep 16 19:19:37.705971 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:37.705893    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06d33010-86ad-4c67-a364-c5f265b03d72-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7\" (UID: \"06d33010-86ad-4c67-a364-c5f265b03d72\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7"
Sep 16 19:19:37.705971 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:37.705962    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfkkd\" (UniqueName: \"kubernetes.io/projected/06d33010-86ad-4c67-a364-c5f265b03d72-kube-api-access-vfkkd\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7\" (UID: \"06d33010-86ad-4c67-a364-c5f265b03d72\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7"
Sep 16 19:19:37.706162 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:37.705997    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06d33010-86ad-4c67-a364-c5f265b03d72-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7\" (UID: \"06d33010-86ad-4c67-a364-c5f265b03d72\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7"
Sep 16 19:19:37.706162 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:37.706037    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06d33010-86ad-4c67-a364-c5f265b03d72-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7\" (UID: \"06d33010-86ad-4c67-a364-c5f265b03d72\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7"
Sep 16 19:19:37.706162 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:37.706067    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06d33010-86ad-4c67-a364-c5f265b03d72-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7\" (UID: \"06d33010-86ad-4c67-a364-c5f265b03d72\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7"
Sep 16 19:19:37.706162 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:37.706092    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06d33010-86ad-4c67-a364-c5f265b03d72-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7\" (UID: \"06d33010-86ad-4c67-a364-c5f265b03d72\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7"
Sep 16 19:19:37.706549 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:37.706504    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06d33010-86ad-4c67-a364-c5f265b03d72-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7\" (UID: \"06d33010-86ad-4c67-a364-c5f265b03d72\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7"
Sep 16 19:19:37.706670 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:37.706562    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06d33010-86ad-4c67-a364-c5f265b03d72-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7\" (UID: \"06d33010-86ad-4c67-a364-c5f265b03d72\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7"
Sep 16 19:19:37.706670 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:37.706596    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06d33010-86ad-4c67-a364-c5f265b03d72-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7\" (UID: \"06d33010-86ad-4c67-a364-c5f265b03d72\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7"
Sep 16 19:19:37.709073 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:37.709053    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06d33010-86ad-4c67-a364-c5f265b03d72-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7\" (UID: \"06d33010-86ad-4c67-a364-c5f265b03d72\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7"
Sep 16 19:19:37.709258 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:37.709242    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06d33010-86ad-4c67-a364-c5f265b03d72-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7\" (UID: \"06d33010-86ad-4c67-a364-c5f265b03d72\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7"
Sep 16 19:19:37.717890 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:37.717870    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfkkd\" (UniqueName: \"kubernetes.io/projected/06d33010-86ad-4c67-a364-c5f265b03d72-kube-api-access-vfkkd\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7\" (UID: \"06d33010-86ad-4c67-a364-c5f265b03d72\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7"
Sep 16 19:19:37.766761 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:37.766730    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7"
Sep 16 19:19:38.050049 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:38.050022    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7"]
Sep 16 19:19:38.051952 ip-10-0-128-247 kubenswrapper[2577]: W0916 19:19:38.051926    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06d33010_86ad_4c67_a364_c5f265b03d72.slice/crio-c02c17fab2d6e64b20b0c0589bd30e7a10a6bc5b7da71de0c9f98934b50dcb90 WatchSource:0}: Error finding container c02c17fab2d6e64b20b0c0589bd30e7a10a6bc5b7da71de0c9f98934b50dcb90: Status 404 returned error can't find the container with id c02c17fab2d6e64b20b0c0589bd30e7a10a6bc5b7da71de0c9f98934b50dcb90
Sep 16 19:19:38.397094 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:38.397058    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7" event={"ID":"06d33010-86ad-4c67-a364-c5f265b03d72","Type":"ContainerStarted","Data":"3f7fbff596a2e342c751794d6307d81335cc11b2469e5d21186cd06cc7466b39"}
Sep 16 19:19:38.397264 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:38.397111    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7" event={"ID":"06d33010-86ad-4c67-a364-c5f265b03d72","Type":"ContainerStarted","Data":"c02c17fab2d6e64b20b0c0589bd30e7a10a6bc5b7da71de0c9f98934b50dcb90"}
Sep 16 19:19:38.422105 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:38.422063    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7" podStartSLOduration=1.422049621 podStartE2EDuration="1.422049621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:19:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:19:38.419920596 +0000 UTC m=+2335.802809958" watchObservedRunningTime="2026-09-16 19:19:38.422049621 +0000 UTC m=+2335.804938988"
Sep 16 19:19:38.767635 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:38.767577    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7"
Sep 16 19:19:47.766849 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:47.766807    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7"
Sep 16 19:19:47.779449 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:47.779418    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7"
Sep 16 19:19:48.450159 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:19:48.450129    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7"
Sep 16 19:20:31.168780 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:20:31.168742    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7"]
Sep 16 19:20:31.177915 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:20:31.177878    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:20:31.177915 ip-10-0-128-247 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:20:31.177915 ip-10-0-128-247 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 19:20:31.177915 ip-10-0-128-247 kubenswrapper[2577]:  > containerID="3f7fbff596a2e342c751794d6307d81335cc11b2469e5d21186cd06cc7466b39" cmd=["/bin/sleep","15"]
Sep 16 19:20:31.178168 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:20:31.177931    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7" message="" err=<
Sep 16 19:20:31.178168 ip-10-0-128-247 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:20:31.178168 ip-10-0-128-247 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 19:20:31.178168 ip-10-0-128-247 kubenswrapper[2577]:  >
Sep 16 19:20:31.178168 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:20:31.177943    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:20:31.178168 ip-10-0-128-247 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:20:31.178168 ip-10-0-128-247 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 19:20:31.178168 ip-10-0-128-247 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7" podUID="06d33010-86ad-4c67-a364-c5f265b03d72" containerName="main" containerID="cri-o://3f7fbff596a2e342c751794d6307d81335cc11b2469e5d21186cd06cc7466b39"
Sep 16 19:20:31.178168 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:20:31.177990    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7" podUID="06d33010-86ad-4c67-a364-c5f265b03d72" containerName="main" containerID="cri-o://3f7fbff596a2e342c751794d6307d81335cc11b2469e5d21186cd06cc7466b39" gracePeriod=60
Sep 16 19:20:31.439966 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:20:31.439882    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7" podUID="06d33010-86ad-4c67-a364-c5f265b03d72" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Sep 16 19:20:31.524561 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:20:31.524524    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7"
Sep 16 19:20:31.618104 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:20:31.618076    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06d33010-86ad-4c67-a364-c5f265b03d72-tls-certs\") pod \"06d33010-86ad-4c67-a364-c5f265b03d72\" (UID: \"06d33010-86ad-4c67-a364-c5f265b03d72\") "
Sep 16 19:20:31.618259 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:20:31.618116    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06d33010-86ad-4c67-a364-c5f265b03d72-model-cache\") pod \"06d33010-86ad-4c67-a364-c5f265b03d72\" (UID: \"06d33010-86ad-4c67-a364-c5f265b03d72\") "
Sep 16 19:20:31.618259 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:20:31.618133    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06d33010-86ad-4c67-a364-c5f265b03d72-dshm\") pod \"06d33010-86ad-4c67-a364-c5f265b03d72\" (UID: \"06d33010-86ad-4c67-a364-c5f265b03d72\") "
Sep 16 19:20:31.618259 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:20:31.618153    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06d33010-86ad-4c67-a364-c5f265b03d72-home\") pod \"06d33010-86ad-4c67-a364-c5f265b03d72\" (UID: \"06d33010-86ad-4c67-a364-c5f265b03d72\") "
Sep 16 19:20:31.618437 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:20:31.618262    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfkkd\" (UniqueName: \"kubernetes.io/projected/06d33010-86ad-4c67-a364-c5f265b03d72-kube-api-access-vfkkd\") pod \"06d33010-86ad-4c67-a364-c5f265b03d72\" (UID: \"06d33010-86ad-4c67-a364-c5f265b03d72\") "
Sep 16 19:20:31.618437 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:20:31.618343    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06d33010-86ad-4c67-a364-c5f265b03d72-tmp-dir\") pod \"06d33010-86ad-4c67-a364-c5f265b03d72\" (UID: \"06d33010-86ad-4c67-a364-c5f265b03d72\") "
Sep 16 19:20:31.618437 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:20:31.618408    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d33010-86ad-4c67-a364-c5f265b03d72-model-cache" (OuterVolumeSpecName: "model-cache") pod "06d33010-86ad-4c67-a364-c5f265b03d72" (UID: "06d33010-86ad-4c67-a364-c5f265b03d72"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:20:31.618705 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:20:31.618446    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d33010-86ad-4c67-a364-c5f265b03d72-home" (OuterVolumeSpecName: "home") pod "06d33010-86ad-4c67-a364-c5f265b03d72" (UID: "06d33010-86ad-4c67-a364-c5f265b03d72"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:20:31.618705 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:20:31.618518    2577 generic.go:356] "Generic (PLEG): container finished" podID="06d33010-86ad-4c67-a364-c5f265b03d72" containerID="3f7fbff596a2e342c751794d6307d81335cc11b2469e5d21186cd06cc7466b39" exitCode=0
Sep 16 19:20:31.618705 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:20:31.618616    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7"
Sep 16 19:20:31.618705 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:20:31.618637    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d33010-86ad-4c67-a364-c5f265b03d72-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "06d33010-86ad-4c67-a364-c5f265b03d72" (UID: "06d33010-86ad-4c67-a364-c5f265b03d72"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:20:31.618705 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:20:31.618671    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7" event={"ID":"06d33010-86ad-4c67-a364-c5f265b03d72","Type":"ContainerDied","Data":"3f7fbff596a2e342c751794d6307d81335cc11b2469e5d21186cd06cc7466b39"}
Sep 16 19:20:31.618977 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:20:31.618718    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7" event={"ID":"06d33010-86ad-4c67-a364-c5f265b03d72","Type":"ContainerDied","Data":"c02c17fab2d6e64b20b0c0589bd30e7a10a6bc5b7da71de0c9f98934b50dcb90"}
Sep 16 19:20:31.618977 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:20:31.618737    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06d33010-86ad-4c67-a364-c5f265b03d72-model-cache\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:20:31.618977 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:20:31.618747    2577 scope.go:117] "RemoveContainer" containerID="3f7fbff596a2e342c751794d6307d81335cc11b2469e5d21186cd06cc7466b39"
Sep 16 19:20:31.618977 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:20:31.618752    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06d33010-86ad-4c67-a364-c5f265b03d72-home\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:20:31.618977 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:20:31.618765    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06d33010-86ad-4c67-a364-c5f265b03d72-tmp-dir\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:20:31.620838 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:20:31.620810    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d33010-86ad-4c67-a364-c5f265b03d72-dshm" (OuterVolumeSpecName: "dshm") pod "06d33010-86ad-4c67-a364-c5f265b03d72" (UID: "06d33010-86ad-4c67-a364-c5f265b03d72"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:20:31.620954 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:20:31.620870    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06d33010-86ad-4c67-a364-c5f265b03d72-kube-api-access-vfkkd" (OuterVolumeSpecName: "kube-api-access-vfkkd") pod "06d33010-86ad-4c67-a364-c5f265b03d72" (UID: "06d33010-86ad-4c67-a364-c5f265b03d72"). InnerVolumeSpecName "kube-api-access-vfkkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:20:31.620954 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:20:31.620878    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06d33010-86ad-4c67-a364-c5f265b03d72-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "06d33010-86ad-4c67-a364-c5f265b03d72" (UID: "06d33010-86ad-4c67-a364-c5f265b03d72"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:20:31.636183 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:20:31.636166    2577 scope.go:117] "RemoveContainer" containerID="3f7fbff596a2e342c751794d6307d81335cc11b2469e5d21186cd06cc7466b39"
Sep 16 19:20:31.636437 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:20:31.636419    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f7fbff596a2e342c751794d6307d81335cc11b2469e5d21186cd06cc7466b39\": container with ID starting with 3f7fbff596a2e342c751794d6307d81335cc11b2469e5d21186cd06cc7466b39 not found: ID does not exist" containerID="3f7fbff596a2e342c751794d6307d81335cc11b2469e5d21186cd06cc7466b39"
Sep 16 19:20:31.636482 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:20:31.636445    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f7fbff596a2e342c751794d6307d81335cc11b2469e5d21186cd06cc7466b39"} err="failed to get container status \"3f7fbff596a2e342c751794d6307d81335cc11b2469e5d21186cd06cc7466b39\": rpc error: code = NotFound desc = could not find container \"3f7fbff596a2e342c751794d6307d81335cc11b2469e5d21186cd06cc7466b39\": container with ID starting with 3f7fbff596a2e342c751794d6307d81335cc11b2469e5d21186cd06cc7466b39 not found: ID does not exist"
Sep 16 19:20:31.720122 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:20:31.720073    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06d33010-86ad-4c67-a364-c5f265b03d72-tls-certs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:20:31.720122 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:20:31.720093    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06d33010-86ad-4c67-a364-c5f265b03d72-dshm\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:20:31.720122 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:20:31.720101    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfkkd\" (UniqueName: \"kubernetes.io/projected/06d33010-86ad-4c67-a364-c5f265b03d72-kube-api-access-vfkkd\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:20:31.968655 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:20:31.968625    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7"]
Sep 16 19:20:31.986641 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:20:31.986578    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9957f2c7"]
Sep 16 19:20:33.154650 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:20:33.154614    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06d33010-86ad-4c67-a364-c5f265b03d72" path="/var/lib/kubelet/pods/06d33010-86ad-4c67-a364-c5f265b03d72/volumes"
Sep 16 19:21:10.715677 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:10.715606    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-dxvk7"]
Sep 16 19:21:25.734437 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:25.734399    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-dxvk7" podUID="1e27102e-9bdd-474d-8039-9865a63f7fa2" containerName="main" containerID="cri-o://085c8e89569f99a69326aa9a522566e72bdfadb87e336a0dfc95e39cc0ae28e3" gracePeriod=45
Sep 16 19:21:25.978689 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:25.978667    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-dxvk7"
Sep 16 19:21:26.124859 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:26.124787    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e27102e-9bdd-474d-8039-9865a63f7fa2-model-cache\") pod \"1e27102e-9bdd-474d-8039-9865a63f7fa2\" (UID: \"1e27102e-9bdd-474d-8039-9865a63f7fa2\") "
Sep 16 19:21:26.124859 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:26.124821    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e27102e-9bdd-474d-8039-9865a63f7fa2-tls-certs\") pod \"1e27102e-9bdd-474d-8039-9865a63f7fa2\" (UID: \"1e27102e-9bdd-474d-8039-9865a63f7fa2\") "
Sep 16 19:21:26.125057 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:26.124863    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e27102e-9bdd-474d-8039-9865a63f7fa2-dshm\") pod \"1e27102e-9bdd-474d-8039-9865a63f7fa2\" (UID: \"1e27102e-9bdd-474d-8039-9865a63f7fa2\") "
Sep 16 19:21:26.125057 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:26.124895    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e27102e-9bdd-474d-8039-9865a63f7fa2-home\") pod \"1e27102e-9bdd-474d-8039-9865a63f7fa2\" (UID: \"1e27102e-9bdd-474d-8039-9865a63f7fa2\") "
Sep 16 19:21:26.125057 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:26.124969    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e27102e-9bdd-474d-8039-9865a63f7fa2-tmp-dir\") pod \"1e27102e-9bdd-474d-8039-9865a63f7fa2\" (UID: \"1e27102e-9bdd-474d-8039-9865a63f7fa2\") "
Sep 16 19:21:26.125057 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:26.125013    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-656z8\" (UniqueName: \"kubernetes.io/projected/1e27102e-9bdd-474d-8039-9865a63f7fa2-kube-api-access-656z8\") pod \"1e27102e-9bdd-474d-8039-9865a63f7fa2\" (UID: \"1e27102e-9bdd-474d-8039-9865a63f7fa2\") "
Sep 16 19:21:26.125259 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:26.125072    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e27102e-9bdd-474d-8039-9865a63f7fa2-model-cache" (OuterVolumeSpecName: "model-cache") pod "1e27102e-9bdd-474d-8039-9865a63f7fa2" (UID: "1e27102e-9bdd-474d-8039-9865a63f7fa2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:26.125259 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:26.125250    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e27102e-9bdd-474d-8039-9865a63f7fa2-model-cache\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:21:26.125338 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:26.125253    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e27102e-9bdd-474d-8039-9865a63f7fa2-home" (OuterVolumeSpecName: "home") pod "1e27102e-9bdd-474d-8039-9865a63f7fa2" (UID: "1e27102e-9bdd-474d-8039-9865a63f7fa2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:26.125338 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:26.125244    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e27102e-9bdd-474d-8039-9865a63f7fa2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1e27102e-9bdd-474d-8039-9865a63f7fa2" (UID: "1e27102e-9bdd-474d-8039-9865a63f7fa2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:26.127211 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:26.127183    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e27102e-9bdd-474d-8039-9865a63f7fa2-dshm" (OuterVolumeSpecName: "dshm") pod "1e27102e-9bdd-474d-8039-9865a63f7fa2" (UID: "1e27102e-9bdd-474d-8039-9865a63f7fa2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:26.127323 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:26.127269    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e27102e-9bdd-474d-8039-9865a63f7fa2-kube-api-access-656z8" (OuterVolumeSpecName: "kube-api-access-656z8") pod "1e27102e-9bdd-474d-8039-9865a63f7fa2" (UID: "1e27102e-9bdd-474d-8039-9865a63f7fa2"). InnerVolumeSpecName "kube-api-access-656z8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:21:26.127323 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:26.127280    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e27102e-9bdd-474d-8039-9865a63f7fa2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1e27102e-9bdd-474d-8039-9865a63f7fa2" (UID: "1e27102e-9bdd-474d-8039-9865a63f7fa2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:21:26.225931 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:26.225904    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e27102e-9bdd-474d-8039-9865a63f7fa2-tmp-dir\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:21:26.225931 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:26.225929    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-656z8\" (UniqueName: \"kubernetes.io/projected/1e27102e-9bdd-474d-8039-9865a63f7fa2-kube-api-access-656z8\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:21:26.226094 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:26.225939    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e27102e-9bdd-474d-8039-9865a63f7fa2-tls-certs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:21:26.226094 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:26.225948    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e27102e-9bdd-474d-8039-9865a63f7fa2-dshm\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:21:26.226094 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:26.225957    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e27102e-9bdd-474d-8039-9865a63f7fa2-home\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:21:26.852461 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:26.852426    2577 generic.go:356] "Generic (PLEG): container finished" podID="1e27102e-9bdd-474d-8039-9865a63f7fa2" containerID="085c8e89569f99a69326aa9a522566e72bdfadb87e336a0dfc95e39cc0ae28e3" exitCode=0
Sep 16 19:21:26.852926 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:26.852469    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-dxvk7"
Sep 16 19:21:26.852926 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:26.852494    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-dxvk7" event={"ID":"1e27102e-9bdd-474d-8039-9865a63f7fa2","Type":"ContainerDied","Data":"085c8e89569f99a69326aa9a522566e72bdfadb87e336a0dfc95e39cc0ae28e3"}
Sep 16 19:21:26.852926 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:26.852563    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-dxvk7" event={"ID":"1e27102e-9bdd-474d-8039-9865a63f7fa2","Type":"ContainerDied","Data":"9400dc92c377178147322bb89ba2a4d51ed7e3239b2af9ac2e2b704868f08726"}
Sep 16 19:21:26.852926 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:26.852588    2577 scope.go:117] "RemoveContainer" containerID="085c8e89569f99a69326aa9a522566e72bdfadb87e336a0dfc95e39cc0ae28e3"
Sep 16 19:21:26.862058 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:26.862041    2577 scope.go:117] "RemoveContainer" containerID="085c8e89569f99a69326aa9a522566e72bdfadb87e336a0dfc95e39cc0ae28e3"
Sep 16 19:21:26.862356 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:21:26.862338    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"085c8e89569f99a69326aa9a522566e72bdfadb87e336a0dfc95e39cc0ae28e3\": container with ID starting with 085c8e89569f99a69326aa9a522566e72bdfadb87e336a0dfc95e39cc0ae28e3 not found: ID does not exist" containerID="085c8e89569f99a69326aa9a522566e72bdfadb87e336a0dfc95e39cc0ae28e3"
Sep 16 19:21:26.862414 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:26.862364    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"085c8e89569f99a69326aa9a522566e72bdfadb87e336a0dfc95e39cc0ae28e3"} err="failed to get container status \"085c8e89569f99a69326aa9a522566e72bdfadb87e336a0dfc95e39cc0ae28e3\": rpc error: code = NotFound desc = could not find container \"085c8e89569f99a69326aa9a522566e72bdfadb87e336a0dfc95e39cc0ae28e3\": container with ID starting with 085c8e89569f99a69326aa9a522566e72bdfadb87e336a0dfc95e39cc0ae28e3 not found: ID does not exist"
Sep 16 19:21:26.881752 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:26.881726    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-dxvk7"]
Sep 16 19:21:26.887968 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:26.887949    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-dxvk7"]
Sep 16 19:21:27.155063 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:27.155034    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e27102e-9bdd-474d-8039-9865a63f7fa2" path="/var/lib/kubelet/pods/1e27102e-9bdd-474d-8039-9865a63f7fa2/volumes"
Sep 16 19:21:29.556044 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:29.556015    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-768tb"]
Sep 16 19:21:29.556424 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:29.556337    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06d33010-86ad-4c67-a364-c5f265b03d72" containerName="main"
Sep 16 19:21:29.556424 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:29.556348    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="06d33010-86ad-4c67-a364-c5f265b03d72" containerName="main"
Sep 16 19:21:29.556424 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:29.556358    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e27102e-9bdd-474d-8039-9865a63f7fa2" containerName="main"
Sep 16 19:21:29.556424 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:29.556363    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e27102e-9bdd-474d-8039-9865a63f7fa2" containerName="main"
Sep 16 19:21:29.556424 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:29.556415    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="06d33010-86ad-4c67-a364-c5f265b03d72" containerName="main"
Sep 16 19:21:29.556424 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:29.556425    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e27102e-9bdd-474d-8039-9865a63f7fa2" containerName="main"
Sep 16 19:21:29.560845 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:29.560826    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-768tb"
Sep 16 19:21:29.565005 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:29.564988    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"kube-root-ca.crt\""
Sep 16 19:21:29.565091 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:29.564992    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"openshift-service-ca.crt\""
Sep 16 19:21:29.566435 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:29.566416    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"default-dockercfg-b8ppm\""
Sep 16 19:21:29.566653 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:29.566632    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"delw-v1-kserve-self-signed-certs\""
Sep 16 19:21:29.573802 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:29.573778    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-768tb"]
Sep 16 19:21:29.650755 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:29.650730    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e83081a-2152-4126-baa7-e4169c9b18a9-dshm\") pod \"delw-v1-kserve-6cf8cbb847-768tb\" (UID: \"3e83081a-2152-4126-baa7-e4169c9b18a9\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-768tb"
Sep 16 19:21:29.650872 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:29.650761    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mwcz\" (UniqueName: \"kubernetes.io/projected/3e83081a-2152-4126-baa7-e4169c9b18a9-kube-api-access-4mwcz\") pod \"delw-v1-kserve-6cf8cbb847-768tb\" (UID: \"3e83081a-2152-4126-baa7-e4169c9b18a9\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-768tb"
Sep 16 19:21:29.650872 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:29.650821    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e83081a-2152-4126-baa7-e4169c9b18a9-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-768tb\" (UID: \"3e83081a-2152-4126-baa7-e4169c9b18a9\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-768tb"
Sep 16 19:21:29.650872 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:29.650851    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e83081a-2152-4126-baa7-e4169c9b18a9-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-768tb\" (UID: \"3e83081a-2152-4126-baa7-e4169c9b18a9\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-768tb"
Sep 16 19:21:29.650872 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:29.650871    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e83081a-2152-4126-baa7-e4169c9b18a9-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-768tb\" (UID: \"3e83081a-2152-4126-baa7-e4169c9b18a9\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-768tb"
Sep 16 19:21:29.651035 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:29.650897    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e83081a-2152-4126-baa7-e4169c9b18a9-home\") pod \"delw-v1-kserve-6cf8cbb847-768tb\" (UID: \"3e83081a-2152-4126-baa7-e4169c9b18a9\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-768tb"
Sep 16 19:21:29.752109 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:29.752084    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e83081a-2152-4126-baa7-e4169c9b18a9-dshm\") pod \"delw-v1-kserve-6cf8cbb847-768tb\" (UID: \"3e83081a-2152-4126-baa7-e4169c9b18a9\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-768tb"
Sep 16 19:21:29.752196 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:29.752115    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mwcz\" (UniqueName: \"kubernetes.io/projected/3e83081a-2152-4126-baa7-e4169c9b18a9-kube-api-access-4mwcz\") pod \"delw-v1-kserve-6cf8cbb847-768tb\" (UID: \"3e83081a-2152-4126-baa7-e4169c9b18a9\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-768tb"
Sep 16 19:21:29.752196 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:29.752139    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e83081a-2152-4126-baa7-e4169c9b18a9-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-768tb\" (UID: \"3e83081a-2152-4126-baa7-e4169c9b18a9\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-768tb"
Sep 16 19:21:29.752196 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:29.752158    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e83081a-2152-4126-baa7-e4169c9b18a9-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-768tb\" (UID: \"3e83081a-2152-4126-baa7-e4169c9b18a9\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-768tb"
Sep 16 19:21:29.752196 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:29.752186    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e83081a-2152-4126-baa7-e4169c9b18a9-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-768tb\" (UID: \"3e83081a-2152-4126-baa7-e4169c9b18a9\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-768tb"
Sep 16 19:21:29.752413 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:29.752223    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e83081a-2152-4126-baa7-e4169c9b18a9-home\") pod \"delw-v1-kserve-6cf8cbb847-768tb\" (UID: \"3e83081a-2152-4126-baa7-e4169c9b18a9\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-768tb"
Sep 16 19:21:29.752624 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:29.752602    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e83081a-2152-4126-baa7-e4169c9b18a9-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-768tb\" (UID: \"3e83081a-2152-4126-baa7-e4169c9b18a9\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-768tb"
Sep 16 19:21:29.752695 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:29.752635    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e83081a-2152-4126-baa7-e4169c9b18a9-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-768tb\" (UID: \"3e83081a-2152-4126-baa7-e4169c9b18a9\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-768tb"
Sep 16 19:21:29.752695 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:29.752664    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e83081a-2152-4126-baa7-e4169c9b18a9-home\") pod \"delw-v1-kserve-6cf8cbb847-768tb\" (UID: \"3e83081a-2152-4126-baa7-e4169c9b18a9\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-768tb"
Sep 16 19:21:29.754558 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:29.754519    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e83081a-2152-4126-baa7-e4169c9b18a9-dshm\") pod \"delw-v1-kserve-6cf8cbb847-768tb\" (UID: \"3e83081a-2152-4126-baa7-e4169c9b18a9\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-768tb"
Sep 16 19:21:29.754915 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:29.754897    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e83081a-2152-4126-baa7-e4169c9b18a9-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-768tb\" (UID: \"3e83081a-2152-4126-baa7-e4169c9b18a9\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-768tb"
Sep 16 19:21:29.771893 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:29.771872    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mwcz\" (UniqueName: \"kubernetes.io/projected/3e83081a-2152-4126-baa7-e4169c9b18a9-kube-api-access-4mwcz\") pod \"delw-v1-kserve-6cf8cbb847-768tb\" (UID: \"3e83081a-2152-4126-baa7-e4169c9b18a9\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-768tb"
Sep 16 19:21:29.872143 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:29.872085    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-768tb"
Sep 16 19:21:30.153151 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:30.153126    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-768tb"]
Sep 16 19:21:30.155512 ip-10-0-128-247 kubenswrapper[2577]: W0916 19:21:30.155488    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e83081a_2152_4126_baa7_e4169c9b18a9.slice/crio-1670c28e0453e253af111349c6fd185b7a7e0c1ac258a790de89a578aff6072d WatchSource:0}: Error finding container 1670c28e0453e253af111349c6fd185b7a7e0c1ac258a790de89a578aff6072d: Status 404 returned error can't find the container with id 1670c28e0453e253af111349c6fd185b7a7e0c1ac258a790de89a578aff6072d
Sep 16 19:21:30.873005 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:30.872973    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-768tb" event={"ID":"3e83081a-2152-4126-baa7-e4169c9b18a9","Type":"ContainerStarted","Data":"dac863257d979e0b157de839d92fa9a11a95b748126d21d15e2fd0e64abea392"}
Sep 16 19:21:30.873457 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:30.873021    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-768tb" event={"ID":"3e83081a-2152-4126-baa7-e4169c9b18a9","Type":"ContainerStarted","Data":"1670c28e0453e253af111349c6fd185b7a7e0c1ac258a790de89a578aff6072d"}
Sep 16 19:21:30.893513 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:30.893462    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-768tb" podStartSLOduration=1.8934444030000002 podStartE2EDuration="1.893444403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:21:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:21:30.892646351 +0000 UTC m=+2448.275535716" watchObservedRunningTime="2026-09-16 19:21:30.893444403 +0000 UTC m=+2448.276333765"
Sep 16 19:21:31.872235 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:31.872194    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-768tb"
Sep 16 19:21:39.872997 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:39.872957    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-768tb"
Sep 16 19:21:39.885823 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:39.885797    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-768tb"
Sep 16 19:21:39.922469 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:21:39.922447    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-768tb"
Sep 16 19:22:28.188400 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:22:28.188362    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-768tb"]
Sep 16 19:22:43.208156 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:22:43.208070    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-768tb" podUID="3e83081a-2152-4126-baa7-e4169c9b18a9" containerName="main" containerID="cri-o://dac863257d979e0b157de839d92fa9a11a95b748126d21d15e2fd0e64abea392" gracePeriod=45
Sep 16 19:22:43.447066 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:22:43.447045    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-768tb"
Sep 16 19:22:43.593061 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:22:43.592988    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e83081a-2152-4126-baa7-e4169c9b18a9-dshm\") pod \"3e83081a-2152-4126-baa7-e4169c9b18a9\" (UID: \"3e83081a-2152-4126-baa7-e4169c9b18a9\") "
Sep 16 19:22:43.593061 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:22:43.593019    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e83081a-2152-4126-baa7-e4169c9b18a9-tls-certs\") pod \"3e83081a-2152-4126-baa7-e4169c9b18a9\" (UID: \"3e83081a-2152-4126-baa7-e4169c9b18a9\") "
Sep 16 19:22:43.593061 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:22:43.593051    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e83081a-2152-4126-baa7-e4169c9b18a9-home\") pod \"3e83081a-2152-4126-baa7-e4169c9b18a9\" (UID: \"3e83081a-2152-4126-baa7-e4169c9b18a9\") "
Sep 16 19:22:43.593329 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:22:43.593079    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mwcz\" (UniqueName: \"kubernetes.io/projected/3e83081a-2152-4126-baa7-e4169c9b18a9-kube-api-access-4mwcz\") pod \"3e83081a-2152-4126-baa7-e4169c9b18a9\" (UID: \"3e83081a-2152-4126-baa7-e4169c9b18a9\") "
Sep 16 19:22:43.593329 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:22:43.593116    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e83081a-2152-4126-baa7-e4169c9b18a9-model-cache\") pod \"3e83081a-2152-4126-baa7-e4169c9b18a9\" (UID: \"3e83081a-2152-4126-baa7-e4169c9b18a9\") "
Sep 16 19:22:43.593329 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:22:43.593163    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e83081a-2152-4126-baa7-e4169c9b18a9-tmp-dir\") pod \"3e83081a-2152-4126-baa7-e4169c9b18a9\" (UID: \"3e83081a-2152-4126-baa7-e4169c9b18a9\") "
Sep 16 19:22:43.593560 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:22:43.593360    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e83081a-2152-4126-baa7-e4169c9b18a9-home" (OuterVolumeSpecName: "home") pod "3e83081a-2152-4126-baa7-e4169c9b18a9" (UID: "3e83081a-2152-4126-baa7-e4169c9b18a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:43.593560 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:22:43.593397    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e83081a-2152-4126-baa7-e4169c9b18a9-model-cache" (OuterVolumeSpecName: "model-cache") pod "3e83081a-2152-4126-baa7-e4169c9b18a9" (UID: "3e83081a-2152-4126-baa7-e4169c9b18a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:43.593677 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:22:43.593592    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e83081a-2152-4126-baa7-e4169c9b18a9-model-cache\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:22:43.593677 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:22:43.593615    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e83081a-2152-4126-baa7-e4169c9b18a9-home\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:22:43.593761 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:22:43.593683    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e83081a-2152-4126-baa7-e4169c9b18a9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3e83081a-2152-4126-baa7-e4169c9b18a9" (UID: "3e83081a-2152-4126-baa7-e4169c9b18a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:43.595640 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:22:43.595616    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e83081a-2152-4126-baa7-e4169c9b18a9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3e83081a-2152-4126-baa7-e4169c9b18a9" (UID: "3e83081a-2152-4126-baa7-e4169c9b18a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:22:43.595928 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:22:43.595902    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e83081a-2152-4126-baa7-e4169c9b18a9-dshm" (OuterVolumeSpecName: "dshm") pod "3e83081a-2152-4126-baa7-e4169c9b18a9" (UID: "3e83081a-2152-4126-baa7-e4169c9b18a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:43.595928 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:22:43.595918    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e83081a-2152-4126-baa7-e4169c9b18a9-kube-api-access-4mwcz" (OuterVolumeSpecName: "kube-api-access-4mwcz") pod "3e83081a-2152-4126-baa7-e4169c9b18a9" (UID: "3e83081a-2152-4126-baa7-e4169c9b18a9"). InnerVolumeSpecName "kube-api-access-4mwcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:22:43.694852 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:22:43.694826    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e83081a-2152-4126-baa7-e4169c9b18a9-dshm\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:22:43.694852 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:22:43.694848    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e83081a-2152-4126-baa7-e4169c9b18a9-tls-certs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:22:43.695000 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:22:43.694859    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mwcz\" (UniqueName: \"kubernetes.io/projected/3e83081a-2152-4126-baa7-e4169c9b18a9-kube-api-access-4mwcz\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:22:43.695000 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:22:43.694867    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e83081a-2152-4126-baa7-e4169c9b18a9-tmp-dir\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:22:44.174036 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:22:44.174002    2577 generic.go:356] "Generic (PLEG): container finished" podID="3e83081a-2152-4126-baa7-e4169c9b18a9" containerID="dac863257d979e0b157de839d92fa9a11a95b748126d21d15e2fd0e64abea392" exitCode=0
Sep 16 19:22:44.174201 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:22:44.174034    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-768tb" event={"ID":"3e83081a-2152-4126-baa7-e4169c9b18a9","Type":"ContainerDied","Data":"dac863257d979e0b157de839d92fa9a11a95b748126d21d15e2fd0e64abea392"}
Sep 16 19:22:44.174201 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:22:44.174089    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-768tb" event={"ID":"3e83081a-2152-4126-baa7-e4169c9b18a9","Type":"ContainerDied","Data":"1670c28e0453e253af111349c6fd185b7a7e0c1ac258a790de89a578aff6072d"}
Sep 16 19:22:44.174201 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:22:44.174112    2577 scope.go:117] "RemoveContainer" containerID="dac863257d979e0b157de839d92fa9a11a95b748126d21d15e2fd0e64abea392"
Sep 16 19:22:44.174201 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:22:44.174048    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-768tb"
Sep 16 19:22:44.182947 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:22:44.182931    2577 scope.go:117] "RemoveContainer" containerID="dac863257d979e0b157de839d92fa9a11a95b748126d21d15e2fd0e64abea392"
Sep 16 19:22:44.183235 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:22:44.183218    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dac863257d979e0b157de839d92fa9a11a95b748126d21d15e2fd0e64abea392\": container with ID starting with dac863257d979e0b157de839d92fa9a11a95b748126d21d15e2fd0e64abea392 not found: ID does not exist" containerID="dac863257d979e0b157de839d92fa9a11a95b748126d21d15e2fd0e64abea392"
Sep 16 19:22:44.183290 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:22:44.183241    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dac863257d979e0b157de839d92fa9a11a95b748126d21d15e2fd0e64abea392"} err="failed to get container status \"dac863257d979e0b157de839d92fa9a11a95b748126d21d15e2fd0e64abea392\": rpc error: code = NotFound desc = could not find container \"dac863257d979e0b157de839d92fa9a11a95b748126d21d15e2fd0e64abea392\": container with ID starting with dac863257d979e0b157de839d92fa9a11a95b748126d21d15e2fd0e64abea392 not found: ID does not exist"
Sep 16 19:22:44.199498 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:22:44.199473    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-768tb"]
Sep 16 19:22:44.206380 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:22:44.206358    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-768tb"]
Sep 16 19:22:45.154551 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:22:45.154492    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e83081a-2152-4126-baa7-e4169c9b18a9" path="/var/lib/kubelet/pods/3e83081a-2152-4126-baa7-e4169c9b18a9/volumes"
Sep 16 19:23:11.859550 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:11.859483    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xtvj8"]
Sep 16 19:23:11.860158 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:11.860012    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e83081a-2152-4126-baa7-e4169c9b18a9" containerName="main"
Sep 16 19:23:11.860158 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:11.860034    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e83081a-2152-4126-baa7-e4169c9b18a9" containerName="main"
Sep 16 19:23:11.860158 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:11.860129    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e83081a-2152-4126-baa7-e4169c9b18a9" containerName="main"
Sep 16 19:23:11.863277 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:11.863255    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xtvj8"
Sep 16 19:23:11.866542 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:11.866510    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"kube-root-ca.crt\""
Sep 16 19:23:11.867969 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:11.867949    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"openshift-service-ca.crt\""
Sep 16 19:23:11.868040 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:11.867954    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"roll-v1-kserve-self-signed-certs\""
Sep 16 19:23:11.868040 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:11.868022    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"default-dockercfg-bxvgw\""
Sep 16 19:23:11.873666 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:11.873645    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xtvj8"]
Sep 16 19:23:11.896647 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:11.896626    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98ac6015-12cb-427b-8278-a895a139c5c5-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-xtvj8\" (UID: \"98ac6015-12cb-427b-8278-a895a139c5c5\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xtvj8"
Sep 16 19:23:11.896760 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:11.896665    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98ac6015-12cb-427b-8278-a895a139c5c5-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-xtvj8\" (UID: \"98ac6015-12cb-427b-8278-a895a139c5c5\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xtvj8"
Sep 16 19:23:11.896760 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:11.896700    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98ac6015-12cb-427b-8278-a895a139c5c5-home\") pod \"roll-v1-kserve-69cf8bcfc-xtvj8\" (UID: \"98ac6015-12cb-427b-8278-a895a139c5c5\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xtvj8"
Sep 16 19:23:11.896852 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:11.896786    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98ac6015-12cb-427b-8278-a895a139c5c5-dshm\") pod \"roll-v1-kserve-69cf8bcfc-xtvj8\" (UID: \"98ac6015-12cb-427b-8278-a895a139c5c5\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xtvj8"
Sep 16 19:23:11.896852 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:11.896813    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvcwx\" (UniqueName: \"kubernetes.io/projected/98ac6015-12cb-427b-8278-a895a139c5c5-kube-api-access-hvcwx\") pod \"roll-v1-kserve-69cf8bcfc-xtvj8\" (UID: \"98ac6015-12cb-427b-8278-a895a139c5c5\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xtvj8"
Sep 16 19:23:11.896852 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:11.896836    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98ac6015-12cb-427b-8278-a895a139c5c5-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-xtvj8\" (UID: \"98ac6015-12cb-427b-8278-a895a139c5c5\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xtvj8"
Sep 16 19:23:11.997474 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:11.997441    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98ac6015-12cb-427b-8278-a895a139c5c5-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-xtvj8\" (UID: \"98ac6015-12cb-427b-8278-a895a139c5c5\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xtvj8"
Sep 16 19:23:11.997668 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:11.997486    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98ac6015-12cb-427b-8278-a895a139c5c5-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-xtvj8\" (UID: \"98ac6015-12cb-427b-8278-a895a139c5c5\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xtvj8"
Sep 16 19:23:11.997668 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:11.997506    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98ac6015-12cb-427b-8278-a895a139c5c5-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-xtvj8\" (UID: \"98ac6015-12cb-427b-8278-a895a139c5c5\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xtvj8"
Sep 16 19:23:11.997668 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:11.997552    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98ac6015-12cb-427b-8278-a895a139c5c5-home\") pod \"roll-v1-kserve-69cf8bcfc-xtvj8\" (UID: \"98ac6015-12cb-427b-8278-a895a139c5c5\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xtvj8"
Sep 16 19:23:11.997668 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:11.997603    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98ac6015-12cb-427b-8278-a895a139c5c5-dshm\") pod \"roll-v1-kserve-69cf8bcfc-xtvj8\" (UID: \"98ac6015-12cb-427b-8278-a895a139c5c5\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xtvj8"
Sep 16 19:23:11.997668 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:11.997631    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvcwx\" (UniqueName: \"kubernetes.io/projected/98ac6015-12cb-427b-8278-a895a139c5c5-kube-api-access-hvcwx\") pod \"roll-v1-kserve-69cf8bcfc-xtvj8\" (UID: \"98ac6015-12cb-427b-8278-a895a139c5c5\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xtvj8"
Sep 16 19:23:11.997949 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:11.997893    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98ac6015-12cb-427b-8278-a895a139c5c5-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-xtvj8\" (UID: \"98ac6015-12cb-427b-8278-a895a139c5c5\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xtvj8"
Sep 16 19:23:11.998000 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:11.997949    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98ac6015-12cb-427b-8278-a895a139c5c5-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-xtvj8\" (UID: \"98ac6015-12cb-427b-8278-a895a139c5c5\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xtvj8"
Sep 16 19:23:11.998038 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:11.997995    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98ac6015-12cb-427b-8278-a895a139c5c5-home\") pod \"roll-v1-kserve-69cf8bcfc-xtvj8\" (UID: \"98ac6015-12cb-427b-8278-a895a139c5c5\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xtvj8"
Sep 16 19:23:12.000090 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:12.000070    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98ac6015-12cb-427b-8278-a895a139c5c5-dshm\") pod \"roll-v1-kserve-69cf8bcfc-xtvj8\" (UID: \"98ac6015-12cb-427b-8278-a895a139c5c5\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xtvj8"
Sep 16 19:23:12.000368 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:12.000349    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98ac6015-12cb-427b-8278-a895a139c5c5-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-xtvj8\" (UID: \"98ac6015-12cb-427b-8278-a895a139c5c5\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xtvj8"
Sep 16 19:23:12.007303 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:12.007284    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvcwx\" (UniqueName: \"kubernetes.io/projected/98ac6015-12cb-427b-8278-a895a139c5c5-kube-api-access-hvcwx\") pod \"roll-v1-kserve-69cf8bcfc-xtvj8\" (UID: \"98ac6015-12cb-427b-8278-a895a139c5c5\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xtvj8"
Sep 16 19:23:12.173615 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:12.173587    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xtvj8"
Sep 16 19:23:12.447688 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:12.447665    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xtvj8"]
Sep 16 19:23:12.450003 ip-10-0-128-247 kubenswrapper[2577]: W0916 19:23:12.449977    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98ac6015_12cb_427b_8278_a895a139c5c5.slice/crio-d6c30635c9ff04b7bfb6a2f2cb10b485fb022764f338645e2f09c957079ea000 WatchSource:0}: Error finding container d6c30635c9ff04b7bfb6a2f2cb10b485fb022764f338645e2f09c957079ea000: Status 404 returned error can't find the container with id d6c30635c9ff04b7bfb6a2f2cb10b485fb022764f338645e2f09c957079ea000
Sep 16 19:23:13.297890 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:13.297852    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xtvj8" event={"ID":"98ac6015-12cb-427b-8278-a895a139c5c5","Type":"ContainerStarted","Data":"73e9f0142f3316c6ca5f159226aa179a0755aa9041f36f13874dedd813843081"}
Sep 16 19:23:13.298274 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:13.297912    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xtvj8" event={"ID":"98ac6015-12cb-427b-8278-a895a139c5c5","Type":"ContainerStarted","Data":"d6c30635c9ff04b7bfb6a2f2cb10b485fb022764f338645e2f09c957079ea000"}
Sep 16 19:23:14.174046 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:14.174014    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xtvj8"
Sep 16 19:23:22.174162 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:22.174131    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xtvj8"
Sep 16 19:23:22.186316 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:22.186296    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xtvj8"
Sep 16 19:23:22.209044 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:22.209001    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xtvj8" podStartSLOduration=11.20898358 podStartE2EDuration="11.20898358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:23:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:23:13.321907883 +0000 UTC m=+2550.704797248" watchObservedRunningTime="2026-09-16 19:23:22.20898358 +0000 UTC m=+2559.591872946"
Sep 16 19:23:22.344563 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:22.344523    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xtvj8"
Sep 16 19:23:29.858236 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:29.858205    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-p8png"]
Sep 16 19:23:29.862049 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:29.862030    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-p8png"
Sep 16 19:23:29.866008 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:29.865985    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"roll-v2-kserve-self-signed-certs\""
Sep 16 19:23:29.874106 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:29.874081    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-p8png"]
Sep 16 19:23:29.925739 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:29.925715    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93d5d73f-4341-488c-a69d-22f1a5925172-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-p8png\" (UID: \"93d5d73f-4341-488c-a69d-22f1a5925172\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-p8png"
Sep 16 19:23:29.925832 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:29.925747    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n769w\" (UniqueName: \"kubernetes.io/projected/93d5d73f-4341-488c-a69d-22f1a5925172-kube-api-access-n769w\") pod \"roll-v2-kserve-d4dbf9d67-p8png\" (UID: \"93d5d73f-4341-488c-a69d-22f1a5925172\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-p8png"
Sep 16 19:23:29.925832 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:29.925775    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93d5d73f-4341-488c-a69d-22f1a5925172-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-p8png\" (UID: \"93d5d73f-4341-488c-a69d-22f1a5925172\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-p8png"
Sep 16 19:23:29.925832 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:29.925799    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93d5d73f-4341-488c-a69d-22f1a5925172-home\") pod \"roll-v2-kserve-d4dbf9d67-p8png\" (UID: \"93d5d73f-4341-488c-a69d-22f1a5925172\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-p8png"
Sep 16 19:23:29.925832 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:29.925813    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93d5d73f-4341-488c-a69d-22f1a5925172-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-p8png\" (UID: \"93d5d73f-4341-488c-a69d-22f1a5925172\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-p8png"
Sep 16 19:23:29.925971 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:29.925841    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93d5d73f-4341-488c-a69d-22f1a5925172-dshm\") pod \"roll-v2-kserve-d4dbf9d67-p8png\" (UID: \"93d5d73f-4341-488c-a69d-22f1a5925172\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-p8png"
Sep 16 19:23:30.026167 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:30.026140    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93d5d73f-4341-488c-a69d-22f1a5925172-dshm\") pod \"roll-v2-kserve-d4dbf9d67-p8png\" (UID: \"93d5d73f-4341-488c-a69d-22f1a5925172\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-p8png"
Sep 16 19:23:30.026253 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:30.026212    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93d5d73f-4341-488c-a69d-22f1a5925172-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-p8png\" (UID: \"93d5d73f-4341-488c-a69d-22f1a5925172\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-p8png"
Sep 16 19:23:30.026293 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:30.026247    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n769w\" (UniqueName: \"kubernetes.io/projected/93d5d73f-4341-488c-a69d-22f1a5925172-kube-api-access-n769w\") pod \"roll-v2-kserve-d4dbf9d67-p8png\" (UID: \"93d5d73f-4341-488c-a69d-22f1a5925172\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-p8png"
Sep 16 19:23:30.026293 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:30.026275    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93d5d73f-4341-488c-a69d-22f1a5925172-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-p8png\" (UID: \"93d5d73f-4341-488c-a69d-22f1a5925172\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-p8png"
Sep 16 19:23:30.026390 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:30.026300    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93d5d73f-4341-488c-a69d-22f1a5925172-home\") pod \"roll-v2-kserve-d4dbf9d67-p8png\" (UID: \"93d5d73f-4341-488c-a69d-22f1a5925172\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-p8png"
Sep 16 19:23:30.026479 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:30.026451    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93d5d73f-4341-488c-a69d-22f1a5925172-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-p8png\" (UID: \"93d5d73f-4341-488c-a69d-22f1a5925172\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-p8png"
Sep 16 19:23:30.026751 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:30.026723    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93d5d73f-4341-488c-a69d-22f1a5925172-home\") pod \"roll-v2-kserve-d4dbf9d67-p8png\" (UID: \"93d5d73f-4341-488c-a69d-22f1a5925172\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-p8png"
Sep 16 19:23:30.026751 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:30.026727    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93d5d73f-4341-488c-a69d-22f1a5925172-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-p8png\" (UID: \"93d5d73f-4341-488c-a69d-22f1a5925172\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-p8png"
Sep 16 19:23:30.026886 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:30.026782    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93d5d73f-4341-488c-a69d-22f1a5925172-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-p8png\" (UID: \"93d5d73f-4341-488c-a69d-22f1a5925172\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-p8png"
Sep 16 19:23:30.028751 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:30.028732    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93d5d73f-4341-488c-a69d-22f1a5925172-dshm\") pod \"roll-v2-kserve-d4dbf9d67-p8png\" (UID: \"93d5d73f-4341-488c-a69d-22f1a5925172\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-p8png"
Sep 16 19:23:30.028935 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:30.028918    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93d5d73f-4341-488c-a69d-22f1a5925172-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-p8png\" (UID: \"93d5d73f-4341-488c-a69d-22f1a5925172\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-p8png"
Sep 16 19:23:30.038041 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:30.038015    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n769w\" (UniqueName: \"kubernetes.io/projected/93d5d73f-4341-488c-a69d-22f1a5925172-kube-api-access-n769w\") pod \"roll-v2-kserve-d4dbf9d67-p8png\" (UID: \"93d5d73f-4341-488c-a69d-22f1a5925172\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-p8png"
Sep 16 19:23:30.172657 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:30.172634    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-p8png"
Sep 16 19:23:30.449583 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:30.449484    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-p8png"]
Sep 16 19:23:30.452426 ip-10-0-128-247 kubenswrapper[2577]: W0916 19:23:30.452395    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93d5d73f_4341_488c_a69d_22f1a5925172.slice/crio-1cece7b2c1ff00bb222dcf3af00406595210d761dcde9d41506171535a102292 WatchSource:0}: Error finding container 1cece7b2c1ff00bb222dcf3af00406595210d761dcde9d41506171535a102292: Status 404 returned error can't find the container with id 1cece7b2c1ff00bb222dcf3af00406595210d761dcde9d41506171535a102292
Sep 16 19:23:31.375123 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:31.375092    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-p8png" event={"ID":"93d5d73f-4341-488c-a69d-22f1a5925172","Type":"ContainerStarted","Data":"8d346b1c11eaf6672a1df3c0f8cca5bbfafc21fb3d65110208d1deb1c1653e64"}
Sep 16 19:23:31.375476 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:31.375136    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-p8png" event={"ID":"93d5d73f-4341-488c-a69d-22f1a5925172","Type":"ContainerStarted","Data":"1cece7b2c1ff00bb222dcf3af00406595210d761dcde9d41506171535a102292"}
Sep 16 19:23:31.399721 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:31.399678    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-p8png" podStartSLOduration=2.399663678 podStartE2EDuration="2.399663678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:23:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:23:31.399420437 +0000 UTC m=+2568.782309817" watchObservedRunningTime="2026-09-16 19:23:31.399663678 +0000 UTC m=+2568.782553042"
Sep 16 19:23:32.173669 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:32.173636    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-p8png"
Sep 16 19:23:40.172925 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:40.172882    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-p8png"
Sep 16 19:23:40.185504 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:40.185471    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-p8png"
Sep 16 19:23:40.427516 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:23:40.427432    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-p8png"
Sep 16 19:24:08.056853 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:08.056770    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl"]
Sep 16 19:24:08.060219 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:08.060200    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl"
Sep 16 19:24:08.066880 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:08.066851    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-6t5rj\""
Sep 16 19:24:08.066880 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:08.066860    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 16 19:24:08.067045 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:08.066941    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 16 19:24:08.067045 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:08.066864    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 16 19:24:08.076116 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:08.076092    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl"]
Sep 16 19:24:08.216704 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:08.216678    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a48cf56f-fe91-4a81-b5cb-03c364ca705c-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl\" (UID: \"a48cf56f-fe91-4a81-b5cb-03c364ca705c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl"
Sep 16 19:24:08.216823 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:08.216715    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skrj8\" (UniqueName: \"kubernetes.io/projected/a48cf56f-fe91-4a81-b5cb-03c364ca705c-kube-api-access-skrj8\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl\" (UID: \"a48cf56f-fe91-4a81-b5cb-03c364ca705c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl"
Sep 16 19:24:08.216823 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:08.216759    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a48cf56f-fe91-4a81-b5cb-03c364ca705c-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl\" (UID: \"a48cf56f-fe91-4a81-b5cb-03c364ca705c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl"
Sep 16 19:24:08.216823 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:08.216805    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a48cf56f-fe91-4a81-b5cb-03c364ca705c-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl\" (UID: \"a48cf56f-fe91-4a81-b5cb-03c364ca705c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl"
Sep 16 19:24:08.216934 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:08.216858    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a48cf56f-fe91-4a81-b5cb-03c364ca705c-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl\" (UID: \"a48cf56f-fe91-4a81-b5cb-03c364ca705c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl"
Sep 16 19:24:08.216934 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:08.216894    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a48cf56f-fe91-4a81-b5cb-03c364ca705c-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl\" (UID: \"a48cf56f-fe91-4a81-b5cb-03c364ca705c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl"
Sep 16 19:24:08.317419 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:08.317350    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a48cf56f-fe91-4a81-b5cb-03c364ca705c-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl\" (UID: \"a48cf56f-fe91-4a81-b5cb-03c364ca705c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl"
Sep 16 19:24:08.317419 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:08.317395    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skrj8\" (UniqueName: \"kubernetes.io/projected/a48cf56f-fe91-4a81-b5cb-03c364ca705c-kube-api-access-skrj8\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl\" (UID: \"a48cf56f-fe91-4a81-b5cb-03c364ca705c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl"
Sep 16 19:24:08.317634 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:08.317444    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a48cf56f-fe91-4a81-b5cb-03c364ca705c-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl\" (UID: \"a48cf56f-fe91-4a81-b5cb-03c364ca705c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl"
Sep 16 19:24:08.317634 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:08.317502    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a48cf56f-fe91-4a81-b5cb-03c364ca705c-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl\" (UID: \"a48cf56f-fe91-4a81-b5cb-03c364ca705c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl"
Sep 16 19:24:08.317634 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:08.317567    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a48cf56f-fe91-4a81-b5cb-03c364ca705c-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl\" (UID: \"a48cf56f-fe91-4a81-b5cb-03c364ca705c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl"
Sep 16 19:24:08.317634 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:08.317605    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a48cf56f-fe91-4a81-b5cb-03c364ca705c-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl\" (UID: \"a48cf56f-fe91-4a81-b5cb-03c364ca705c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl"
Sep 16 19:24:08.317843 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:08.317800    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a48cf56f-fe91-4a81-b5cb-03c364ca705c-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl\" (UID: \"a48cf56f-fe91-4a81-b5cb-03c364ca705c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl"
Sep 16 19:24:08.317929 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:08.317910    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a48cf56f-fe91-4a81-b5cb-03c364ca705c-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl\" (UID: \"a48cf56f-fe91-4a81-b5cb-03c364ca705c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl"
Sep 16 19:24:08.317992 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:08.317966    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a48cf56f-fe91-4a81-b5cb-03c364ca705c-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl\" (UID: \"a48cf56f-fe91-4a81-b5cb-03c364ca705c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl"
Sep 16 19:24:08.319949 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:08.319925    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a48cf56f-fe91-4a81-b5cb-03c364ca705c-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl\" (UID: \"a48cf56f-fe91-4a81-b5cb-03c364ca705c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl"
Sep 16 19:24:08.320164 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:08.320141    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a48cf56f-fe91-4a81-b5cb-03c364ca705c-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl\" (UID: \"a48cf56f-fe91-4a81-b5cb-03c364ca705c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl"
Sep 16 19:24:08.327710 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:08.327684    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skrj8\" (UniqueName: \"kubernetes.io/projected/a48cf56f-fe91-4a81-b5cb-03c364ca705c-kube-api-access-skrj8\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl\" (UID: \"a48cf56f-fe91-4a81-b5cb-03c364ca705c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl"
Sep 16 19:24:08.370878 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:08.370856    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl"
Sep 16 19:24:08.649866 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:08.649844    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl"]
Sep 16 19:24:08.652046 ip-10-0-128-247 kubenswrapper[2577]: W0916 19:24:08.652019    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda48cf56f_fe91_4a81_b5cb_03c364ca705c.slice/crio-9da031a2876789a8ad5215169e869161049aa80e73b4c1373947e1d4b2e037d3 WatchSource:0}: Error finding container 9da031a2876789a8ad5215169e869161049aa80e73b4c1373947e1d4b2e037d3: Status 404 returned error can't find the container with id 9da031a2876789a8ad5215169e869161049aa80e73b4c1373947e1d4b2e037d3
Sep 16 19:24:09.542135 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:09.542092    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl" event={"ID":"a48cf56f-fe91-4a81-b5cb-03c364ca705c","Type":"ContainerStarted","Data":"a81313ce32ba6df645c0e2621ef2b0055014dfe972b67b2085269bde7506c6bf"}
Sep 16 19:24:09.542498 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:09.542151    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl" event={"ID":"a48cf56f-fe91-4a81-b5cb-03c364ca705c","Type":"ContainerStarted","Data":"9da031a2876789a8ad5215169e869161049aa80e73b4c1373947e1d4b2e037d3"}
Sep 16 19:24:09.572671 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:09.572626    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl" podStartSLOduration=1.572610436 podStartE2EDuration="1.572610436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:24:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:24:09.570223812 +0000 UTC m=+2606.953113168" watchObservedRunningTime="2026-09-16 19:24:09.572610436 +0000 UTC m=+2606.955499857"
Sep 16 19:24:10.371128 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:10.371096    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl"
Sep 16 19:24:18.371960 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:18.371914    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl"
Sep 16 19:24:18.384636 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:18.384606    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl"
Sep 16 19:24:18.596605 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:18.596581    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl"
Sep 16 19:24:42.619315 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:42.619280    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl"]
Sep 16 19:24:42.630584 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:24:42.630552    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:24:42.630584 ip-10-0-128-247 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:24:42.630584 ip-10-0-128-247 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 19:24:42.630584 ip-10-0-128-247 kubenswrapper[2577]:  > containerID="a81313ce32ba6df645c0e2621ef2b0055014dfe972b67b2085269bde7506c6bf" cmd=["/bin/sleep","15"]
Sep 16 19:24:42.630843 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:42.630603    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl" message="" err=<
Sep 16 19:24:42.630843 ip-10-0-128-247 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:24:42.630843 ip-10-0-128-247 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 19:24:42.630843 ip-10-0-128-247 kubenswrapper[2577]:  >
Sep 16 19:24:42.630843 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:24:42.630614    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:24:42.630843 ip-10-0-128-247 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:24:42.630843 ip-10-0-128-247 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 19:24:42.630843 ip-10-0-128-247 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl" podUID="a48cf56f-fe91-4a81-b5cb-03c364ca705c" containerName="main" containerID="cri-o://a81313ce32ba6df645c0e2621ef2b0055014dfe972b67b2085269bde7506c6bf"
Sep 16 19:24:42.630843 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:42.630658    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl" podUID="a48cf56f-fe91-4a81-b5cb-03c364ca705c" containerName="main" containerID="cri-o://a81313ce32ba6df645c0e2621ef2b0055014dfe972b67b2085269bde7506c6bf" gracePeriod=60
Sep 16 19:24:42.978076 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:42.978054    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl"
Sep 16 19:24:43.084693 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:43.084665    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a48cf56f-fe91-4a81-b5cb-03c364ca705c-tls-certs\") pod \"a48cf56f-fe91-4a81-b5cb-03c364ca705c\" (UID: \"a48cf56f-fe91-4a81-b5cb-03c364ca705c\") "
Sep 16 19:24:43.084856 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:43.084712    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-skrj8\" (UniqueName: \"kubernetes.io/projected/a48cf56f-fe91-4a81-b5cb-03c364ca705c-kube-api-access-skrj8\") pod \"a48cf56f-fe91-4a81-b5cb-03c364ca705c\" (UID: \"a48cf56f-fe91-4a81-b5cb-03c364ca705c\") "
Sep 16 19:24:43.084856 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:43.084746    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a48cf56f-fe91-4a81-b5cb-03c364ca705c-home\") pod \"a48cf56f-fe91-4a81-b5cb-03c364ca705c\" (UID: \"a48cf56f-fe91-4a81-b5cb-03c364ca705c\") "
Sep 16 19:24:43.084856 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:43.084785    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a48cf56f-fe91-4a81-b5cb-03c364ca705c-tmp-dir\") pod \"a48cf56f-fe91-4a81-b5cb-03c364ca705c\" (UID: \"a48cf56f-fe91-4a81-b5cb-03c364ca705c\") "
Sep 16 19:24:43.084856 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:43.084804    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a48cf56f-fe91-4a81-b5cb-03c364ca705c-model-cache\") pod \"a48cf56f-fe91-4a81-b5cb-03c364ca705c\" (UID: \"a48cf56f-fe91-4a81-b5cb-03c364ca705c\") "
Sep 16 19:24:43.084856 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:43.084821    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a48cf56f-fe91-4a81-b5cb-03c364ca705c-dshm\") pod \"a48cf56f-fe91-4a81-b5cb-03c364ca705c\" (UID: \"a48cf56f-fe91-4a81-b5cb-03c364ca705c\") "
Sep 16 19:24:43.085127 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:43.085054    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a48cf56f-fe91-4a81-b5cb-03c364ca705c-home" (OuterVolumeSpecName: "home") pod "a48cf56f-fe91-4a81-b5cb-03c364ca705c" (UID: "a48cf56f-fe91-4a81-b5cb-03c364ca705c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:24:43.085127 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:43.085064    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a48cf56f-fe91-4a81-b5cb-03c364ca705c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a48cf56f-fe91-4a81-b5cb-03c364ca705c" (UID: "a48cf56f-fe91-4a81-b5cb-03c364ca705c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:24:43.085127 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:43.085085    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a48cf56f-fe91-4a81-b5cb-03c364ca705c-model-cache" (OuterVolumeSpecName: "model-cache") pod "a48cf56f-fe91-4a81-b5cb-03c364ca705c" (UID: "a48cf56f-fe91-4a81-b5cb-03c364ca705c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:24:43.087026 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:43.086993    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a48cf56f-fe91-4a81-b5cb-03c364ca705c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a48cf56f-fe91-4a81-b5cb-03c364ca705c" (UID: "a48cf56f-fe91-4a81-b5cb-03c364ca705c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:24:43.087121 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:43.087102    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a48cf56f-fe91-4a81-b5cb-03c364ca705c-kube-api-access-skrj8" (OuterVolumeSpecName: "kube-api-access-skrj8") pod "a48cf56f-fe91-4a81-b5cb-03c364ca705c" (UID: "a48cf56f-fe91-4a81-b5cb-03c364ca705c"). InnerVolumeSpecName "kube-api-access-skrj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:24:43.087164 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:43.087133    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a48cf56f-fe91-4a81-b5cb-03c364ca705c-dshm" (OuterVolumeSpecName: "dshm") pod "a48cf56f-fe91-4a81-b5cb-03c364ca705c" (UID: "a48cf56f-fe91-4a81-b5cb-03c364ca705c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:24:43.185670 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:43.185644    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a48cf56f-fe91-4a81-b5cb-03c364ca705c-tmp-dir\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:24:43.185670 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:43.185667    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a48cf56f-fe91-4a81-b5cb-03c364ca705c-model-cache\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:24:43.185803 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:43.185675    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a48cf56f-fe91-4a81-b5cb-03c364ca705c-dshm\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:24:43.185803 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:43.185682    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a48cf56f-fe91-4a81-b5cb-03c364ca705c-tls-certs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:24:43.185803 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:43.185690    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skrj8\" (UniqueName: \"kubernetes.io/projected/a48cf56f-fe91-4a81-b5cb-03c364ca705c-kube-api-access-skrj8\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:24:43.185803 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:43.185698    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a48cf56f-fe91-4a81-b5cb-03c364ca705c-home\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:24:43.695131 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:43.695092    2577 generic.go:356] "Generic (PLEG): container finished" podID="a48cf56f-fe91-4a81-b5cb-03c364ca705c" containerID="a81313ce32ba6df645c0e2621ef2b0055014dfe972b67b2085269bde7506c6bf" exitCode=0
Sep 16 19:24:43.695564 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:43.695157    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl"
Sep 16 19:24:43.695564 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:43.695174    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl" event={"ID":"a48cf56f-fe91-4a81-b5cb-03c364ca705c","Type":"ContainerDied","Data":"a81313ce32ba6df645c0e2621ef2b0055014dfe972b67b2085269bde7506c6bf"}
Sep 16 19:24:43.695564 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:43.695233    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl" event={"ID":"a48cf56f-fe91-4a81-b5cb-03c364ca705c","Type":"ContainerDied","Data":"9da031a2876789a8ad5215169e869161049aa80e73b4c1373947e1d4b2e037d3"}
Sep 16 19:24:43.695564 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:43.695261    2577 scope.go:117] "RemoveContainer" containerID="a81313ce32ba6df645c0e2621ef2b0055014dfe972b67b2085269bde7506c6bf"
Sep 16 19:24:43.704486 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:43.704469    2577 scope.go:117] "RemoveContainer" containerID="a81313ce32ba6df645c0e2621ef2b0055014dfe972b67b2085269bde7506c6bf"
Sep 16 19:24:43.704801 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:24:43.704780    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a81313ce32ba6df645c0e2621ef2b0055014dfe972b67b2085269bde7506c6bf\": container with ID starting with a81313ce32ba6df645c0e2621ef2b0055014dfe972b67b2085269bde7506c6bf not found: ID does not exist" containerID="a81313ce32ba6df645c0e2621ef2b0055014dfe972b67b2085269bde7506c6bf"
Sep 16 19:24:43.704882 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:43.704807    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a81313ce32ba6df645c0e2621ef2b0055014dfe972b67b2085269bde7506c6bf"} err="failed to get container status \"a81313ce32ba6df645c0e2621ef2b0055014dfe972b67b2085269bde7506c6bf\": rpc error: code = NotFound desc = could not find container \"a81313ce32ba6df645c0e2621ef2b0055014dfe972b67b2085269bde7506c6bf\": container with ID starting with a81313ce32ba6df645c0e2621ef2b0055014dfe972b67b2085269bde7506c6bf not found: ID does not exist"
Sep 16 19:24:43.715802 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:43.715775    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl"]
Sep 16 19:24:43.718705 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:43.718683    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-qv6pl"]
Sep 16 19:24:45.156115 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:24:45.156078    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a48cf56f-fe91-4a81-b5cb-03c364ca705c" path="/var/lib/kubelet/pods/a48cf56f-fe91-4a81-b5cb-03c364ca705c/volumes"
Sep 16 19:25:15.262579 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:15.262545    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-p8png"]
Sep 16 19:25:15.583335 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:15.583251    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xtvj8"]
Sep 16 19:25:30.281925 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.281881    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-p8png" podUID="93d5d73f-4341-488c-a69d-22f1a5925172" containerName="main" containerID="cri-o://8d346b1c11eaf6672a1df3c0f8cca5bbfafc21fb3d65110208d1deb1c1653e64" gracePeriod=45
Sep 16 19:25:30.416936 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.416898    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-p8png" podUID="93d5d73f-4341-488c-a69d-22f1a5925172" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 16 19:25:30.529805 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.529778    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-p8png"
Sep 16 19:25:30.602105 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.602036    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xtvj8" podUID="98ac6015-12cb-427b-8278-a895a139c5c5" containerName="main" containerID="cri-o://73e9f0142f3316c6ca5f159226aa179a0755aa9041f36f13874dedd813843081" gracePeriod=45
Sep 16 19:25:30.654645 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.654622    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93d5d73f-4341-488c-a69d-22f1a5925172-home\") pod \"93d5d73f-4341-488c-a69d-22f1a5925172\" (UID: \"93d5d73f-4341-488c-a69d-22f1a5925172\") "
Sep 16 19:25:30.654754 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.654652    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93d5d73f-4341-488c-a69d-22f1a5925172-tls-certs\") pod \"93d5d73f-4341-488c-a69d-22f1a5925172\" (UID: \"93d5d73f-4341-488c-a69d-22f1a5925172\") "
Sep 16 19:25:30.654754 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.654689    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93d5d73f-4341-488c-a69d-22f1a5925172-tmp-dir\") pod \"93d5d73f-4341-488c-a69d-22f1a5925172\" (UID: \"93d5d73f-4341-488c-a69d-22f1a5925172\") "
Sep 16 19:25:30.654754 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.654712    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93d5d73f-4341-488c-a69d-22f1a5925172-model-cache\") pod \"93d5d73f-4341-488c-a69d-22f1a5925172\" (UID: \"93d5d73f-4341-488c-a69d-22f1a5925172\") "
Sep 16 19:25:30.654754 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.654739    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n769w\" (UniqueName: \"kubernetes.io/projected/93d5d73f-4341-488c-a69d-22f1a5925172-kube-api-access-n769w\") pod \"93d5d73f-4341-488c-a69d-22f1a5925172\" (UID: \"93d5d73f-4341-488c-a69d-22f1a5925172\") "
Sep 16 19:25:30.654971 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.654794    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93d5d73f-4341-488c-a69d-22f1a5925172-dshm\") pod \"93d5d73f-4341-488c-a69d-22f1a5925172\" (UID: \"93d5d73f-4341-488c-a69d-22f1a5925172\") "
Sep 16 19:25:30.654971 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.654919    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93d5d73f-4341-488c-a69d-22f1a5925172-home" (OuterVolumeSpecName: "home") pod "93d5d73f-4341-488c-a69d-22f1a5925172" (UID: "93d5d73f-4341-488c-a69d-22f1a5925172"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:30.655066 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.654998    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93d5d73f-4341-488c-a69d-22f1a5925172-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "93d5d73f-4341-488c-a69d-22f1a5925172" (UID: "93d5d73f-4341-488c-a69d-22f1a5925172"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:30.655066 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.655010    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93d5d73f-4341-488c-a69d-22f1a5925172-model-cache" (OuterVolumeSpecName: "model-cache") pod "93d5d73f-4341-488c-a69d-22f1a5925172" (UID: "93d5d73f-4341-488c-a69d-22f1a5925172"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:30.655156 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.655137    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93d5d73f-4341-488c-a69d-22f1a5925172-tmp-dir\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:25:30.655214 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.655162    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93d5d73f-4341-488c-a69d-22f1a5925172-model-cache\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:25:30.655214 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.655174    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93d5d73f-4341-488c-a69d-22f1a5925172-home\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:25:30.657023 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.657001    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93d5d73f-4341-488c-a69d-22f1a5925172-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "93d5d73f-4341-488c-a69d-22f1a5925172" (UID: "93d5d73f-4341-488c-a69d-22f1a5925172"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:25:30.657098 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.657026    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93d5d73f-4341-488c-a69d-22f1a5925172-kube-api-access-n769w" (OuterVolumeSpecName: "kube-api-access-n769w") pod "93d5d73f-4341-488c-a69d-22f1a5925172" (UID: "93d5d73f-4341-488c-a69d-22f1a5925172"). InnerVolumeSpecName "kube-api-access-n769w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:25:30.657098 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.657052    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93d5d73f-4341-488c-a69d-22f1a5925172-dshm" (OuterVolumeSpecName: "dshm") pod "93d5d73f-4341-488c-a69d-22f1a5925172" (UID: "93d5d73f-4341-488c-a69d-22f1a5925172"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:30.756338 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.756312    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93d5d73f-4341-488c-a69d-22f1a5925172-tls-certs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:25:30.756459 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.756343    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n769w\" (UniqueName: \"kubernetes.io/projected/93d5d73f-4341-488c-a69d-22f1a5925172-kube-api-access-n769w\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:25:30.756459 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.756353    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93d5d73f-4341-488c-a69d-22f1a5925172-dshm\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:25:30.839050 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.839027    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xtvj8"
Sep 16 19:25:30.910742 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.910715    2577 generic.go:356] "Generic (PLEG): container finished" podID="98ac6015-12cb-427b-8278-a895a139c5c5" containerID="73e9f0142f3316c6ca5f159226aa179a0755aa9041f36f13874dedd813843081" exitCode=0
Sep 16 19:25:30.910887 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.910754    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xtvj8"
Sep 16 19:25:30.910887 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.910779    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xtvj8" event={"ID":"98ac6015-12cb-427b-8278-a895a139c5c5","Type":"ContainerDied","Data":"73e9f0142f3316c6ca5f159226aa179a0755aa9041f36f13874dedd813843081"}
Sep 16 19:25:30.910887 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.910824    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xtvj8" event={"ID":"98ac6015-12cb-427b-8278-a895a139c5c5","Type":"ContainerDied","Data":"d6c30635c9ff04b7bfb6a2f2cb10b485fb022764f338645e2f09c957079ea000"}
Sep 16 19:25:30.910887 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.910852    2577 scope.go:117] "RemoveContainer" containerID="73e9f0142f3316c6ca5f159226aa179a0755aa9041f36f13874dedd813843081"
Sep 16 19:25:30.912106 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.912089    2577 generic.go:356] "Generic (PLEG): container finished" podID="93d5d73f-4341-488c-a69d-22f1a5925172" containerID="8d346b1c11eaf6672a1df3c0f8cca5bbfafc21fb3d65110208d1deb1c1653e64" exitCode=0
Sep 16 19:25:30.912187 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.912116    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-p8png" event={"ID":"93d5d73f-4341-488c-a69d-22f1a5925172","Type":"ContainerDied","Data":"8d346b1c11eaf6672a1df3c0f8cca5bbfafc21fb3d65110208d1deb1c1653e64"}
Sep 16 19:25:30.912187 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.912136    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-p8png"
Sep 16 19:25:30.912187 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.912145    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-p8png" event={"ID":"93d5d73f-4341-488c-a69d-22f1a5925172","Type":"ContainerDied","Data":"1cece7b2c1ff00bb222dcf3af00406595210d761dcde9d41506171535a102292"}
Sep 16 19:25:30.919954 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.919926    2577 scope.go:117] "RemoveContainer" containerID="73e9f0142f3316c6ca5f159226aa179a0755aa9041f36f13874dedd813843081"
Sep 16 19:25:30.920298 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:25:30.920269    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73e9f0142f3316c6ca5f159226aa179a0755aa9041f36f13874dedd813843081\": container with ID starting with 73e9f0142f3316c6ca5f159226aa179a0755aa9041f36f13874dedd813843081 not found: ID does not exist" containerID="73e9f0142f3316c6ca5f159226aa179a0755aa9041f36f13874dedd813843081"
Sep 16 19:25:30.920402 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.920305    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73e9f0142f3316c6ca5f159226aa179a0755aa9041f36f13874dedd813843081"} err="failed to get container status \"73e9f0142f3316c6ca5f159226aa179a0755aa9041f36f13874dedd813843081\": rpc error: code = NotFound desc = could not find container \"73e9f0142f3316c6ca5f159226aa179a0755aa9041f36f13874dedd813843081\": container with ID starting with 73e9f0142f3316c6ca5f159226aa179a0755aa9041f36f13874dedd813843081 not found: ID does not exist"
Sep 16 19:25:30.920402 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.920323    2577 scope.go:117] "RemoveContainer" containerID="8d346b1c11eaf6672a1df3c0f8cca5bbfafc21fb3d65110208d1deb1c1653e64"
Sep 16 19:25:30.928085 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.928069    2577 scope.go:117] "RemoveContainer" containerID="8d346b1c11eaf6672a1df3c0f8cca5bbfafc21fb3d65110208d1deb1c1653e64"
Sep 16 19:25:30.928341 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:25:30.928324    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d346b1c11eaf6672a1df3c0f8cca5bbfafc21fb3d65110208d1deb1c1653e64\": container with ID starting with 8d346b1c11eaf6672a1df3c0f8cca5bbfafc21fb3d65110208d1deb1c1653e64 not found: ID does not exist" containerID="8d346b1c11eaf6672a1df3c0f8cca5bbfafc21fb3d65110208d1deb1c1653e64"
Sep 16 19:25:30.928408 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.928346    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d346b1c11eaf6672a1df3c0f8cca5bbfafc21fb3d65110208d1deb1c1653e64"} err="failed to get container status \"8d346b1c11eaf6672a1df3c0f8cca5bbfafc21fb3d65110208d1deb1c1653e64\": rpc error: code = NotFound desc = could not find container \"8d346b1c11eaf6672a1df3c0f8cca5bbfafc21fb3d65110208d1deb1c1653e64\": container with ID starting with 8d346b1c11eaf6672a1df3c0f8cca5bbfafc21fb3d65110208d1deb1c1653e64 not found: ID does not exist"
Sep 16 19:25:30.938860 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.938839    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-p8png"]
Sep 16 19:25:30.946241 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.946223    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-p8png"]
Sep 16 19:25:30.958861 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.958839    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98ac6015-12cb-427b-8278-a895a139c5c5-dshm\") pod \"98ac6015-12cb-427b-8278-a895a139c5c5\" (UID: \"98ac6015-12cb-427b-8278-a895a139c5c5\") "
Sep 16 19:25:30.958928 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.958870    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98ac6015-12cb-427b-8278-a895a139c5c5-home\") pod \"98ac6015-12cb-427b-8278-a895a139c5c5\" (UID: \"98ac6015-12cb-427b-8278-a895a139c5c5\") "
Sep 16 19:25:30.958928 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.958897    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98ac6015-12cb-427b-8278-a895a139c5c5-model-cache\") pod \"98ac6015-12cb-427b-8278-a895a139c5c5\" (UID: \"98ac6015-12cb-427b-8278-a895a139c5c5\") "
Sep 16 19:25:30.958928 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.958915    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98ac6015-12cb-427b-8278-a895a139c5c5-tls-certs\") pod \"98ac6015-12cb-427b-8278-a895a139c5c5\" (UID: \"98ac6015-12cb-427b-8278-a895a139c5c5\") "
Sep 16 19:25:30.959053 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.958961    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98ac6015-12cb-427b-8278-a895a139c5c5-tmp-dir\") pod \"98ac6015-12cb-427b-8278-a895a139c5c5\" (UID: \"98ac6015-12cb-427b-8278-a895a139c5c5\") "
Sep 16 19:25:30.959053 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.958980    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvcwx\" (UniqueName: \"kubernetes.io/projected/98ac6015-12cb-427b-8278-a895a139c5c5-kube-api-access-hvcwx\") pod \"98ac6015-12cb-427b-8278-a895a139c5c5\" (UID: \"98ac6015-12cb-427b-8278-a895a139c5c5\") "
Sep 16 19:25:30.959152 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.959126    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ac6015-12cb-427b-8278-a895a139c5c5-model-cache" (OuterVolumeSpecName: "model-cache") pod "98ac6015-12cb-427b-8278-a895a139c5c5" (UID: "98ac6015-12cb-427b-8278-a895a139c5c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:30.959204 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.959173    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ac6015-12cb-427b-8278-a895a139c5c5-home" (OuterVolumeSpecName: "home") pod "98ac6015-12cb-427b-8278-a895a139c5c5" (UID: "98ac6015-12cb-427b-8278-a895a139c5c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:30.959273 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.959204    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ac6015-12cb-427b-8278-a895a139c5c5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "98ac6015-12cb-427b-8278-a895a139c5c5" (UID: "98ac6015-12cb-427b-8278-a895a139c5c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:30.959340 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.959324    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98ac6015-12cb-427b-8278-a895a139c5c5-home\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:25:30.959392 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.959340    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98ac6015-12cb-427b-8278-a895a139c5c5-model-cache\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:25:30.959392 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.959349    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98ac6015-12cb-427b-8278-a895a139c5c5-tmp-dir\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:25:30.961076 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.961054    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98ac6015-12cb-427b-8278-a895a139c5c5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "98ac6015-12cb-427b-8278-a895a139c5c5" (UID: "98ac6015-12cb-427b-8278-a895a139c5c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:25:30.961160 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.961108    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ac6015-12cb-427b-8278-a895a139c5c5-dshm" (OuterVolumeSpecName: "dshm") pod "98ac6015-12cb-427b-8278-a895a139c5c5" (UID: "98ac6015-12cb-427b-8278-a895a139c5c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:30.961160 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:30.961121    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98ac6015-12cb-427b-8278-a895a139c5c5-kube-api-access-hvcwx" (OuterVolumeSpecName: "kube-api-access-hvcwx") pod "98ac6015-12cb-427b-8278-a895a139c5c5" (UID: "98ac6015-12cb-427b-8278-a895a139c5c5"). InnerVolumeSpecName "kube-api-access-hvcwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:25:31.060479 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:31.060458    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98ac6015-12cb-427b-8278-a895a139c5c5-dshm\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:25:31.060479 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:31.060479    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98ac6015-12cb-427b-8278-a895a139c5c5-tls-certs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:25:31.060618 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:31.060489    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvcwx\" (UniqueName: \"kubernetes.io/projected/98ac6015-12cb-427b-8278-a895a139c5c5-kube-api-access-hvcwx\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:25:31.158706 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:31.156915    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93d5d73f-4341-488c-a69d-22f1a5925172" path="/var/lib/kubelet/pods/93d5d73f-4341-488c-a69d-22f1a5925172/volumes"
Sep 16 19:25:31.228761 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:31.228730    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xtvj8"]
Sep 16 19:25:31.237758 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:31.237737    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xtvj8"]
Sep 16 19:25:33.154826 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:33.154788    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98ac6015-12cb-427b-8278-a895a139c5c5" path="/var/lib/kubelet/pods/98ac6015-12cb-427b-8278-a895a139c5c5/volumes"
Sep 16 19:25:35.534444 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:35.534373    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gk8lx"]
Sep 16 19:25:35.534807 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:35.534720    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a48cf56f-fe91-4a81-b5cb-03c364ca705c" containerName="main"
Sep 16 19:25:35.534807 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:35.534732    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a48cf56f-fe91-4a81-b5cb-03c364ca705c" containerName="main"
Sep 16 19:25:35.534807 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:35.534742    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98ac6015-12cb-427b-8278-a895a139c5c5" containerName="main"
Sep 16 19:25:35.534807 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:35.534749    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="98ac6015-12cb-427b-8278-a895a139c5c5" containerName="main"
Sep 16 19:25:35.534807 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:35.534755    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93d5d73f-4341-488c-a69d-22f1a5925172" containerName="main"
Sep 16 19:25:35.534807 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:35.534760    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="93d5d73f-4341-488c-a69d-22f1a5925172" containerName="main"
Sep 16 19:25:35.534807 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:35.534808    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a48cf56f-fe91-4a81-b5cb-03c364ca705c" containerName="main"
Sep 16 19:25:35.535048 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:35.534817    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="98ac6015-12cb-427b-8278-a895a139c5c5" containerName="main"
Sep 16 19:25:35.535048 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:35.534822    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="93d5d73f-4341-488c-a69d-22f1a5925172" containerName="main"
Sep 16 19:25:35.539507 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:35.539484    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gk8lx"
Sep 16 19:25:35.542476 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:35.542452    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"kube-root-ca.crt\""
Sep 16 19:25:35.542756 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:35.542741    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"openshift-service-ca.crt\""
Sep 16 19:25:35.543812 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:35.543792    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"default-dockercfg-kqd2q\""
Sep 16 19:25:35.543902 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:35.543826    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"owner-v1-kserve-self-signed-certs\""
Sep 16 19:25:35.551449 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:35.551429    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gk8lx"]
Sep 16 19:25:35.695917 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:35.695894    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a209f21d-69bb-4f8d-aa16-8a0c72dfc098-home\") pod \"owner-v1-kserve-75579ff5bc-gk8lx\" (UID: \"a209f21d-69bb-4f8d-aa16-8a0c72dfc098\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gk8lx"
Sep 16 19:25:35.696012 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:35.695926    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a209f21d-69bb-4f8d-aa16-8a0c72dfc098-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-gk8lx\" (UID: \"a209f21d-69bb-4f8d-aa16-8a0c72dfc098\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gk8lx"
Sep 16 19:25:35.696012 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:35.695943    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a209f21d-69bb-4f8d-aa16-8a0c72dfc098-model-cache\") pod \"owner-v1-kserve-75579ff5bc-gk8lx\" (UID: \"a209f21d-69bb-4f8d-aa16-8a0c72dfc098\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gk8lx"
Sep 16 19:25:35.696012 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:35.695962    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz52m\" (UniqueName: \"kubernetes.io/projected/a209f21d-69bb-4f8d-aa16-8a0c72dfc098-kube-api-access-nz52m\") pod \"owner-v1-kserve-75579ff5bc-gk8lx\" (UID: \"a209f21d-69bb-4f8d-aa16-8a0c72dfc098\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gk8lx"
Sep 16 19:25:35.696122 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:35.696036    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a209f21d-69bb-4f8d-aa16-8a0c72dfc098-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-gk8lx\" (UID: \"a209f21d-69bb-4f8d-aa16-8a0c72dfc098\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gk8lx"
Sep 16 19:25:35.696122 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:35.696067    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a209f21d-69bb-4f8d-aa16-8a0c72dfc098-dshm\") pod \"owner-v1-kserve-75579ff5bc-gk8lx\" (UID: \"a209f21d-69bb-4f8d-aa16-8a0c72dfc098\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gk8lx"
Sep 16 19:25:35.796701 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:35.796647    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a209f21d-69bb-4f8d-aa16-8a0c72dfc098-home\") pod \"owner-v1-kserve-75579ff5bc-gk8lx\" (UID: \"a209f21d-69bb-4f8d-aa16-8a0c72dfc098\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gk8lx"
Sep 16 19:25:35.796701 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:35.796681    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a209f21d-69bb-4f8d-aa16-8a0c72dfc098-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-gk8lx\" (UID: \"a209f21d-69bb-4f8d-aa16-8a0c72dfc098\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gk8lx"
Sep 16 19:25:35.796701 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:35.796699    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a209f21d-69bb-4f8d-aa16-8a0c72dfc098-model-cache\") pod \"owner-v1-kserve-75579ff5bc-gk8lx\" (UID: \"a209f21d-69bb-4f8d-aa16-8a0c72dfc098\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gk8lx"
Sep 16 19:25:35.796912 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:35.796720    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz52m\" (UniqueName: \"kubernetes.io/projected/a209f21d-69bb-4f8d-aa16-8a0c72dfc098-kube-api-access-nz52m\") pod \"owner-v1-kserve-75579ff5bc-gk8lx\" (UID: \"a209f21d-69bb-4f8d-aa16-8a0c72dfc098\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gk8lx"
Sep 16 19:25:35.796912 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:35.796751    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a209f21d-69bb-4f8d-aa16-8a0c72dfc098-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-gk8lx\" (UID: \"a209f21d-69bb-4f8d-aa16-8a0c72dfc098\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gk8lx"
Sep 16 19:25:35.796912 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:35.796880    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a209f21d-69bb-4f8d-aa16-8a0c72dfc098-dshm\") pod \"owner-v1-kserve-75579ff5bc-gk8lx\" (UID: \"a209f21d-69bb-4f8d-aa16-8a0c72dfc098\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gk8lx"
Sep 16 19:25:35.797137 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:35.797118    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a209f21d-69bb-4f8d-aa16-8a0c72dfc098-home\") pod \"owner-v1-kserve-75579ff5bc-gk8lx\" (UID: \"a209f21d-69bb-4f8d-aa16-8a0c72dfc098\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gk8lx"
Sep 16 19:25:35.797196 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:35.797157    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a209f21d-69bb-4f8d-aa16-8a0c72dfc098-model-cache\") pod \"owner-v1-kserve-75579ff5bc-gk8lx\" (UID: \"a209f21d-69bb-4f8d-aa16-8a0c72dfc098\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gk8lx"
Sep 16 19:25:35.797233 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:35.797193    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a209f21d-69bb-4f8d-aa16-8a0c72dfc098-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-gk8lx\" (UID: \"a209f21d-69bb-4f8d-aa16-8a0c72dfc098\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gk8lx"
Sep 16 19:25:35.799191 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:35.799169    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a209f21d-69bb-4f8d-aa16-8a0c72dfc098-dshm\") pod \"owner-v1-kserve-75579ff5bc-gk8lx\" (UID: \"a209f21d-69bb-4f8d-aa16-8a0c72dfc098\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gk8lx"
Sep 16 19:25:35.799492 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:35.799472    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a209f21d-69bb-4f8d-aa16-8a0c72dfc098-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-gk8lx\" (UID: \"a209f21d-69bb-4f8d-aa16-8a0c72dfc098\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gk8lx"
Sep 16 19:25:35.806493 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:35.806473    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz52m\" (UniqueName: \"kubernetes.io/projected/a209f21d-69bb-4f8d-aa16-8a0c72dfc098-kube-api-access-nz52m\") pod \"owner-v1-kserve-75579ff5bc-gk8lx\" (UID: \"a209f21d-69bb-4f8d-aa16-8a0c72dfc098\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gk8lx"
Sep 16 19:25:35.848986 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:35.848963    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gk8lx"
Sep 16 19:25:36.123122 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:36.123043    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gk8lx"]
Sep 16 19:25:36.126278 ip-10-0-128-247 kubenswrapper[2577]: W0916 19:25:36.126236    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda209f21d_69bb_4f8d_aa16_8a0c72dfc098.slice/crio-447324e58cdf854e3dfadcf0f48fe91947c6603b428bcffadc5f6fac7947819b WatchSource:0}: Error finding container 447324e58cdf854e3dfadcf0f48fe91947c6603b428bcffadc5f6fac7947819b: Status 404 returned error can't find the container with id 447324e58cdf854e3dfadcf0f48fe91947c6603b428bcffadc5f6fac7947819b
Sep 16 19:25:36.943659 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:36.943626    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gk8lx" event={"ID":"a209f21d-69bb-4f8d-aa16-8a0c72dfc098","Type":"ContainerStarted","Data":"ae03247f435276872efd8f403673f45fac8c4b340aba6150b02d66314fc1006c"}
Sep 16 19:25:36.944034 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:36.943701    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gk8lx" event={"ID":"a209f21d-69bb-4f8d-aa16-8a0c72dfc098","Type":"ContainerStarted","Data":"447324e58cdf854e3dfadcf0f48fe91947c6603b428bcffadc5f6fac7947819b"}
Sep 16 19:25:36.963685 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:36.963630    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gk8lx" podStartSLOduration=1.963616 podStartE2EDuration="1.963616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:25:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:25:36.961779058 +0000 UTC m=+2694.344668423" watchObservedRunningTime="2026-09-16 19:25:36.963616 +0000 UTC m=+2694.346505408"
Sep 16 19:25:37.849298 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:37.849265    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gk8lx"
Sep 16 19:25:45.849232 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:45.849197    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gk8lx"
Sep 16 19:25:45.862205 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:45.862178    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gk8lx"
Sep 16 19:25:45.993704 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:45.993676    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gk8lx"
Sep 16 19:25:47.739804 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:47.739762    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-q5gpc"]
Sep 16 19:25:47.744429 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:47.744406    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-q5gpc"
Sep 16 19:25:47.748139 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:47.748117    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"owner-v2-kserve-self-signed-certs\""
Sep 16 19:25:47.752601 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:47.752580    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-q5gpc"]
Sep 16 19:25:47.886298 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:47.886272    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d3fe742-a5b0-49cc-84c8-7495e91fcb9f-dshm\") pod \"owner-v2-kserve-5cf8959f95-q5gpc\" (UID: \"3d3fe742-a5b0-49cc-84c8-7495e91fcb9f\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-q5gpc"
Sep 16 19:25:47.886422 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:47.886306    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcwzt\" (UniqueName: \"kubernetes.io/projected/3d3fe742-a5b0-49cc-84c8-7495e91fcb9f-kube-api-access-vcwzt\") pod \"owner-v2-kserve-5cf8959f95-q5gpc\" (UID: \"3d3fe742-a5b0-49cc-84c8-7495e91fcb9f\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-q5gpc"
Sep 16 19:25:47.886422 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:47.886372    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d3fe742-a5b0-49cc-84c8-7495e91fcb9f-model-cache\") pod \"owner-v2-kserve-5cf8959f95-q5gpc\" (UID: \"3d3fe742-a5b0-49cc-84c8-7495e91fcb9f\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-q5gpc"
Sep 16 19:25:47.886422 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:47.886387    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d3fe742-a5b0-49cc-84c8-7495e91fcb9f-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-q5gpc\" (UID: \"3d3fe742-a5b0-49cc-84c8-7495e91fcb9f\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-q5gpc"
Sep 16 19:25:47.886422 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:47.886412    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d3fe742-a5b0-49cc-84c8-7495e91fcb9f-home\") pod \"owner-v2-kserve-5cf8959f95-q5gpc\" (UID: \"3d3fe742-a5b0-49cc-84c8-7495e91fcb9f\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-q5gpc"
Sep 16 19:25:47.886646 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:47.886488    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d3fe742-a5b0-49cc-84c8-7495e91fcb9f-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-q5gpc\" (UID: \"3d3fe742-a5b0-49cc-84c8-7495e91fcb9f\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-q5gpc"
Sep 16 19:25:47.987358 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:47.987333    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d3fe742-a5b0-49cc-84c8-7495e91fcb9f-home\") pod \"owner-v2-kserve-5cf8959f95-q5gpc\" (UID: \"3d3fe742-a5b0-49cc-84c8-7495e91fcb9f\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-q5gpc"
Sep 16 19:25:47.987464 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:47.987368    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d3fe742-a5b0-49cc-84c8-7495e91fcb9f-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-q5gpc\" (UID: \"3d3fe742-a5b0-49cc-84c8-7495e91fcb9f\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-q5gpc"
Sep 16 19:25:47.987464 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:47.987405    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d3fe742-a5b0-49cc-84c8-7495e91fcb9f-dshm\") pod \"owner-v2-kserve-5cf8959f95-q5gpc\" (UID: \"3d3fe742-a5b0-49cc-84c8-7495e91fcb9f\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-q5gpc"
Sep 16 19:25:47.987464 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:47.987435    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcwzt\" (UniqueName: \"kubernetes.io/projected/3d3fe742-a5b0-49cc-84c8-7495e91fcb9f-kube-api-access-vcwzt\") pod \"owner-v2-kserve-5cf8959f95-q5gpc\" (UID: \"3d3fe742-a5b0-49cc-84c8-7495e91fcb9f\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-q5gpc"
Sep 16 19:25:47.987675 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:47.987488    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d3fe742-a5b0-49cc-84c8-7495e91fcb9f-model-cache\") pod \"owner-v2-kserve-5cf8959f95-q5gpc\" (UID: \"3d3fe742-a5b0-49cc-84c8-7495e91fcb9f\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-q5gpc"
Sep 16 19:25:47.987675 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:47.987551    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d3fe742-a5b0-49cc-84c8-7495e91fcb9f-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-q5gpc\" (UID: \"3d3fe742-a5b0-49cc-84c8-7495e91fcb9f\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-q5gpc"
Sep 16 19:25:47.987930 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:47.987908    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d3fe742-a5b0-49cc-84c8-7495e91fcb9f-home\") pod \"owner-v2-kserve-5cf8959f95-q5gpc\" (UID: \"3d3fe742-a5b0-49cc-84c8-7495e91fcb9f\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-q5gpc"
Sep 16 19:25:47.988127 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:47.988102    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d3fe742-a5b0-49cc-84c8-7495e91fcb9f-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-q5gpc\" (UID: \"3d3fe742-a5b0-49cc-84c8-7495e91fcb9f\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-q5gpc"
Sep 16 19:25:47.988389 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:47.988186    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d3fe742-a5b0-49cc-84c8-7495e91fcb9f-model-cache\") pod \"owner-v2-kserve-5cf8959f95-q5gpc\" (UID: \"3d3fe742-a5b0-49cc-84c8-7495e91fcb9f\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-q5gpc"
Sep 16 19:25:47.990052 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:47.989989    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d3fe742-a5b0-49cc-84c8-7495e91fcb9f-dshm\") pod \"owner-v2-kserve-5cf8959f95-q5gpc\" (UID: \"3d3fe742-a5b0-49cc-84c8-7495e91fcb9f\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-q5gpc"
Sep 16 19:25:47.990418 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:47.990399    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d3fe742-a5b0-49cc-84c8-7495e91fcb9f-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-q5gpc\" (UID: \"3d3fe742-a5b0-49cc-84c8-7495e91fcb9f\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-q5gpc"
Sep 16 19:25:47.996951 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:47.996929    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcwzt\" (UniqueName: \"kubernetes.io/projected/3d3fe742-a5b0-49cc-84c8-7495e91fcb9f-kube-api-access-vcwzt\") pod \"owner-v2-kserve-5cf8959f95-q5gpc\" (UID: \"3d3fe742-a5b0-49cc-84c8-7495e91fcb9f\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-q5gpc"
Sep 16 19:25:48.055131 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:48.055108    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-q5gpc"
Sep 16 19:25:48.326655 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:48.326632    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-q5gpc"]
Sep 16 19:25:48.328753 ip-10-0-128-247 kubenswrapper[2577]: W0916 19:25:48.328728    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d3fe742_a5b0_49cc_84c8_7495e91fcb9f.slice/crio-dc5e6089a17ebd02cfac9a726ee9ea5d4ea27b265b05cbe66e6cc6dfd371fad9 WatchSource:0}: Error finding container dc5e6089a17ebd02cfac9a726ee9ea5d4ea27b265b05cbe66e6cc6dfd371fad9: Status 404 returned error can't find the container with id dc5e6089a17ebd02cfac9a726ee9ea5d4ea27b265b05cbe66e6cc6dfd371fad9
Sep 16 19:25:48.998507 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:48.998469    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-q5gpc" event={"ID":"3d3fe742-a5b0-49cc-84c8-7495e91fcb9f","Type":"ContainerStarted","Data":"b67f84754066943007f949e81f7bb37b8fa2173df3ff3d775080d15b2a2ee2d3"}
Sep 16 19:25:48.998907 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:48.998553    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-q5gpc" event={"ID":"3d3fe742-a5b0-49cc-84c8-7495e91fcb9f","Type":"ContainerStarted","Data":"dc5e6089a17ebd02cfac9a726ee9ea5d4ea27b265b05cbe66e6cc6dfd371fad9"}
Sep 16 19:25:49.018103 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:49.018049    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-q5gpc" podStartSLOduration=2.018033766 podStartE2EDuration="2.018033766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:25:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:25:49.016904777 +0000 UTC m=+2706.399794143" watchObservedRunningTime="2026-09-16 19:25:49.018033766 +0000 UTC m=+2706.400923131"
Sep 16 19:25:49.055477 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:49.055436    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-q5gpc"
Sep 16 19:25:58.055563 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:58.055511    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-q5gpc"
Sep 16 19:25:58.068250 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:58.068225    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-q5gpc"
Sep 16 19:25:59.050942 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:25:59.050914    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-q5gpc"
Sep 16 19:27:04.542967 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:04.542934    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gk8lx"]
Sep 16 19:27:19.562102 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:19.562051    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gk8lx" podUID="a209f21d-69bb-4f8d-aa16-8a0c72dfc098" containerName="main" containerID="cri-o://ae03247f435276872efd8f403673f45fac8c4b340aba6150b02d66314fc1006c" gracePeriod=45
Sep 16 19:27:19.802380 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:19.802358    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gk8lx"
Sep 16 19:27:19.880001 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:19.879942    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a209f21d-69bb-4f8d-aa16-8a0c72dfc098-home\") pod \"a209f21d-69bb-4f8d-aa16-8a0c72dfc098\" (UID: \"a209f21d-69bb-4f8d-aa16-8a0c72dfc098\") "
Sep 16 19:27:19.880001 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:19.879978    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a209f21d-69bb-4f8d-aa16-8a0c72dfc098-model-cache\") pod \"a209f21d-69bb-4f8d-aa16-8a0c72dfc098\" (UID: \"a209f21d-69bb-4f8d-aa16-8a0c72dfc098\") "
Sep 16 19:27:19.880174 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:19.880005    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a209f21d-69bb-4f8d-aa16-8a0c72dfc098-tmp-dir\") pod \"a209f21d-69bb-4f8d-aa16-8a0c72dfc098\" (UID: \"a209f21d-69bb-4f8d-aa16-8a0c72dfc098\") "
Sep 16 19:27:19.880174 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:19.880051    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a209f21d-69bb-4f8d-aa16-8a0c72dfc098-dshm\") pod \"a209f21d-69bb-4f8d-aa16-8a0c72dfc098\" (UID: \"a209f21d-69bb-4f8d-aa16-8a0c72dfc098\") "
Sep 16 19:27:19.880174 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:19.880071    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nz52m\" (UniqueName: \"kubernetes.io/projected/a209f21d-69bb-4f8d-aa16-8a0c72dfc098-kube-api-access-nz52m\") pod \"a209f21d-69bb-4f8d-aa16-8a0c72dfc098\" (UID: \"a209f21d-69bb-4f8d-aa16-8a0c72dfc098\") "
Sep 16 19:27:19.880174 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:19.880110    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a209f21d-69bb-4f8d-aa16-8a0c72dfc098-tls-certs\") pod \"a209f21d-69bb-4f8d-aa16-8a0c72dfc098\" (UID: \"a209f21d-69bb-4f8d-aa16-8a0c72dfc098\") "
Sep 16 19:27:19.880385 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:19.880181    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a209f21d-69bb-4f8d-aa16-8a0c72dfc098-home" (OuterVolumeSpecName: "home") pod "a209f21d-69bb-4f8d-aa16-8a0c72dfc098" (UID: "a209f21d-69bb-4f8d-aa16-8a0c72dfc098"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:19.880385 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:19.880226    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a209f21d-69bb-4f8d-aa16-8a0c72dfc098-model-cache" (OuterVolumeSpecName: "model-cache") pod "a209f21d-69bb-4f8d-aa16-8a0c72dfc098" (UID: "a209f21d-69bb-4f8d-aa16-8a0c72dfc098"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:19.880385 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:19.880333    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a209f21d-69bb-4f8d-aa16-8a0c72dfc098-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a209f21d-69bb-4f8d-aa16-8a0c72dfc098" (UID: "a209f21d-69bb-4f8d-aa16-8a0c72dfc098"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:19.880385 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:19.880351    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a209f21d-69bb-4f8d-aa16-8a0c72dfc098-home\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:27:19.880385 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:19.880363    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a209f21d-69bb-4f8d-aa16-8a0c72dfc098-model-cache\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:27:19.882369 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:19.882341    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a209f21d-69bb-4f8d-aa16-8a0c72dfc098-dshm" (OuterVolumeSpecName: "dshm") pod "a209f21d-69bb-4f8d-aa16-8a0c72dfc098" (UID: "a209f21d-69bb-4f8d-aa16-8a0c72dfc098"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:19.882799 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:19.882777    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a209f21d-69bb-4f8d-aa16-8a0c72dfc098-kube-api-access-nz52m" (OuterVolumeSpecName: "kube-api-access-nz52m") pod "a209f21d-69bb-4f8d-aa16-8a0c72dfc098" (UID: "a209f21d-69bb-4f8d-aa16-8a0c72dfc098"). InnerVolumeSpecName "kube-api-access-nz52m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:27:19.882799 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:19.882788    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a209f21d-69bb-4f8d-aa16-8a0c72dfc098-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a209f21d-69bb-4f8d-aa16-8a0c72dfc098" (UID: "a209f21d-69bb-4f8d-aa16-8a0c72dfc098"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:27:19.981704 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:19.981680    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a209f21d-69bb-4f8d-aa16-8a0c72dfc098-tmp-dir\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:27:19.981704 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:19.981700    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a209f21d-69bb-4f8d-aa16-8a0c72dfc098-dshm\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:27:19.981827 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:19.981709    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz52m\" (UniqueName: \"kubernetes.io/projected/a209f21d-69bb-4f8d-aa16-8a0c72dfc098-kube-api-access-nz52m\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:27:19.981827 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:19.981718    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a209f21d-69bb-4f8d-aa16-8a0c72dfc098-tls-certs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:27:20.364392 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:20.364294    2577 generic.go:356] "Generic (PLEG): container finished" podID="a209f21d-69bb-4f8d-aa16-8a0c72dfc098" containerID="ae03247f435276872efd8f403673f45fac8c4b340aba6150b02d66314fc1006c" exitCode=0
Sep 16 19:27:20.364392 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:20.364347    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gk8lx"
Sep 16 19:27:20.364392 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:20.364378    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gk8lx" event={"ID":"a209f21d-69bb-4f8d-aa16-8a0c72dfc098","Type":"ContainerDied","Data":"ae03247f435276872efd8f403673f45fac8c4b340aba6150b02d66314fc1006c"}
Sep 16 19:27:20.375118 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:20.364432    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gk8lx" event={"ID":"a209f21d-69bb-4f8d-aa16-8a0c72dfc098","Type":"ContainerDied","Data":"447324e58cdf854e3dfadcf0f48fe91947c6603b428bcffadc5f6fac7947819b"}
Sep 16 19:27:20.375118 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:20.364455    2577 scope.go:117] "RemoveContainer" containerID="ae03247f435276872efd8f403673f45fac8c4b340aba6150b02d66314fc1006c"
Sep 16 19:27:20.375118 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:20.373515    2577 scope.go:117] "RemoveContainer" containerID="ae03247f435276872efd8f403673f45fac8c4b340aba6150b02d66314fc1006c"
Sep 16 19:27:20.375118 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:27:20.373814    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae03247f435276872efd8f403673f45fac8c4b340aba6150b02d66314fc1006c\": container with ID starting with ae03247f435276872efd8f403673f45fac8c4b340aba6150b02d66314fc1006c not found: ID does not exist" containerID="ae03247f435276872efd8f403673f45fac8c4b340aba6150b02d66314fc1006c"
Sep 16 19:27:20.375118 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:20.373844    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae03247f435276872efd8f403673f45fac8c4b340aba6150b02d66314fc1006c"} err="failed to get container status \"ae03247f435276872efd8f403673f45fac8c4b340aba6150b02d66314fc1006c\": rpc error: code = NotFound desc = could not find container \"ae03247f435276872efd8f403673f45fac8c4b340aba6150b02d66314fc1006c\": container with ID starting with ae03247f435276872efd8f403673f45fac8c4b340aba6150b02d66314fc1006c not found: ID does not exist"
Sep 16 19:27:20.407447 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:20.407425    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gk8lx"]
Sep 16 19:27:20.413317 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:20.413297    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gk8lx"]
Sep 16 19:27:21.155230 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:21.155194    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a209f21d-69bb-4f8d-aa16-8a0c72dfc098" path="/var/lib/kubelet/pods/a209f21d-69bb-4f8d-aa16-8a0c72dfc098/volumes"
Sep 16 19:27:27.944480 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:27.944444    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-q5gpc"]
Sep 16 19:27:36.370405 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:36.370369    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt"]
Sep 16 19:27:36.370889 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:36.370739    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a209f21d-69bb-4f8d-aa16-8a0c72dfc098" containerName="main"
Sep 16 19:27:36.370889 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:36.370751    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a209f21d-69bb-4f8d-aa16-8a0c72dfc098" containerName="main"
Sep 16 19:27:36.370889 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:36.370806    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a209f21d-69bb-4f8d-aa16-8a0c72dfc098" containerName="main"
Sep 16 19:27:36.373721 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:36.373704    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt"
Sep 16 19:27:36.378337 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:36.378317    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\""
Sep 16 19:27:36.378457 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:36.378320    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 16 19:27:36.378586 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:36.378570    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"default-dockercfg-c8sjd\""
Sep 16 19:27:36.378669 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:36.378624    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 16 19:27:36.395827 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:36.395801    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt"]
Sep 16 19:27:36.507750 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:36.507718    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vhnc\" (UniqueName: \"kubernetes.io/projected/903f78ec-c30a-4e43-b1de-88b4fdfc81e7-kube-api-access-8vhnc\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt\" (UID: \"903f78ec-c30a-4e43-b1de-88b4fdfc81e7\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt"
Sep 16 19:27:36.507932 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:36.507758    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/903f78ec-c30a-4e43-b1de-88b4fdfc81e7-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt\" (UID: \"903f78ec-c30a-4e43-b1de-88b4fdfc81e7\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt"
Sep 16 19:27:36.507932 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:36.507821    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/903f78ec-c30a-4e43-b1de-88b4fdfc81e7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt\" (UID: \"903f78ec-c30a-4e43-b1de-88b4fdfc81e7\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt"
Sep 16 19:27:36.507932 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:36.507841    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/903f78ec-c30a-4e43-b1de-88b4fdfc81e7-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt\" (UID: \"903f78ec-c30a-4e43-b1de-88b4fdfc81e7\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt"
Sep 16 19:27:36.507932 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:36.507867    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/903f78ec-c30a-4e43-b1de-88b4fdfc81e7-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt\" (UID: \"903f78ec-c30a-4e43-b1de-88b4fdfc81e7\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt"
Sep 16 19:27:36.507932 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:36.507918    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/903f78ec-c30a-4e43-b1de-88b4fdfc81e7-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt\" (UID: \"903f78ec-c30a-4e43-b1de-88b4fdfc81e7\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt"
Sep 16 19:27:36.609007 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:36.608974    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/903f78ec-c30a-4e43-b1de-88b4fdfc81e7-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt\" (UID: \"903f78ec-c30a-4e43-b1de-88b4fdfc81e7\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt"
Sep 16 19:27:36.609201 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:36.609016    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vhnc\" (UniqueName: \"kubernetes.io/projected/903f78ec-c30a-4e43-b1de-88b4fdfc81e7-kube-api-access-8vhnc\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt\" (UID: \"903f78ec-c30a-4e43-b1de-88b4fdfc81e7\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt"
Sep 16 19:27:36.609201 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:36.609043    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/903f78ec-c30a-4e43-b1de-88b4fdfc81e7-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt\" (UID: \"903f78ec-c30a-4e43-b1de-88b4fdfc81e7\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt"
Sep 16 19:27:36.609201 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:36.609067    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/903f78ec-c30a-4e43-b1de-88b4fdfc81e7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt\" (UID: \"903f78ec-c30a-4e43-b1de-88b4fdfc81e7\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt"
Sep 16 19:27:36.609201 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:36.609085    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/903f78ec-c30a-4e43-b1de-88b4fdfc81e7-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt\" (UID: \"903f78ec-c30a-4e43-b1de-88b4fdfc81e7\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt"
Sep 16 19:27:36.609201 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:36.609105    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/903f78ec-c30a-4e43-b1de-88b4fdfc81e7-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt\" (UID: \"903f78ec-c30a-4e43-b1de-88b4fdfc81e7\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt"
Sep 16 19:27:36.609495 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:36.609442    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/903f78ec-c30a-4e43-b1de-88b4fdfc81e7-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt\" (UID: \"903f78ec-c30a-4e43-b1de-88b4fdfc81e7\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt"
Sep 16 19:27:36.609579 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:36.609513    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/903f78ec-c30a-4e43-b1de-88b4fdfc81e7-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt\" (UID: \"903f78ec-c30a-4e43-b1de-88b4fdfc81e7\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt"
Sep 16 19:27:36.609579 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:36.609521    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/903f78ec-c30a-4e43-b1de-88b4fdfc81e7-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt\" (UID: \"903f78ec-c30a-4e43-b1de-88b4fdfc81e7\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt"
Sep 16 19:27:36.611511 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:36.611488    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/903f78ec-c30a-4e43-b1de-88b4fdfc81e7-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt\" (UID: \"903f78ec-c30a-4e43-b1de-88b4fdfc81e7\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt"
Sep 16 19:27:36.611819 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:36.611798    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/903f78ec-c30a-4e43-b1de-88b4fdfc81e7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt\" (UID: \"903f78ec-c30a-4e43-b1de-88b4fdfc81e7\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt"
Sep 16 19:27:36.618988 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:36.618966    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vhnc\" (UniqueName: \"kubernetes.io/projected/903f78ec-c30a-4e43-b1de-88b4fdfc81e7-kube-api-access-8vhnc\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt\" (UID: \"903f78ec-c30a-4e43-b1de-88b4fdfc81e7\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt"
Sep 16 19:27:36.683918 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:36.683895    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt"
Sep 16 19:27:36.958068 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:36.958022    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt"]
Sep 16 19:27:36.960396 ip-10-0-128-247 kubenswrapper[2577]: W0916 19:27:36.960368    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod903f78ec_c30a_4e43_b1de_88b4fdfc81e7.slice/crio-3c05430f1e6ebcb87437ac1132cecaa4502badcf6de9db98f30b99e4e94d7527 WatchSource:0}: Error finding container 3c05430f1e6ebcb87437ac1132cecaa4502badcf6de9db98f30b99e4e94d7527: Status 404 returned error can't find the container with id 3c05430f1e6ebcb87437ac1132cecaa4502badcf6de9db98f30b99e4e94d7527
Sep 16 19:27:37.437827 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:37.437797    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt" event={"ID":"903f78ec-c30a-4e43-b1de-88b4fdfc81e7","Type":"ContainerStarted","Data":"a41e37f84f39315d05f30dfb9dc64d27db9663775db913edde91399539c347f6"}
Sep 16 19:27:37.438205 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:37.437847    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt" event={"ID":"903f78ec-c30a-4e43-b1de-88b4fdfc81e7","Type":"ContainerStarted","Data":"3c05430f1e6ebcb87437ac1132cecaa4502badcf6de9db98f30b99e4e94d7527"}
Sep 16 19:27:37.458622 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:37.458573    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt" podStartSLOduration=1.4585590480000001 podStartE2EDuration="1.458559048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:27:37.457106015 +0000 UTC m=+2814.839995380" watchObservedRunningTime="2026-09-16 19:27:37.458559048 +0000 UTC m=+2814.841448412"
Sep 16 19:27:37.684693 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:37.684664    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt"
Sep 16 19:27:42.963494 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:42.963424    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-q5gpc" podUID="3d3fe742-a5b0-49cc-84c8-7495e91fcb9f" containerName="main" containerID="cri-o://b67f84754066943007f949e81f7bb37b8fa2173df3ff3d775080d15b2a2ee2d3" gracePeriod=45
Sep 16 19:27:43.039953 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:43.039918    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-q5gpc" podUID="3d3fe742-a5b0-49cc-84c8-7495e91fcb9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Sep 16 19:27:43.212233 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:43.212204    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-q5gpc"
Sep 16 19:27:43.362233 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:43.362169    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d3fe742-a5b0-49cc-84c8-7495e91fcb9f-home\") pod \"3d3fe742-a5b0-49cc-84c8-7495e91fcb9f\" (UID: \"3d3fe742-a5b0-49cc-84c8-7495e91fcb9f\") "
Sep 16 19:27:43.362233 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:43.362209    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d3fe742-a5b0-49cc-84c8-7495e91fcb9f-dshm\") pod \"3d3fe742-a5b0-49cc-84c8-7495e91fcb9f\" (UID: \"3d3fe742-a5b0-49cc-84c8-7495e91fcb9f\") "
Sep 16 19:27:43.362233 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:43.362228    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d3fe742-a5b0-49cc-84c8-7495e91fcb9f-tmp-dir\") pod \"3d3fe742-a5b0-49cc-84c8-7495e91fcb9f\" (UID: \"3d3fe742-a5b0-49cc-84c8-7495e91fcb9f\") "
Sep 16 19:27:43.362503 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:43.362264    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcwzt\" (UniqueName: \"kubernetes.io/projected/3d3fe742-a5b0-49cc-84c8-7495e91fcb9f-kube-api-access-vcwzt\") pod \"3d3fe742-a5b0-49cc-84c8-7495e91fcb9f\" (UID: \"3d3fe742-a5b0-49cc-84c8-7495e91fcb9f\") "
Sep 16 19:27:43.362503 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:43.362302    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d3fe742-a5b0-49cc-84c8-7495e91fcb9f-model-cache\") pod \"3d3fe742-a5b0-49cc-84c8-7495e91fcb9f\" (UID: \"3d3fe742-a5b0-49cc-84c8-7495e91fcb9f\") "
Sep 16 19:27:43.362503 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:43.362319    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d3fe742-a5b0-49cc-84c8-7495e91fcb9f-tls-certs\") pod \"3d3fe742-a5b0-49cc-84c8-7495e91fcb9f\" (UID: \"3d3fe742-a5b0-49cc-84c8-7495e91fcb9f\") "
Sep 16 19:27:43.362503 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:43.362476    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3fe742-a5b0-49cc-84c8-7495e91fcb9f-home" (OuterVolumeSpecName: "home") pod "3d3fe742-a5b0-49cc-84c8-7495e91fcb9f" (UID: "3d3fe742-a5b0-49cc-84c8-7495e91fcb9f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:43.362503 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:43.362491    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3fe742-a5b0-49cc-84c8-7495e91fcb9f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3d3fe742-a5b0-49cc-84c8-7495e91fcb9f" (UID: "3d3fe742-a5b0-49cc-84c8-7495e91fcb9f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:43.362740 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:43.362634    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3fe742-a5b0-49cc-84c8-7495e91fcb9f-model-cache" (OuterVolumeSpecName: "model-cache") pod "3d3fe742-a5b0-49cc-84c8-7495e91fcb9f" (UID: "3d3fe742-a5b0-49cc-84c8-7495e91fcb9f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:43.364463 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:43.364437    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3fe742-a5b0-49cc-84c8-7495e91fcb9f-dshm" (OuterVolumeSpecName: "dshm") pod "3d3fe742-a5b0-49cc-84c8-7495e91fcb9f" (UID: "3d3fe742-a5b0-49cc-84c8-7495e91fcb9f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:43.364463 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:43.364445    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d3fe742-a5b0-49cc-84c8-7495e91fcb9f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3d3fe742-a5b0-49cc-84c8-7495e91fcb9f" (UID: "3d3fe742-a5b0-49cc-84c8-7495e91fcb9f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:27:43.364647 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:43.364516    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d3fe742-a5b0-49cc-84c8-7495e91fcb9f-kube-api-access-vcwzt" (OuterVolumeSpecName: "kube-api-access-vcwzt") pod "3d3fe742-a5b0-49cc-84c8-7495e91fcb9f" (UID: "3d3fe742-a5b0-49cc-84c8-7495e91fcb9f"). InnerVolumeSpecName "kube-api-access-vcwzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:27:43.463619 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:43.463592    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d3fe742-a5b0-49cc-84c8-7495e91fcb9f-home\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:27:43.463619 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:43.463613    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d3fe742-a5b0-49cc-84c8-7495e91fcb9f-dshm\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:27:43.463795 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:43.463622    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d3fe742-a5b0-49cc-84c8-7495e91fcb9f-tmp-dir\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:27:43.463795 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:43.463632    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcwzt\" (UniqueName: \"kubernetes.io/projected/3d3fe742-a5b0-49cc-84c8-7495e91fcb9f-kube-api-access-vcwzt\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:27:43.463795 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:43.463642    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d3fe742-a5b0-49cc-84c8-7495e91fcb9f-model-cache\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:27:43.463795 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:43.463651    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d3fe742-a5b0-49cc-84c8-7495e91fcb9f-tls-certs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:27:43.465618 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:43.465596    2577 generic.go:356] "Generic (PLEG): container finished" podID="3d3fe742-a5b0-49cc-84c8-7495e91fcb9f" containerID="b67f84754066943007f949e81f7bb37b8fa2173df3ff3d775080d15b2a2ee2d3" exitCode=0
Sep 16 19:27:43.465723 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:43.465638    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-q5gpc"
Sep 16 19:27:43.465723 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:43.465675    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-q5gpc" event={"ID":"3d3fe742-a5b0-49cc-84c8-7495e91fcb9f","Type":"ContainerDied","Data":"b67f84754066943007f949e81f7bb37b8fa2173df3ff3d775080d15b2a2ee2d3"}
Sep 16 19:27:43.465840 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:43.465725    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-q5gpc" event={"ID":"3d3fe742-a5b0-49cc-84c8-7495e91fcb9f","Type":"ContainerDied","Data":"dc5e6089a17ebd02cfac9a726ee9ea5d4ea27b265b05cbe66e6cc6dfd371fad9"}
Sep 16 19:27:43.465840 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:43.465748    2577 scope.go:117] "RemoveContainer" containerID="b67f84754066943007f949e81f7bb37b8fa2173df3ff3d775080d15b2a2ee2d3"
Sep 16 19:27:43.473789 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:43.473770    2577 scope.go:117] "RemoveContainer" containerID="b67f84754066943007f949e81f7bb37b8fa2173df3ff3d775080d15b2a2ee2d3"
Sep 16 19:27:43.474109 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:27:43.474088    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b67f84754066943007f949e81f7bb37b8fa2173df3ff3d775080d15b2a2ee2d3\": container with ID starting with b67f84754066943007f949e81f7bb37b8fa2173df3ff3d775080d15b2a2ee2d3 not found: ID does not exist" containerID="b67f84754066943007f949e81f7bb37b8fa2173df3ff3d775080d15b2a2ee2d3"
Sep 16 19:27:43.474177 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:43.474117    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b67f84754066943007f949e81f7bb37b8fa2173df3ff3d775080d15b2a2ee2d3"} err="failed to get container status \"b67f84754066943007f949e81f7bb37b8fa2173df3ff3d775080d15b2a2ee2d3\": rpc error: code = NotFound desc = could not find container \"b67f84754066943007f949e81f7bb37b8fa2173df3ff3d775080d15b2a2ee2d3\": container with ID starting with b67f84754066943007f949e81f7bb37b8fa2173df3ff3d775080d15b2a2ee2d3 not found: ID does not exist"
Sep 16 19:27:43.489924 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:43.489893    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-q5gpc"]
Sep 16 19:27:43.496194 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:43.496175    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-q5gpc"]
Sep 16 19:27:45.155764 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:45.155719    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d3fe742-a5b0-49cc-84c8-7495e91fcb9f" path="/var/lib/kubelet/pods/3d3fe742-a5b0-49cc-84c8-7495e91fcb9f/volumes"
Sep 16 19:27:46.684195 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:46.684162    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt"
Sep 16 19:27:46.696602 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:46.696579    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt"
Sep 16 19:27:47.494965 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:27:47.494931    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt"
Sep 16 19:30:01.028904 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:01.028872    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt"]
Sep 16 19:30:01.038372 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:30:01.038335    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:30:01.038372 ip-10-0-128-247 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:30:01.038372 ip-10-0-128-247 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 19:30:01.038372 ip-10-0-128-247 kubenswrapper[2577]:  > containerID="a41e37f84f39315d05f30dfb9dc64d27db9663775db913edde91399539c347f6" cmd=["/bin/sleep","15"]
Sep 16 19:30:01.038709 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:01.038389    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt" message="" err=<
Sep 16 19:30:01.038709 ip-10-0-128-247 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:30:01.038709 ip-10-0-128-247 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 19:30:01.038709 ip-10-0-128-247 kubenswrapper[2577]:  >
Sep 16 19:30:01.038709 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:30:01.038402    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:30:01.038709 ip-10-0-128-247 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:30:01.038709 ip-10-0-128-247 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 19:30:01.038709 ip-10-0-128-247 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt" podUID="903f78ec-c30a-4e43-b1de-88b4fdfc81e7" containerName="main" containerID="cri-o://a41e37f84f39315d05f30dfb9dc64d27db9663775db913edde91399539c347f6"
Sep 16 19:30:01.038709 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:01.038444    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt" podUID="903f78ec-c30a-4e43-b1de-88b4fdfc81e7" containerName="main" containerID="cri-o://a41e37f84f39315d05f30dfb9dc64d27db9663775db913edde91399539c347f6" gracePeriod=60
Sep 16 19:30:01.383213 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:01.383193    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt"
Sep 16 19:30:01.484618 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:01.484586    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/903f78ec-c30a-4e43-b1de-88b4fdfc81e7-tmp-dir\") pod \"903f78ec-c30a-4e43-b1de-88b4fdfc81e7\" (UID: \"903f78ec-c30a-4e43-b1de-88b4fdfc81e7\") "
Sep 16 19:30:01.484767 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:01.484627    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vhnc\" (UniqueName: \"kubernetes.io/projected/903f78ec-c30a-4e43-b1de-88b4fdfc81e7-kube-api-access-8vhnc\") pod \"903f78ec-c30a-4e43-b1de-88b4fdfc81e7\" (UID: \"903f78ec-c30a-4e43-b1de-88b4fdfc81e7\") "
Sep 16 19:30:01.484767 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:01.484669    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/903f78ec-c30a-4e43-b1de-88b4fdfc81e7-model-cache\") pod \"903f78ec-c30a-4e43-b1de-88b4fdfc81e7\" (UID: \"903f78ec-c30a-4e43-b1de-88b4fdfc81e7\") "
Sep 16 19:30:01.484767 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:01.484716    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/903f78ec-c30a-4e43-b1de-88b4fdfc81e7-tls-certs\") pod \"903f78ec-c30a-4e43-b1de-88b4fdfc81e7\" (UID: \"903f78ec-c30a-4e43-b1de-88b4fdfc81e7\") "
Sep 16 19:30:01.484767 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:01.484740    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/903f78ec-c30a-4e43-b1de-88b4fdfc81e7-home\") pod \"903f78ec-c30a-4e43-b1de-88b4fdfc81e7\" (UID: \"903f78ec-c30a-4e43-b1de-88b4fdfc81e7\") "
Sep 16 19:30:01.484767 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:01.484763    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/903f78ec-c30a-4e43-b1de-88b4fdfc81e7-dshm\") pod \"903f78ec-c30a-4e43-b1de-88b4fdfc81e7\" (UID: \"903f78ec-c30a-4e43-b1de-88b4fdfc81e7\") "
Sep 16 19:30:01.485014 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:01.484864    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/903f78ec-c30a-4e43-b1de-88b4fdfc81e7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "903f78ec-c30a-4e43-b1de-88b4fdfc81e7" (UID: "903f78ec-c30a-4e43-b1de-88b4fdfc81e7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:30:01.485014 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:01.484984    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/903f78ec-c30a-4e43-b1de-88b4fdfc81e7-model-cache" (OuterVolumeSpecName: "model-cache") pod "903f78ec-c30a-4e43-b1de-88b4fdfc81e7" (UID: "903f78ec-c30a-4e43-b1de-88b4fdfc81e7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:30:01.485014 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:01.484995    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/903f78ec-c30a-4e43-b1de-88b4fdfc81e7-home" (OuterVolumeSpecName: "home") pod "903f78ec-c30a-4e43-b1de-88b4fdfc81e7" (UID: "903f78ec-c30a-4e43-b1de-88b4fdfc81e7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:30:01.485174 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:01.485082    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/903f78ec-c30a-4e43-b1de-88b4fdfc81e7-tmp-dir\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:30:01.485174 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:01.485102    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/903f78ec-c30a-4e43-b1de-88b4fdfc81e7-model-cache\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:30:01.485174 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:01.485114    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/903f78ec-c30a-4e43-b1de-88b4fdfc81e7-home\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:30:01.487125 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:01.487096    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/903f78ec-c30a-4e43-b1de-88b4fdfc81e7-dshm" (OuterVolumeSpecName: "dshm") pod "903f78ec-c30a-4e43-b1de-88b4fdfc81e7" (UID: "903f78ec-c30a-4e43-b1de-88b4fdfc81e7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:30:01.487231 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:01.487179    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/903f78ec-c30a-4e43-b1de-88b4fdfc81e7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "903f78ec-c30a-4e43-b1de-88b4fdfc81e7" (UID: "903f78ec-c30a-4e43-b1de-88b4fdfc81e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:30:01.487231 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:01.487182    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/903f78ec-c30a-4e43-b1de-88b4fdfc81e7-kube-api-access-8vhnc" (OuterVolumeSpecName: "kube-api-access-8vhnc") pod "903f78ec-c30a-4e43-b1de-88b4fdfc81e7" (UID: "903f78ec-c30a-4e43-b1de-88b4fdfc81e7"). InnerVolumeSpecName "kube-api-access-8vhnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:30:01.586152 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:01.586094    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vhnc\" (UniqueName: \"kubernetes.io/projected/903f78ec-c30a-4e43-b1de-88b4fdfc81e7-kube-api-access-8vhnc\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:30:01.586152 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:01.586117    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/903f78ec-c30a-4e43-b1de-88b4fdfc81e7-tls-certs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:30:01.586152 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:01.586126    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/903f78ec-c30a-4e43-b1de-88b4fdfc81e7-dshm\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:30:02.046917 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:02.046878    2577 generic.go:356] "Generic (PLEG): container finished" podID="903f78ec-c30a-4e43-b1de-88b4fdfc81e7" containerID="a41e37f84f39315d05f30dfb9dc64d27db9663775db913edde91399539c347f6" exitCode=0
Sep 16 19:30:02.047390 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:02.046942    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt"
Sep 16 19:30:02.047390 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:02.046960    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt" event={"ID":"903f78ec-c30a-4e43-b1de-88b4fdfc81e7","Type":"ContainerDied","Data":"a41e37f84f39315d05f30dfb9dc64d27db9663775db913edde91399539c347f6"}
Sep 16 19:30:02.047390 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:02.047038    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt" event={"ID":"903f78ec-c30a-4e43-b1de-88b4fdfc81e7","Type":"ContainerDied","Data":"3c05430f1e6ebcb87437ac1132cecaa4502badcf6de9db98f30b99e4e94d7527"}
Sep 16 19:30:02.047390 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:02.047062    2577 scope.go:117] "RemoveContainer" containerID="a41e37f84f39315d05f30dfb9dc64d27db9663775db913edde91399539c347f6"
Sep 16 19:30:02.056868 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:02.056853    2577 scope.go:117] "RemoveContainer" containerID="a41e37f84f39315d05f30dfb9dc64d27db9663775db913edde91399539c347f6"
Sep 16 19:30:02.057122 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:30:02.057104    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a41e37f84f39315d05f30dfb9dc64d27db9663775db913edde91399539c347f6\": container with ID starting with a41e37f84f39315d05f30dfb9dc64d27db9663775db913edde91399539c347f6 not found: ID does not exist" containerID="a41e37f84f39315d05f30dfb9dc64d27db9663775db913edde91399539c347f6"
Sep 16 19:30:02.057175 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:02.057129    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a41e37f84f39315d05f30dfb9dc64d27db9663775db913edde91399539c347f6"} err="failed to get container status \"a41e37f84f39315d05f30dfb9dc64d27db9663775db913edde91399539c347f6\": rpc error: code = NotFound desc = could not find container \"a41e37f84f39315d05f30dfb9dc64d27db9663775db913edde91399539c347f6\": container with ID starting with a41e37f84f39315d05f30dfb9dc64d27db9663775db913edde91399539c347f6 not found: ID does not exist"
Sep 16 19:30:02.069510 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:02.069486    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt"]
Sep 16 19:30:02.075910 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:02.075890    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997lgkdt"]
Sep 16 19:30:03.156565 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:03.156512    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="903f78ec-c30a-4e43-b1de-88b4fdfc81e7" path="/var/lib/kubelet/pods/903f78ec-c30a-4e43-b1de-88b4fdfc81e7/volumes"
Sep 16 19:30:08.459968 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:08.459887    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk"]
Sep 16 19:30:08.460498 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:08.460354    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="903f78ec-c30a-4e43-b1de-88b4fdfc81e7" containerName="main"
Sep 16 19:30:08.460498 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:08.460371    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="903f78ec-c30a-4e43-b1de-88b4fdfc81e7" containerName="main"
Sep 16 19:30:08.460498 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:08.460401    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d3fe742-a5b0-49cc-84c8-7495e91fcb9f" containerName="main"
Sep 16 19:30:08.460498 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:08.460410    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d3fe742-a5b0-49cc-84c8-7495e91fcb9f" containerName="main"
Sep 16 19:30:08.460498 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:08.460495    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="903f78ec-c30a-4e43-b1de-88b4fdfc81e7" containerName="main"
Sep 16 19:30:08.460809 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:08.460506    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d3fe742-a5b0-49cc-84c8-7495e91fcb9f" containerName="main"
Sep 16 19:30:08.465090 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:08.465068    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk"
Sep 16 19:30:08.470141 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:08.470122    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"seaweedfs-s3-creds\""
Sep 16 19:30:08.470356 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:08.470334    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:30:08.470514 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:08.470493    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 16 19:30:08.470608 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:08.470568    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-przv5\""
Sep 16 19:30:08.472013 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:08.471998    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 16 19:30:08.472659 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:08.472640    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 16 19:30:08.483618 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:08.483600    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk"]
Sep 16 19:30:08.536001 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:08.535978    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk\" (UID: \"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk"
Sep 16 19:30:08.536104 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:08.536011    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxvjz\" (UniqueName: \"kubernetes.io/projected/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-kube-api-access-bxvjz\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk\" (UID: \"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk"
Sep 16 19:30:08.536104 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:08.536030    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk\" (UID: \"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk"
Sep 16 19:30:08.536104 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:08.536064    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk\" (UID: \"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk"
Sep 16 19:30:08.536274 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:08.536105    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk\" (UID: \"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk"
Sep 16 19:30:08.536274 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:08.536132    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk\" (UID: \"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk"
Sep 16 19:30:08.536274 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:08.536189    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk\" (UID: \"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk"
Sep 16 19:30:08.536274 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:08.536233    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk\" (UID: \"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk"
Sep 16 19:30:08.636911 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:08.636874    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk\" (UID: \"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk"
Sep 16 19:30:08.637013 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:08.636920    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk\" (UID: \"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk"
Sep 16 19:30:08.637013 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:08.636950    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk\" (UID: \"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk"
Sep 16 19:30:08.637013 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:08.636987    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk\" (UID: \"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk"
Sep 16 19:30:08.637180 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:08.637017    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk\" (UID: \"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk"
Sep 16 19:30:08.637180 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:08.637049    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxvjz\" (UniqueName: \"kubernetes.io/projected/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-kube-api-access-bxvjz\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk\" (UID: \"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk"
Sep 16 19:30:08.637180 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:08.637074    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk\" (UID: \"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk"
Sep 16 19:30:08.637180 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:08.637133    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk\" (UID: \"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk"
Sep 16 19:30:08.637580 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:08.637557    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk\" (UID: \"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk"
Sep 16 19:30:08.637735 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:08.637601    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk\" (UID: \"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk"
Sep 16 19:30:08.637735 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:08.637640    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk\" (UID: \"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk"
Sep 16 19:30:08.637874 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:08.637674    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk\" (UID: \"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk"
Sep 16 19:30:08.637874 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:08.637720    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk\" (UID: \"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk"
Sep 16 19:30:08.640106 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:08.640081    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk\" (UID: \"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk"
Sep 16 19:30:08.640348 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:08.640330    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk\" (UID: \"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk"
Sep 16 19:30:08.649514 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:08.649494    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxvjz\" (UniqueName: \"kubernetes.io/projected/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-kube-api-access-bxvjz\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk\" (UID: \"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk"
Sep 16 19:30:08.776171 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:08.776105    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk"
Sep 16 19:30:09.056975 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:09.056917    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk"]
Sep 16 19:30:09.059352 ip-10-0-128-247 kubenswrapper[2577]: W0916 19:30:09.059326    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5e8b2ad_ba43_4e6d_985e_5ac6dc434be9.slice/crio-b8494eda15bda8f4def31e2c1967f4c3ab4dfde466569a1b95d7f7728e171262 WatchSource:0}: Error finding container b8494eda15bda8f4def31e2c1967f4c3ab4dfde466569a1b95d7f7728e171262: Status 404 returned error can't find the container with id b8494eda15bda8f4def31e2c1967f4c3ab4dfde466569a1b95d7f7728e171262
Sep 16 19:30:09.079215 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:09.079191    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk" event={"ID":"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9","Type":"ContainerStarted","Data":"b8494eda15bda8f4def31e2c1967f4c3ab4dfde466569a1b95d7f7728e171262"}
Sep 16 19:30:10.084450 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:10.084414    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk" event={"ID":"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9","Type":"ContainerStarted","Data":"5528f85ec3d937dd463b101a1af1c85589843a51d1c88c49728b70013e7900ec"}
Sep 16 19:30:12.094407 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:12.094368    2577 generic.go:356] "Generic (PLEG): container finished" podID="e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9" containerID="5528f85ec3d937dd463b101a1af1c85589843a51d1c88c49728b70013e7900ec" exitCode=0
Sep 16 19:30:12.094805 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:12.094432    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk" event={"ID":"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9","Type":"ContainerDied","Data":"5528f85ec3d937dd463b101a1af1c85589843a51d1c88c49728b70013e7900ec"}
Sep 16 19:30:13.101093 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:13.101050    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk" event={"ID":"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9","Type":"ContainerStarted","Data":"7eadd6a5a09e0a80811815e4bc12b76e27900e4d9740cda4e4d8f1afc690e895"}
Sep 16 19:30:13.126246 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:13.126195    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk" podStartSLOduration=3.088755361 podStartE2EDuration="5.126179263s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.037423902s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:30:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:30:13.124786842 +0000 UTC m=+2970.507676218" watchObservedRunningTime="2026-09-16 19:30:13.126179263 +0000 UTC m=+2970.509068627"
Sep 16 19:30:13.776705 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:13.776611    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk"
Sep 16 19:30:18.776496 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:18.776464    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk"
Sep 16 19:30:18.777971 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:18.777944    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk" podUID="e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 16 19:30:28.776861 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:28.776819    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk" podUID="e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 16 19:30:38.777163 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:38.777124    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk" podUID="e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 16 19:30:48.777310 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:48.777264    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk" podUID="e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 16 19:30:58.777475 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:30:58.777431    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk" podUID="e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 16 19:31:08.776717 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:31:08.776666    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk" podUID="e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 16 19:31:18.776451 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:31:18.776410    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk" podUID="e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 16 19:31:28.777406 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:31:28.777364    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk" podUID="e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 16 19:31:38.777246 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:31:38.777153    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk" podUID="e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 16 19:31:48.776580 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:31:48.776514    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk" podUID="e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 16 19:31:58.787263 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:31:58.787234    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk"
Sep 16 19:31:58.795162 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:31:58.795127    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk"
Sep 16 19:32:12.718169 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:12.718131    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk"]
Sep 16 19:32:27.740507 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:27.740465    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk" podUID="e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9" containerName="main" containerID="cri-o://7eadd6a5a09e0a80811815e4bc12b76e27900e4d9740cda4e4d8f1afc690e895" gracePeriod=45
Sep 16 19:32:27.788334 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:27.788300    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk" podUID="e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 16 19:32:28.390860 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:28.390836    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk"
Sep 16 19:32:28.468245 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:28.468177    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-dshm\") pod \"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9\" (UID: \"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9\") "
Sep 16 19:32:28.468245 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:28.468215    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-cabundle-cert\") pod \"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9\" (UID: \"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9\") "
Sep 16 19:32:28.468431 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:28.468250    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-tmp-dir\") pod \"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9\" (UID: \"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9\") "
Sep 16 19:32:28.468431 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:28.468273    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-tls-certs\") pod \"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9\" (UID: \"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9\") "
Sep 16 19:32:28.468431 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:28.468313    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-kserve-provision-location\") pod \"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9\" (UID: \"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9\") "
Sep 16 19:32:28.468431 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:28.468360    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-home\") pod \"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9\" (UID: \"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9\") "
Sep 16 19:32:28.468431 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:28.468403    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxvjz\" (UniqueName: \"kubernetes.io/projected/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-kube-api-access-bxvjz\") pod \"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9\" (UID: \"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9\") "
Sep 16 19:32:28.468720 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:28.468431    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-model-cache\") pod \"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9\" (UID: \"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9\") "
Sep 16 19:32:28.468778 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:28.468740    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9" (UID: "e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:32:28.468949 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:28.468916    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-model-cache" (OuterVolumeSpecName: "model-cache") pod "e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9" (UID: "e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:32:28.469358 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:28.469327    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-home" (OuterVolumeSpecName: "home") pod "e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9" (UID: "e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:32:28.470626 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:28.470606    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-dshm" (OuterVolumeSpecName: "dshm") pod "e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9" (UID: "e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:32:28.470872 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:28.470854    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9" (UID: "e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:32:28.471063 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:28.471029    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-kube-api-access-bxvjz" (OuterVolumeSpecName: "kube-api-access-bxvjz") pod "e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9" (UID: "e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9"). InnerVolumeSpecName "kube-api-access-bxvjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:32:28.481690 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:28.481666    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9" (UID: "e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:32:28.491989 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:28.491963    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9" (UID: "e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:32:28.569492 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:28.569472    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-dshm\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:32:28.569595 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:28.569496    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-cabundle-cert\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:32:28.569595 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:28.569513    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-tmp-dir\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:32:28.569595 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:28.569521    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-tls-certs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:32:28.569595 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:28.569549    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-kserve-provision-location\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:32:28.569595 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:28.569558    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-home\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:32:28.569595 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:28.569567    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxvjz\" (UniqueName: \"kubernetes.io/projected/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-kube-api-access-bxvjz\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:32:28.569595 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:28.569578    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9-model-cache\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:32:28.715592 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:28.715562    2577 generic.go:356] "Generic (PLEG): container finished" podID="e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9" containerID="7eadd6a5a09e0a80811815e4bc12b76e27900e4d9740cda4e4d8f1afc690e895" exitCode=0
Sep 16 19:32:28.715706 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:28.715632    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk"
Sep 16 19:32:28.715789 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:28.715636    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk" event={"ID":"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9","Type":"ContainerDied","Data":"7eadd6a5a09e0a80811815e4bc12b76e27900e4d9740cda4e4d8f1afc690e895"}
Sep 16 19:32:28.715789 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:28.715753    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk" event={"ID":"e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9","Type":"ContainerDied","Data":"b8494eda15bda8f4def31e2c1967f4c3ab4dfde466569a1b95d7f7728e171262"}
Sep 16 19:32:28.715789 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:28.715781    2577 scope.go:117] "RemoveContainer" containerID="7eadd6a5a09e0a80811815e4bc12b76e27900e4d9740cda4e4d8f1afc690e895"
Sep 16 19:32:28.725008 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:28.724993    2577 scope.go:117] "RemoveContainer" containerID="5528f85ec3d937dd463b101a1af1c85589843a51d1c88c49728b70013e7900ec"
Sep 16 19:32:28.732716 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:28.732700    2577 scope.go:117] "RemoveContainer" containerID="7eadd6a5a09e0a80811815e4bc12b76e27900e4d9740cda4e4d8f1afc690e895"
Sep 16 19:32:28.732982 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:32:28.732964    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eadd6a5a09e0a80811815e4bc12b76e27900e4d9740cda4e4d8f1afc690e895\": container with ID starting with 7eadd6a5a09e0a80811815e4bc12b76e27900e4d9740cda4e4d8f1afc690e895 not found: ID does not exist" containerID="7eadd6a5a09e0a80811815e4bc12b76e27900e4d9740cda4e4d8f1afc690e895"
Sep 16 19:32:28.733035 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:28.732990    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eadd6a5a09e0a80811815e4bc12b76e27900e4d9740cda4e4d8f1afc690e895"} err="failed to get container status \"7eadd6a5a09e0a80811815e4bc12b76e27900e4d9740cda4e4d8f1afc690e895\": rpc error: code = NotFound desc = could not find container \"7eadd6a5a09e0a80811815e4bc12b76e27900e4d9740cda4e4d8f1afc690e895\": container with ID starting with 7eadd6a5a09e0a80811815e4bc12b76e27900e4d9740cda4e4d8f1afc690e895 not found: ID does not exist"
Sep 16 19:32:28.733035 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:28.733004    2577 scope.go:117] "RemoveContainer" containerID="5528f85ec3d937dd463b101a1af1c85589843a51d1c88c49728b70013e7900ec"
Sep 16 19:32:28.733254 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:32:28.733235    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5528f85ec3d937dd463b101a1af1c85589843a51d1c88c49728b70013e7900ec\": container with ID starting with 5528f85ec3d937dd463b101a1af1c85589843a51d1c88c49728b70013e7900ec not found: ID does not exist" containerID="5528f85ec3d937dd463b101a1af1c85589843a51d1c88c49728b70013e7900ec"
Sep 16 19:32:28.733308 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:28.733264    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5528f85ec3d937dd463b101a1af1c85589843a51d1c88c49728b70013e7900ec"} err="failed to get container status \"5528f85ec3d937dd463b101a1af1c85589843a51d1c88c49728b70013e7900ec\": rpc error: code = NotFound desc = could not find container \"5528f85ec3d937dd463b101a1af1c85589843a51d1c88c49728b70013e7900ec\": container with ID starting with 5528f85ec3d937dd463b101a1af1c85589843a51d1c88c49728b70013e7900ec not found: ID does not exist"
Sep 16 19:32:28.740963 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:28.740944    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk"]
Sep 16 19:32:28.746896 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:28.746874    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-85b9d8cdc9pjgzk"]
Sep 16 19:32:29.155895 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:29.155865    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9" path="/var/lib/kubelet/pods/e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9/volumes"
Sep 16 19:32:33.571303 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:33.571266    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-7c4b79bc4c-c6nh9"]
Sep 16 19:32:33.571815 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:33.571798    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9" containerName="main"
Sep 16 19:32:33.571864 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:33.571818    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9" containerName="main"
Sep 16 19:32:33.571864 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:33.571835    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9" containerName="storage-initializer"
Sep 16 19:32:33.571864 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:33.571845    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9" containerName="storage-initializer"
Sep 16 19:32:33.571967 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:33.571939    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5e8b2ad-ba43-4e6d-985e-5ac6dc434be9" containerName="main"
Sep 16 19:32:33.576911 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:33.576890    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-7c4b79bc4c-c6nh9"
Sep 16 19:32:33.580321 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:33.580296    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"default-dockercfg-v6ksb\""
Sep 16 19:32:33.581506 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:33.581485    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"kube-root-ca.crt\""
Sep 16 19:32:33.581651 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:33.581602    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"openshift-service-ca.crt\""
Sep 16 19:32:33.582824 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:33.582720    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"seaweedfs-s3-creds\""
Sep 16 19:32:33.584945 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:33.584628    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-7c4b79bc4c-c6nh9"]
Sep 16 19:32:33.709788 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:33.709760    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38d8bd86-7d98-4ca1-911b-c72585c59636-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-7c4b79bc4c-c6nh9\" (UID: \"38d8bd86-7d98-4ca1-911b-c72585c59636\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-7c4b79bc4c-c6nh9"
Sep 16 19:32:33.709932 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:33.709794    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw85b\" (UniqueName: \"kubernetes.io/projected/38d8bd86-7d98-4ca1-911b-c72585c59636-kube-api-access-zw85b\") pod \"e2e-wk-cfg-implicit-svc-kserve-7c4b79bc4c-c6nh9\" (UID: \"38d8bd86-7d98-4ca1-911b-c72585c59636\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-7c4b79bc4c-c6nh9"
Sep 16 19:32:33.811177 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:33.811149    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38d8bd86-7d98-4ca1-911b-c72585c59636-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-7c4b79bc4c-c6nh9\" (UID: \"38d8bd86-7d98-4ca1-911b-c72585c59636\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-7c4b79bc4c-c6nh9"
Sep 16 19:32:33.811316 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:33.811191    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw85b\" (UniqueName: \"kubernetes.io/projected/38d8bd86-7d98-4ca1-911b-c72585c59636-kube-api-access-zw85b\") pod \"e2e-wk-cfg-implicit-svc-kserve-7c4b79bc4c-c6nh9\" (UID: \"38d8bd86-7d98-4ca1-911b-c72585c59636\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-7c4b79bc4c-c6nh9"
Sep 16 19:32:33.811624 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:33.811600    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38d8bd86-7d98-4ca1-911b-c72585c59636-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-7c4b79bc4c-c6nh9\" (UID: \"38d8bd86-7d98-4ca1-911b-c72585c59636\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-7c4b79bc4c-c6nh9"
Sep 16 19:32:33.824419 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:33.824355    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw85b\" (UniqueName: \"kubernetes.io/projected/38d8bd86-7d98-4ca1-911b-c72585c59636-kube-api-access-zw85b\") pod \"e2e-wk-cfg-implicit-svc-kserve-7c4b79bc4c-c6nh9\" (UID: \"38d8bd86-7d98-4ca1-911b-c72585c59636\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-7c4b79bc4c-c6nh9"
Sep 16 19:32:33.888271 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:33.888247    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-7c4b79bc4c-c6nh9"
Sep 16 19:32:34.163589 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:34.163559    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-7c4b79bc4c-c6nh9"]
Sep 16 19:32:34.167303 ip-10-0-128-247 kubenswrapper[2577]: W0916 19:32:34.167278    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38d8bd86_7d98_4ca1_911b_c72585c59636.slice/crio-e504f99a67a252347b2169e07202687616860d74f1e9599587d75c8d4d8971b9 WatchSource:0}: Error finding container e504f99a67a252347b2169e07202687616860d74f1e9599587d75c8d4d8971b9: Status 404 returned error can't find the container with id e504f99a67a252347b2169e07202687616860d74f1e9599587d75c8d4d8971b9
Sep 16 19:32:34.742913 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:34.742876    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-7c4b79bc4c-c6nh9" event={"ID":"38d8bd86-7d98-4ca1-911b-c72585c59636","Type":"ContainerStarted","Data":"d4550fafd20d614a82f5baae3561c275e2cdf013aeedf5c5140ace996c270f62"}
Sep 16 19:32:34.743310 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:34.742929    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-7c4b79bc4c-c6nh9" event={"ID":"38d8bd86-7d98-4ca1-911b-c72585c59636","Type":"ContainerStarted","Data":"e504f99a67a252347b2169e07202687616860d74f1e9599587d75c8d4d8971b9"}
Sep 16 19:32:35.181031 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:35.180988    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-7c4b79bc4c-c6nh9"]
Sep 16 19:32:36.750301 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:36.750258    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-7c4b79bc4c-c6nh9" podUID="38d8bd86-7d98-4ca1-911b-c72585c59636" containerName="storage-initializer" containerID="cri-o://d4550fafd20d614a82f5baae3561c275e2cdf013aeedf5c5140ace996c270f62" gracePeriod=30
Sep 16 19:32:38.592142 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:38.592115    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-7c4b79bc4c-c6nh9"
Sep 16 19:32:38.757121 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:38.757045    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38d8bd86-7d98-4ca1-911b-c72585c59636-kserve-provision-location\") pod \"38d8bd86-7d98-4ca1-911b-c72585c59636\" (UID: \"38d8bd86-7d98-4ca1-911b-c72585c59636\") "
Sep 16 19:32:38.757121 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:38.757086    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zw85b\" (UniqueName: \"kubernetes.io/projected/38d8bd86-7d98-4ca1-911b-c72585c59636-kube-api-access-zw85b\") pod \"38d8bd86-7d98-4ca1-911b-c72585c59636\" (UID: \"38d8bd86-7d98-4ca1-911b-c72585c59636\") "
Sep 16 19:32:38.759552 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:38.759497    2577 generic.go:356] "Generic (PLEG): container finished" podID="38d8bd86-7d98-4ca1-911b-c72585c59636" containerID="d4550fafd20d614a82f5baae3561c275e2cdf013aeedf5c5140ace996c270f62" exitCode=0
Sep 16 19:32:38.759672 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:38.759565    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-7c4b79bc4c-c6nh9"
Sep 16 19:32:38.759672 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:38.759598    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-7c4b79bc4c-c6nh9" event={"ID":"38d8bd86-7d98-4ca1-911b-c72585c59636","Type":"ContainerDied","Data":"d4550fafd20d614a82f5baae3561c275e2cdf013aeedf5c5140ace996c270f62"}
Sep 16 19:32:38.759804 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:38.759673    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-7c4b79bc4c-c6nh9" event={"ID":"38d8bd86-7d98-4ca1-911b-c72585c59636","Type":"ContainerDied","Data":"e504f99a67a252347b2169e07202687616860d74f1e9599587d75c8d4d8971b9"}
Sep 16 19:32:38.759804 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:38.759705    2577 scope.go:117] "RemoveContainer" containerID="d4550fafd20d614a82f5baae3561c275e2cdf013aeedf5c5140ace996c270f62"
Sep 16 19:32:38.759804 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:38.759734    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38d8bd86-7d98-4ca1-911b-c72585c59636-kube-api-access-zw85b" (OuterVolumeSpecName: "kube-api-access-zw85b") pod "38d8bd86-7d98-4ca1-911b-c72585c59636" (UID: "38d8bd86-7d98-4ca1-911b-c72585c59636"). InnerVolumeSpecName "kube-api-access-zw85b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:32:38.770186 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:38.770166    2577 scope.go:117] "RemoveContainer" containerID="d4550fafd20d614a82f5baae3561c275e2cdf013aeedf5c5140ace996c270f62"
Sep 16 19:32:38.770475 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:32:38.770453    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4550fafd20d614a82f5baae3561c275e2cdf013aeedf5c5140ace996c270f62\": container with ID starting with d4550fafd20d614a82f5baae3561c275e2cdf013aeedf5c5140ace996c270f62 not found: ID does not exist" containerID="d4550fafd20d614a82f5baae3561c275e2cdf013aeedf5c5140ace996c270f62"
Sep 16 19:32:38.770593 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:38.770482    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4550fafd20d614a82f5baae3561c275e2cdf013aeedf5c5140ace996c270f62"} err="failed to get container status \"d4550fafd20d614a82f5baae3561c275e2cdf013aeedf5c5140ace996c270f62\": rpc error: code = NotFound desc = could not find container \"d4550fafd20d614a82f5baae3561c275e2cdf013aeedf5c5140ace996c270f62\": container with ID starting with d4550fafd20d614a82f5baae3561c275e2cdf013aeedf5c5140ace996c270f62 not found: ID does not exist"
Sep 16 19:32:38.833084 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:38.833053    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d8bd86-7d98-4ca1-911b-c72585c59636-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "38d8bd86-7d98-4ca1-911b-c72585c59636" (UID: "38d8bd86-7d98-4ca1-911b-c72585c59636"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:32:38.857653 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:38.857630    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38d8bd86-7d98-4ca1-911b-c72585c59636-kserve-provision-location\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:32:38.857653 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:38.857654    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw85b\" (UniqueName: \"kubernetes.io/projected/38d8bd86-7d98-4ca1-911b-c72585c59636-kube-api-access-zw85b\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:32:39.110772 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:39.110742    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-7c4b79bc4c-c6nh9"]
Sep 16 19:32:39.115699 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:39.115677    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-7c4b79bc4c-c6nh9"]
Sep 16 19:32:39.155370 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:39.155342    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38d8bd86-7d98-4ca1-911b-c72585c59636" path="/var/lib/kubelet/pods/38d8bd86-7d98-4ca1-911b-c72585c59636/volumes"
Sep 16 19:32:41.065771 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:41.065743    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6"]
Sep 16 19:32:41.066109 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:41.066090    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38d8bd86-7d98-4ca1-911b-c72585c59636" containerName="storage-initializer"
Sep 16 19:32:41.066109 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:41.066103    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="38d8bd86-7d98-4ca1-911b-c72585c59636" containerName="storage-initializer"
Sep 16 19:32:41.066188 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:41.066154    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="38d8bd86-7d98-4ca1-911b-c72585c59636" containerName="storage-initializer"
Sep 16 19:32:41.070417 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:41.070397    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6"
Sep 16 19:32:41.073542 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:41.073505    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 16 19:32:41.073542 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:41.073516    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 16 19:32:41.073730 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:41.073548    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 16 19:32:41.073730 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:41.073585    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:32:41.073730 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:41.073522    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 16 19:32:41.073730 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:41.073595    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-2fs62\""
Sep 16 19:32:41.084999 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:41.084976    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6"]
Sep 16 19:32:41.176458 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:41.176435    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b76308c-4051-48a7-b6af-a6c55195c647-cabundle-cert\") pod \"stop-feature-test-kserve-646f6cd48-hp5k6\" (UID: \"5b76308c-4051-48a7-b6af-a6c55195c647\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6"
Sep 16 19:32:41.176617 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:41.176484    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b76308c-4051-48a7-b6af-a6c55195c647-dshm\") pod \"stop-feature-test-kserve-646f6cd48-hp5k6\" (UID: \"5b76308c-4051-48a7-b6af-a6c55195c647\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6"
Sep 16 19:32:41.176617 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:41.176523    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b76308c-4051-48a7-b6af-a6c55195c647-tmp-dir\") pod \"stop-feature-test-kserve-646f6cd48-hp5k6\" (UID: \"5b76308c-4051-48a7-b6af-a6c55195c647\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6"
Sep 16 19:32:41.176617 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:41.176569    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b76308c-4051-48a7-b6af-a6c55195c647-kserve-provision-location\") pod \"stop-feature-test-kserve-646f6cd48-hp5k6\" (UID: \"5b76308c-4051-48a7-b6af-a6c55195c647\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6"
Sep 16 19:32:41.176617 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:41.176612    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b76308c-4051-48a7-b6af-a6c55195c647-tls-certs\") pod \"stop-feature-test-kserve-646f6cd48-hp5k6\" (UID: \"5b76308c-4051-48a7-b6af-a6c55195c647\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6"
Sep 16 19:32:41.176808 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:41.176648    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk7cq\" (UniqueName: \"kubernetes.io/projected/5b76308c-4051-48a7-b6af-a6c55195c647-kube-api-access-bk7cq\") pod \"stop-feature-test-kserve-646f6cd48-hp5k6\" (UID: \"5b76308c-4051-48a7-b6af-a6c55195c647\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6"
Sep 16 19:32:41.176808 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:41.176686    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b76308c-4051-48a7-b6af-a6c55195c647-home\") pod \"stop-feature-test-kserve-646f6cd48-hp5k6\" (UID: \"5b76308c-4051-48a7-b6af-a6c55195c647\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6"
Sep 16 19:32:41.176808 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:41.176721    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b76308c-4051-48a7-b6af-a6c55195c647-model-cache\") pod \"stop-feature-test-kserve-646f6cd48-hp5k6\" (UID: \"5b76308c-4051-48a7-b6af-a6c55195c647\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6"
Sep 16 19:32:41.277467 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:41.277432    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b76308c-4051-48a7-b6af-a6c55195c647-tmp-dir\") pod \"stop-feature-test-kserve-646f6cd48-hp5k6\" (UID: \"5b76308c-4051-48a7-b6af-a6c55195c647\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6"
Sep 16 19:32:41.277467 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:41.277462    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b76308c-4051-48a7-b6af-a6c55195c647-kserve-provision-location\") pod \"stop-feature-test-kserve-646f6cd48-hp5k6\" (UID: \"5b76308c-4051-48a7-b6af-a6c55195c647\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6"
Sep 16 19:32:41.277687 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:41.277484    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b76308c-4051-48a7-b6af-a6c55195c647-tls-certs\") pod \"stop-feature-test-kserve-646f6cd48-hp5k6\" (UID: \"5b76308c-4051-48a7-b6af-a6c55195c647\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6"
Sep 16 19:32:41.277687 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:41.277521    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk7cq\" (UniqueName: \"kubernetes.io/projected/5b76308c-4051-48a7-b6af-a6c55195c647-kube-api-access-bk7cq\") pod \"stop-feature-test-kserve-646f6cd48-hp5k6\" (UID: \"5b76308c-4051-48a7-b6af-a6c55195c647\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6"
Sep 16 19:32:41.277687 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:41.277582    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b76308c-4051-48a7-b6af-a6c55195c647-home\") pod \"stop-feature-test-kserve-646f6cd48-hp5k6\" (UID: \"5b76308c-4051-48a7-b6af-a6c55195c647\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6"
Sep 16 19:32:41.277687 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:41.277619    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b76308c-4051-48a7-b6af-a6c55195c647-model-cache\") pod \"stop-feature-test-kserve-646f6cd48-hp5k6\" (UID: \"5b76308c-4051-48a7-b6af-a6c55195c647\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6"
Sep 16 19:32:41.277687 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:41.277641    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b76308c-4051-48a7-b6af-a6c55195c647-cabundle-cert\") pod \"stop-feature-test-kserve-646f6cd48-hp5k6\" (UID: \"5b76308c-4051-48a7-b6af-a6c55195c647\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6"
Sep 16 19:32:41.277946 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:41.277690    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b76308c-4051-48a7-b6af-a6c55195c647-dshm\") pod \"stop-feature-test-kserve-646f6cd48-hp5k6\" (UID: \"5b76308c-4051-48a7-b6af-a6c55195c647\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6"
Sep 16 19:32:41.278124 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:41.278102    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b76308c-4051-48a7-b6af-a6c55195c647-tmp-dir\") pod \"stop-feature-test-kserve-646f6cd48-hp5k6\" (UID: \"5b76308c-4051-48a7-b6af-a6c55195c647\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6"
Sep 16 19:32:41.278326 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:41.278297    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b76308c-4051-48a7-b6af-a6c55195c647-model-cache\") pod \"stop-feature-test-kserve-646f6cd48-hp5k6\" (UID: \"5b76308c-4051-48a7-b6af-a6c55195c647\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6"
Sep 16 19:32:41.278490 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:41.278469    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b76308c-4051-48a7-b6af-a6c55195c647-home\") pod \"stop-feature-test-kserve-646f6cd48-hp5k6\" (UID: \"5b76308c-4051-48a7-b6af-a6c55195c647\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6"
Sep 16 19:32:41.278611 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:41.278513    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b76308c-4051-48a7-b6af-a6c55195c647-cabundle-cert\") pod \"stop-feature-test-kserve-646f6cd48-hp5k6\" (UID: \"5b76308c-4051-48a7-b6af-a6c55195c647\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6"
Sep 16 19:32:41.278677 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:41.278645    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b76308c-4051-48a7-b6af-a6c55195c647-kserve-provision-location\") pod \"stop-feature-test-kserve-646f6cd48-hp5k6\" (UID: \"5b76308c-4051-48a7-b6af-a6c55195c647\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6"
Sep 16 19:32:41.280716 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:41.280694    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b76308c-4051-48a7-b6af-a6c55195c647-tls-certs\") pod \"stop-feature-test-kserve-646f6cd48-hp5k6\" (UID: \"5b76308c-4051-48a7-b6af-a6c55195c647\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6"
Sep 16 19:32:41.280825 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:41.280740    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b76308c-4051-48a7-b6af-a6c55195c647-dshm\") pod \"stop-feature-test-kserve-646f6cd48-hp5k6\" (UID: \"5b76308c-4051-48a7-b6af-a6c55195c647\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6"
Sep 16 19:32:41.291449 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:41.291426    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk7cq\" (UniqueName: \"kubernetes.io/projected/5b76308c-4051-48a7-b6af-a6c55195c647-kube-api-access-bk7cq\") pod \"stop-feature-test-kserve-646f6cd48-hp5k6\" (UID: \"5b76308c-4051-48a7-b6af-a6c55195c647\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6"
Sep 16 19:32:41.381158 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:41.381099    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6"
Sep 16 19:32:41.663408 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:41.663385    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6"]
Sep 16 19:32:41.665289 ip-10-0-128-247 kubenswrapper[2577]: W0916 19:32:41.665263    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b76308c_4051_48a7_b6af_a6c55195c647.slice/crio-577bff685b98f5117679f4de08a2cf3e7ded73e794a22cf114dd1cf6a5ddcc71 WatchSource:0}: Error finding container 577bff685b98f5117679f4de08a2cf3e7ded73e794a22cf114dd1cf6a5ddcc71: Status 404 returned error can't find the container with id 577bff685b98f5117679f4de08a2cf3e7ded73e794a22cf114dd1cf6a5ddcc71
Sep 16 19:32:41.775270 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:41.775231    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6" event={"ID":"5b76308c-4051-48a7-b6af-a6c55195c647","Type":"ContainerStarted","Data":"797cc855ec48bc72a458f94ab288b3d69133e7ff2b2a4499d6a93a0d81aa3ee4"}
Sep 16 19:32:41.775386 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:41.775296    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6" event={"ID":"5b76308c-4051-48a7-b6af-a6c55195c647","Type":"ContainerStarted","Data":"577bff685b98f5117679f4de08a2cf3e7ded73e794a22cf114dd1cf6a5ddcc71"}
Sep 16 19:32:44.790048 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:44.790012    2577 generic.go:356] "Generic (PLEG): container finished" podID="5b76308c-4051-48a7-b6af-a6c55195c647" containerID="797cc855ec48bc72a458f94ab288b3d69133e7ff2b2a4499d6a93a0d81aa3ee4" exitCode=0
Sep 16 19:32:44.790443 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:44.790088    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6" event={"ID":"5b76308c-4051-48a7-b6af-a6c55195c647","Type":"ContainerDied","Data":"797cc855ec48bc72a458f94ab288b3d69133e7ff2b2a4499d6a93a0d81aa3ee4"}
Sep 16 19:32:45.796822 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:45.796781    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6" event={"ID":"5b76308c-4051-48a7-b6af-a6c55195c647","Type":"ContainerStarted","Data":"37602472c0ad8c8a5f3f045fa144d19ee7b1cf98e7dfbf527032f9e9935956b9"}
Sep 16 19:32:45.822713 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:45.822662    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6" podStartSLOduration=2.671376511 podStartE2EDuration="4.822648101s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.15127159s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:32:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:32:45.82126288 +0000 UTC m=+3123.204152256" watchObservedRunningTime="2026-09-16 19:32:45.822648101 +0000 UTC m=+3123.205537466"
Sep 16 19:32:46.381821 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:46.381780    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6"
Sep 16 19:32:51.381456 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:51.381420    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6"
Sep 16 19:32:51.383078 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:32:51.383048    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6" podUID="5b76308c-4051-48a7-b6af-a6c55195c647" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Sep 16 19:33:01.381817 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:33:01.381766    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6" podUID="5b76308c-4051-48a7-b6af-a6c55195c647" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Sep 16 19:33:11.382427 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:33:11.382327    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6" podUID="5b76308c-4051-48a7-b6af-a6c55195c647" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Sep 16 19:33:21.381745 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:33:21.381698    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6" podUID="5b76308c-4051-48a7-b6af-a6c55195c647" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Sep 16 19:33:31.382567 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:33:31.382499    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6" podUID="5b76308c-4051-48a7-b6af-a6c55195c647" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Sep 16 19:33:41.382343 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:33:41.382302    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6" podUID="5b76308c-4051-48a7-b6af-a6c55195c647" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Sep 16 19:33:51.382144 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:33:51.382100    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6" podUID="5b76308c-4051-48a7-b6af-a6c55195c647" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Sep 16 19:34:01.382316 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:01.382277    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6" podUID="5b76308c-4051-48a7-b6af-a6c55195c647" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Sep 16 19:34:11.381885 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:11.381844    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6" podUID="5b76308c-4051-48a7-b6af-a6c55195c647" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Sep 16 19:34:21.381902 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:21.381855    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6" podUID="5b76308c-4051-48a7-b6af-a6c55195c647" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Sep 16 19:34:31.391688 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:31.391651    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6"
Sep 16 19:34:31.399462 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:31.399437    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6"
Sep 16 19:34:33.357547 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:33.357491    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6"]
Sep 16 19:34:48.380496 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:48.380391    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6" podUID="5b76308c-4051-48a7-b6af-a6c55195c647" containerName="main" containerID="cri-o://37602472c0ad8c8a5f3f045fa144d19ee7b1cf98e7dfbf527032f9e9935956b9" gracePeriod=45
Sep 16 19:34:49.030124 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:49.030100    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6"
Sep 16 19:34:49.108797 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:49.108769    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b76308c-4051-48a7-b6af-a6c55195c647-home\") pod \"5b76308c-4051-48a7-b6af-a6c55195c647\" (UID: \"5b76308c-4051-48a7-b6af-a6c55195c647\") "
Sep 16 19:34:49.108919 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:49.108823    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bk7cq\" (UniqueName: \"kubernetes.io/projected/5b76308c-4051-48a7-b6af-a6c55195c647-kube-api-access-bk7cq\") pod \"5b76308c-4051-48a7-b6af-a6c55195c647\" (UID: \"5b76308c-4051-48a7-b6af-a6c55195c647\") "
Sep 16 19:34:49.108919 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:49.108848    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b76308c-4051-48a7-b6af-a6c55195c647-cabundle-cert\") pod \"5b76308c-4051-48a7-b6af-a6c55195c647\" (UID: \"5b76308c-4051-48a7-b6af-a6c55195c647\") "
Sep 16 19:34:49.109030 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:49.108941    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b76308c-4051-48a7-b6af-a6c55195c647-tls-certs\") pod \"5b76308c-4051-48a7-b6af-a6c55195c647\" (UID: \"5b76308c-4051-48a7-b6af-a6c55195c647\") "
Sep 16 19:34:49.109030 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:49.108991    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b76308c-4051-48a7-b6af-a6c55195c647-dshm\") pod \"5b76308c-4051-48a7-b6af-a6c55195c647\" (UID: \"5b76308c-4051-48a7-b6af-a6c55195c647\") "
Sep 16 19:34:49.109148 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:49.109041    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b76308c-4051-48a7-b6af-a6c55195c647-kserve-provision-location\") pod \"5b76308c-4051-48a7-b6af-a6c55195c647\" (UID: \"5b76308c-4051-48a7-b6af-a6c55195c647\") "
Sep 16 19:34:49.109148 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:49.109074    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b76308c-4051-48a7-b6af-a6c55195c647-model-cache\") pod \"5b76308c-4051-48a7-b6af-a6c55195c647\" (UID: \"5b76308c-4051-48a7-b6af-a6c55195c647\") "
Sep 16 19:34:49.109148 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:49.109114    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b76308c-4051-48a7-b6af-a6c55195c647-tmp-dir\") pod \"5b76308c-4051-48a7-b6af-a6c55195c647\" (UID: \"5b76308c-4051-48a7-b6af-a6c55195c647\") "
Sep 16 19:34:49.109463 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:49.109304    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b76308c-4051-48a7-b6af-a6c55195c647-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5b76308c-4051-48a7-b6af-a6c55195c647" (UID: "5b76308c-4051-48a7-b6af-a6c55195c647"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:34:49.109463 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:49.109411    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b76308c-4051-48a7-b6af-a6c55195c647-cabundle-cert\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:34:49.109773 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:49.109579    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b76308c-4051-48a7-b6af-a6c55195c647-model-cache" (OuterVolumeSpecName: "model-cache") pod "5b76308c-4051-48a7-b6af-a6c55195c647" (UID: "5b76308c-4051-48a7-b6af-a6c55195c647"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:34:49.109823 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:49.109781    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b76308c-4051-48a7-b6af-a6c55195c647-home" (OuterVolumeSpecName: "home") pod "5b76308c-4051-48a7-b6af-a6c55195c647" (UID: "5b76308c-4051-48a7-b6af-a6c55195c647"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:34:49.111396 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:49.111364    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b76308c-4051-48a7-b6af-a6c55195c647-kube-api-access-bk7cq" (OuterVolumeSpecName: "kube-api-access-bk7cq") pod "5b76308c-4051-48a7-b6af-a6c55195c647" (UID: "5b76308c-4051-48a7-b6af-a6c55195c647"). InnerVolumeSpecName "kube-api-access-bk7cq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:34:49.111470 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:49.111442    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b76308c-4051-48a7-b6af-a6c55195c647-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5b76308c-4051-48a7-b6af-a6c55195c647" (UID: "5b76308c-4051-48a7-b6af-a6c55195c647"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:34:49.111648 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:49.111630    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b76308c-4051-48a7-b6af-a6c55195c647-dshm" (OuterVolumeSpecName: "dshm") pod "5b76308c-4051-48a7-b6af-a6c55195c647" (UID: "5b76308c-4051-48a7-b6af-a6c55195c647"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:34:49.122246 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:49.122221    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b76308c-4051-48a7-b6af-a6c55195c647-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5b76308c-4051-48a7-b6af-a6c55195c647" (UID: "5b76308c-4051-48a7-b6af-a6c55195c647"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:34:49.135025 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:49.134999    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b76308c-4051-48a7-b6af-a6c55195c647-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5b76308c-4051-48a7-b6af-a6c55195c647" (UID: "5b76308c-4051-48a7-b6af-a6c55195c647"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:34:49.209826 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:49.209801    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b76308c-4051-48a7-b6af-a6c55195c647-tls-certs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:34:49.209826 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:49.209825    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b76308c-4051-48a7-b6af-a6c55195c647-dshm\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:34:49.209945 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:49.209836    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b76308c-4051-48a7-b6af-a6c55195c647-kserve-provision-location\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:34:49.209945 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:49.209845    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b76308c-4051-48a7-b6af-a6c55195c647-model-cache\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:34:49.209945 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:49.209854    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b76308c-4051-48a7-b6af-a6c55195c647-tmp-dir\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:34:49.209945 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:49.209862    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b76308c-4051-48a7-b6af-a6c55195c647-home\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:34:49.209945 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:49.209869    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk7cq\" (UniqueName: \"kubernetes.io/projected/5b76308c-4051-48a7-b6af-a6c55195c647-kube-api-access-bk7cq\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:34:49.344111 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:49.344080    2577 generic.go:356] "Generic (PLEG): container finished" podID="5b76308c-4051-48a7-b6af-a6c55195c647" containerID="37602472c0ad8c8a5f3f045fa144d19ee7b1cf98e7dfbf527032f9e9935956b9" exitCode=0
Sep 16 19:34:49.344217 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:49.344139    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6"
Sep 16 19:34:49.344217 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:49.344142    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6" event={"ID":"5b76308c-4051-48a7-b6af-a6c55195c647","Type":"ContainerDied","Data":"37602472c0ad8c8a5f3f045fa144d19ee7b1cf98e7dfbf527032f9e9935956b9"}
Sep 16 19:34:49.344217 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:49.344194    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6" event={"ID":"5b76308c-4051-48a7-b6af-a6c55195c647","Type":"ContainerDied","Data":"577bff685b98f5117679f4de08a2cf3e7ded73e794a22cf114dd1cf6a5ddcc71"}
Sep 16 19:34:49.344217 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:49.344215    2577 scope.go:117] "RemoveContainer" containerID="37602472c0ad8c8a5f3f045fa144d19ee7b1cf98e7dfbf527032f9e9935956b9"
Sep 16 19:34:49.353813 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:49.353794    2577 scope.go:117] "RemoveContainer" containerID="797cc855ec48bc72a458f94ab288b3d69133e7ff2b2a4499d6a93a0d81aa3ee4"
Sep 16 19:34:49.361433 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:49.361416    2577 scope.go:117] "RemoveContainer" containerID="37602472c0ad8c8a5f3f045fa144d19ee7b1cf98e7dfbf527032f9e9935956b9"
Sep 16 19:34:49.361752 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:34:49.361731    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37602472c0ad8c8a5f3f045fa144d19ee7b1cf98e7dfbf527032f9e9935956b9\": container with ID starting with 37602472c0ad8c8a5f3f045fa144d19ee7b1cf98e7dfbf527032f9e9935956b9 not found: ID does not exist" containerID="37602472c0ad8c8a5f3f045fa144d19ee7b1cf98e7dfbf527032f9e9935956b9"
Sep 16 19:34:49.361819 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:49.361760    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37602472c0ad8c8a5f3f045fa144d19ee7b1cf98e7dfbf527032f9e9935956b9"} err="failed to get container status \"37602472c0ad8c8a5f3f045fa144d19ee7b1cf98e7dfbf527032f9e9935956b9\": rpc error: code = NotFound desc = could not find container \"37602472c0ad8c8a5f3f045fa144d19ee7b1cf98e7dfbf527032f9e9935956b9\": container with ID starting with 37602472c0ad8c8a5f3f045fa144d19ee7b1cf98e7dfbf527032f9e9935956b9 not found: ID does not exist"
Sep 16 19:34:49.361819 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:49.361777    2577 scope.go:117] "RemoveContainer" containerID="797cc855ec48bc72a458f94ab288b3d69133e7ff2b2a4499d6a93a0d81aa3ee4"
Sep 16 19:34:49.362051 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:34:49.362037    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"797cc855ec48bc72a458f94ab288b3d69133e7ff2b2a4499d6a93a0d81aa3ee4\": container with ID starting with 797cc855ec48bc72a458f94ab288b3d69133e7ff2b2a4499d6a93a0d81aa3ee4 not found: ID does not exist" containerID="797cc855ec48bc72a458f94ab288b3d69133e7ff2b2a4499d6a93a0d81aa3ee4"
Sep 16 19:34:49.362104 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:49.362054    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"797cc855ec48bc72a458f94ab288b3d69133e7ff2b2a4499d6a93a0d81aa3ee4"} err="failed to get container status \"797cc855ec48bc72a458f94ab288b3d69133e7ff2b2a4499d6a93a0d81aa3ee4\": rpc error: code = NotFound desc = could not find container \"797cc855ec48bc72a458f94ab288b3d69133e7ff2b2a4499d6a93a0d81aa3ee4\": container with ID starting with 797cc855ec48bc72a458f94ab288b3d69133e7ff2b2a4499d6a93a0d81aa3ee4 not found: ID does not exist"
Sep 16 19:34:49.372561 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:49.372489    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6"]
Sep 16 19:34:49.385699 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:49.385680    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-hp5k6"]
Sep 16 19:34:51.155765 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:51.155732    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b76308c-4051-48a7-b6af-a6c55195c647" path="/var/lib/kubelet/pods/5b76308c-4051-48a7-b6af-a6c55195c647/volumes"
Sep 16 19:34:53.361971 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:53.361937    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5"]
Sep 16 19:34:53.362332 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:53.362284    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b76308c-4051-48a7-b6af-a6c55195c647" containerName="storage-initializer"
Sep 16 19:34:53.362332 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:53.362295    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b76308c-4051-48a7-b6af-a6c55195c647" containerName="storage-initializer"
Sep 16 19:34:53.362332 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:53.362306    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b76308c-4051-48a7-b6af-a6c55195c647" containerName="main"
Sep 16 19:34:53.362332 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:53.362311    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b76308c-4051-48a7-b6af-a6c55195c647" containerName="main"
Sep 16 19:34:53.362472 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:53.362366    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b76308c-4051-48a7-b6af-a6c55195c647" containerName="main"
Sep 16 19:34:53.365421 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:53.365402    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5"
Sep 16 19:34:53.369081 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:53.369056    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:34:53.369176 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:53.369123    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-2fs62\""
Sep 16 19:34:53.369378 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:53.369358    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 16 19:34:53.369448 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:53.369428    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 16 19:34:53.369504 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:53.369431    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 16 19:34:53.370291 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:53.370277    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 16 19:34:53.381335 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:53.381314    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5"]
Sep 16 19:34:53.439630 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:53.439604    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-cabundle-cert\") pod \"stop-feature-test-kserve-646f6cd48-fbmc5\" (UID: \"e8623810-ff6c-4bc5-b8b0-49e963a6aefc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5"
Sep 16 19:34:53.439720 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:53.439636    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-tmp-dir\") pod \"stop-feature-test-kserve-646f6cd48-fbmc5\" (UID: \"e8623810-ff6c-4bc5-b8b0-49e963a6aefc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5"
Sep 16 19:34:53.439720 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:53.439667    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-tls-certs\") pod \"stop-feature-test-kserve-646f6cd48-fbmc5\" (UID: \"e8623810-ff6c-4bc5-b8b0-49e963a6aefc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5"
Sep 16 19:34:53.439720 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:53.439711    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-kserve-provision-location\") pod \"stop-feature-test-kserve-646f6cd48-fbmc5\" (UID: \"e8623810-ff6c-4bc5-b8b0-49e963a6aefc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5"
Sep 16 19:34:53.439840 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:53.439741    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpcbj\" (UniqueName: \"kubernetes.io/projected/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-kube-api-access-lpcbj\") pod \"stop-feature-test-kserve-646f6cd48-fbmc5\" (UID: \"e8623810-ff6c-4bc5-b8b0-49e963a6aefc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5"
Sep 16 19:34:53.439840 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:53.439790    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-dshm\") pod \"stop-feature-test-kserve-646f6cd48-fbmc5\" (UID: \"e8623810-ff6c-4bc5-b8b0-49e963a6aefc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5"
Sep 16 19:34:53.439840 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:53.439814    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-home\") pod \"stop-feature-test-kserve-646f6cd48-fbmc5\" (UID: \"e8623810-ff6c-4bc5-b8b0-49e963a6aefc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5"
Sep 16 19:34:53.439935 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:53.439843    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-model-cache\") pod \"stop-feature-test-kserve-646f6cd48-fbmc5\" (UID: \"e8623810-ff6c-4bc5-b8b0-49e963a6aefc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5"
Sep 16 19:34:53.540730 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:53.540706    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-cabundle-cert\") pod \"stop-feature-test-kserve-646f6cd48-fbmc5\" (UID: \"e8623810-ff6c-4bc5-b8b0-49e963a6aefc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5"
Sep 16 19:34:53.540852 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:53.540746    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-tmp-dir\") pod \"stop-feature-test-kserve-646f6cd48-fbmc5\" (UID: \"e8623810-ff6c-4bc5-b8b0-49e963a6aefc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5"
Sep 16 19:34:53.540852 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:53.540780    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-tls-certs\") pod \"stop-feature-test-kserve-646f6cd48-fbmc5\" (UID: \"e8623810-ff6c-4bc5-b8b0-49e963a6aefc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5"
Sep 16 19:34:53.540852 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:53.540803    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-kserve-provision-location\") pod \"stop-feature-test-kserve-646f6cd48-fbmc5\" (UID: \"e8623810-ff6c-4bc5-b8b0-49e963a6aefc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5"
Sep 16 19:34:53.540985 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:53.540957    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpcbj\" (UniqueName: \"kubernetes.io/projected/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-kube-api-access-lpcbj\") pod \"stop-feature-test-kserve-646f6cd48-fbmc5\" (UID: \"e8623810-ff6c-4bc5-b8b0-49e963a6aefc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5"
Sep 16 19:34:53.541034 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:53.540990    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-dshm\") pod \"stop-feature-test-kserve-646f6cd48-fbmc5\" (UID: \"e8623810-ff6c-4bc5-b8b0-49e963a6aefc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5"
Sep 16 19:34:53.541034 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:53.541018    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-home\") pod \"stop-feature-test-kserve-646f6cd48-fbmc5\" (UID: \"e8623810-ff6c-4bc5-b8b0-49e963a6aefc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5"
Sep 16 19:34:53.541154 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:53.541060    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-model-cache\") pod \"stop-feature-test-kserve-646f6cd48-fbmc5\" (UID: \"e8623810-ff6c-4bc5-b8b0-49e963a6aefc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5"
Sep 16 19:34:53.541209 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:53.541191    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-tmp-dir\") pod \"stop-feature-test-kserve-646f6cd48-fbmc5\" (UID: \"e8623810-ff6c-4bc5-b8b0-49e963a6aefc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5"
Sep 16 19:34:53.541270 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:53.541246    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-kserve-provision-location\") pod \"stop-feature-test-kserve-646f6cd48-fbmc5\" (UID: \"e8623810-ff6c-4bc5-b8b0-49e963a6aefc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5"
Sep 16 19:34:53.541496 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:53.541467    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-model-cache\") pod \"stop-feature-test-kserve-646f6cd48-fbmc5\" (UID: \"e8623810-ff6c-4bc5-b8b0-49e963a6aefc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5"
Sep 16 19:34:53.541630 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:53.541481    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-home\") pod \"stop-feature-test-kserve-646f6cd48-fbmc5\" (UID: \"e8623810-ff6c-4bc5-b8b0-49e963a6aefc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5"
Sep 16 19:34:53.541630 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:53.541578    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-cabundle-cert\") pod \"stop-feature-test-kserve-646f6cd48-fbmc5\" (UID: \"e8623810-ff6c-4bc5-b8b0-49e963a6aefc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5"
Sep 16 19:34:53.543412 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:53.543391    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-dshm\") pod \"stop-feature-test-kserve-646f6cd48-fbmc5\" (UID: \"e8623810-ff6c-4bc5-b8b0-49e963a6aefc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5"
Sep 16 19:34:53.543582 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:53.543562    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-tls-certs\") pod \"stop-feature-test-kserve-646f6cd48-fbmc5\" (UID: \"e8623810-ff6c-4bc5-b8b0-49e963a6aefc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5"
Sep 16 19:34:53.551519 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:53.551500    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpcbj\" (UniqueName: \"kubernetes.io/projected/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-kube-api-access-lpcbj\") pod \"stop-feature-test-kserve-646f6cd48-fbmc5\" (UID: \"e8623810-ff6c-4bc5-b8b0-49e963a6aefc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5"
Sep 16 19:34:53.675821 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:53.675794    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5"
Sep 16 19:34:53.953501 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:53.953418    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5"]
Sep 16 19:34:53.957576 ip-10-0-128-247 kubenswrapper[2577]: W0916 19:34:53.957543    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8623810_ff6c_4bc5_b8b0_49e963a6aefc.slice/crio-ac550b5618fdc2a03597e5085646f94e791932b6061e32b8f07eb27d4b3d437b WatchSource:0}: Error finding container ac550b5618fdc2a03597e5085646f94e791932b6061e32b8f07eb27d4b3d437b: Status 404 returned error can't find the container with id ac550b5618fdc2a03597e5085646f94e791932b6061e32b8f07eb27d4b3d437b
Sep 16 19:34:54.367073 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:54.366988    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5" event={"ID":"e8623810-ff6c-4bc5-b8b0-49e963a6aefc","Type":"ContainerStarted","Data":"b58ed1cefb35c4ce4ea643c272c02ed9a7e293ee8ba4dd7820bf40a792353d12"}
Sep 16 19:34:54.367073 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:54.367041    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5" event={"ID":"e8623810-ff6c-4bc5-b8b0-49e963a6aefc","Type":"ContainerStarted","Data":"ac550b5618fdc2a03597e5085646f94e791932b6061e32b8f07eb27d4b3d437b"}
Sep 16 19:34:56.378156 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:56.378074    2577 generic.go:356] "Generic (PLEG): container finished" podID="e8623810-ff6c-4bc5-b8b0-49e963a6aefc" containerID="b58ed1cefb35c4ce4ea643c272c02ed9a7e293ee8ba4dd7820bf40a792353d12" exitCode=0
Sep 16 19:34:56.378156 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:56.378122    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5" event={"ID":"e8623810-ff6c-4bc5-b8b0-49e963a6aefc","Type":"ContainerDied","Data":"b58ed1cefb35c4ce4ea643c272c02ed9a7e293ee8ba4dd7820bf40a792353d12"}
Sep 16 19:34:57.386854 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:57.386811    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5" event={"ID":"e8623810-ff6c-4bc5-b8b0-49e963a6aefc","Type":"ContainerStarted","Data":"8744c67524cbfd6b9e30a702082f7bd39c4fa80065cfb44381be4a20665243ca"}
Sep 16 19:34:57.412121 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:57.411977    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5" podStartSLOduration=2.369227519 podStartE2EDuration="4.41195812s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.042730601s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:34:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:34:57.411025848 +0000 UTC m=+3254.793915213" watchObservedRunningTime="2026-09-16 19:34:57.41195812 +0000 UTC m=+3254.794847486"
Sep 16 19:34:57.676340 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:34:57.676259    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5"
Sep 16 19:35:03.676283 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:35:03.676241    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5"
Sep 16 19:35:03.677819 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:35:03.677788    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5" podUID="e8623810-ff6c-4bc5-b8b0-49e963a6aefc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 16 19:35:13.676830 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:35:13.676789    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5" podUID="e8623810-ff6c-4bc5-b8b0-49e963a6aefc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 16 19:35:23.676885 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:35:23.676846    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5" podUID="e8623810-ff6c-4bc5-b8b0-49e963a6aefc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 16 19:35:33.676323 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:35:33.676275    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5" podUID="e8623810-ff6c-4bc5-b8b0-49e963a6aefc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 16 19:35:43.676305 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:35:43.676262    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5" podUID="e8623810-ff6c-4bc5-b8b0-49e963a6aefc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 16 19:35:53.676862 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:35:53.676821    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5" podUID="e8623810-ff6c-4bc5-b8b0-49e963a6aefc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 16 19:36:03.676662 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:36:03.676611    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5" podUID="e8623810-ff6c-4bc5-b8b0-49e963a6aefc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 16 19:36:13.676439 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:36:13.676329    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5" podUID="e8623810-ff6c-4bc5-b8b0-49e963a6aefc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 16 19:36:23.676388 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:36:23.676343    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5" podUID="e8623810-ff6c-4bc5-b8b0-49e963a6aefc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 16 19:36:33.676891 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:36:33.676837    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5" podUID="e8623810-ff6c-4bc5-b8b0-49e963a6aefc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 16 19:36:43.687622 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:36:43.687593    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5"
Sep 16 19:36:43.695176 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:36:43.695146    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5"
Sep 16 19:36:45.908338 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:36:45.908305    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5"]
Sep 16 19:37:00.930605 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:00.930550    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5" podUID="e8623810-ff6c-4bc5-b8b0-49e963a6aefc" containerName="main" containerID="cri-o://8744c67524cbfd6b9e30a702082f7bd39c4fa80065cfb44381be4a20665243ca" gracePeriod=45
Sep 16 19:37:01.598729 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:01.598708    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5"
Sep 16 19:37:01.708391 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:01.708332    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-cabundle-cert\") pod \"e8623810-ff6c-4bc5-b8b0-49e963a6aefc\" (UID: \"e8623810-ff6c-4bc5-b8b0-49e963a6aefc\") "
Sep 16 19:37:01.708391 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:01.708379    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-tls-certs\") pod \"e8623810-ff6c-4bc5-b8b0-49e963a6aefc\" (UID: \"e8623810-ff6c-4bc5-b8b0-49e963a6aefc\") "
Sep 16 19:37:01.708655 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:01.708422    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-kserve-provision-location\") pod \"e8623810-ff6c-4bc5-b8b0-49e963a6aefc\" (UID: \"e8623810-ff6c-4bc5-b8b0-49e963a6aefc\") "
Sep 16 19:37:01.708655 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:01.708443    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-home\") pod \"e8623810-ff6c-4bc5-b8b0-49e963a6aefc\" (UID: \"e8623810-ff6c-4bc5-b8b0-49e963a6aefc\") "
Sep 16 19:37:01.708655 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:01.708481    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-model-cache\") pod \"e8623810-ff6c-4bc5-b8b0-49e963a6aefc\" (UID: \"e8623810-ff6c-4bc5-b8b0-49e963a6aefc\") "
Sep 16 19:37:01.708655 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:01.708509    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-tmp-dir\") pod \"e8623810-ff6c-4bc5-b8b0-49e963a6aefc\" (UID: \"e8623810-ff6c-4bc5-b8b0-49e963a6aefc\") "
Sep 16 19:37:01.708655 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:01.708568    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpcbj\" (UniqueName: \"kubernetes.io/projected/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-kube-api-access-lpcbj\") pod \"e8623810-ff6c-4bc5-b8b0-49e963a6aefc\" (UID: \"e8623810-ff6c-4bc5-b8b0-49e963a6aefc\") "
Sep 16 19:37:01.708655 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:01.708593    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-dshm\") pod \"e8623810-ff6c-4bc5-b8b0-49e963a6aefc\" (UID: \"e8623810-ff6c-4bc5-b8b0-49e963a6aefc\") "
Sep 16 19:37:01.708966 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:01.708811    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-model-cache" (OuterVolumeSpecName: "model-cache") pod "e8623810-ff6c-4bc5-b8b0-49e963a6aefc" (UID: "e8623810-ff6c-4bc5-b8b0-49e963a6aefc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:37:01.708966 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:01.708818    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e8623810-ff6c-4bc5-b8b0-49e963a6aefc" (UID: "e8623810-ff6c-4bc5-b8b0-49e963a6aefc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:37:01.709196 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:01.709171    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-home" (OuterVolumeSpecName: "home") pod "e8623810-ff6c-4bc5-b8b0-49e963a6aefc" (UID: "e8623810-ff6c-4bc5-b8b0-49e963a6aefc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:37:01.711104 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:01.711071    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-dshm" (OuterVolumeSpecName: "dshm") pod "e8623810-ff6c-4bc5-b8b0-49e963a6aefc" (UID: "e8623810-ff6c-4bc5-b8b0-49e963a6aefc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:37:01.711218 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:01.711181    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e8623810-ff6c-4bc5-b8b0-49e963a6aefc" (UID: "e8623810-ff6c-4bc5-b8b0-49e963a6aefc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:37:01.711389 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:01.711363    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-kube-api-access-lpcbj" (OuterVolumeSpecName: "kube-api-access-lpcbj") pod "e8623810-ff6c-4bc5-b8b0-49e963a6aefc" (UID: "e8623810-ff6c-4bc5-b8b0-49e963a6aefc"). InnerVolumeSpecName "kube-api-access-lpcbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:37:01.721468 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:01.721440    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e8623810-ff6c-4bc5-b8b0-49e963a6aefc" (UID: "e8623810-ff6c-4bc5-b8b0-49e963a6aefc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:37:01.733910 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:01.733886    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e8623810-ff6c-4bc5-b8b0-49e963a6aefc" (UID: "e8623810-ff6c-4bc5-b8b0-49e963a6aefc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:37:01.809358 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:01.809336    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpcbj\" (UniqueName: \"kubernetes.io/projected/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-kube-api-access-lpcbj\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:37:01.809358 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:01.809357    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-dshm\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:37:01.809471 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:01.809387    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-cabundle-cert\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:37:01.809471 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:01.809396    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-tls-certs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:37:01.809471 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:01.809405    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-kserve-provision-location\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:37:01.809471 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:01.809413    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-home\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:37:01.809471 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:01.809421    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-model-cache\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:37:01.809471 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:01.809428    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8623810-ff6c-4bc5-b8b0-49e963a6aefc-tmp-dir\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:37:01.943989 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:01.943961    2577 generic.go:356] "Generic (PLEG): container finished" podID="e8623810-ff6c-4bc5-b8b0-49e963a6aefc" containerID="8744c67524cbfd6b9e30a702082f7bd39c4fa80065cfb44381be4a20665243ca" exitCode=0
Sep 16 19:37:01.944364 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:01.944015    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5"
Sep 16 19:37:01.944364 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:01.944049    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5" event={"ID":"e8623810-ff6c-4bc5-b8b0-49e963a6aefc","Type":"ContainerDied","Data":"8744c67524cbfd6b9e30a702082f7bd39c4fa80065cfb44381be4a20665243ca"}
Sep 16 19:37:01.944364 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:01.944103    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5" event={"ID":"e8623810-ff6c-4bc5-b8b0-49e963a6aefc","Type":"ContainerDied","Data":"ac550b5618fdc2a03597e5085646f94e791932b6061e32b8f07eb27d4b3d437b"}
Sep 16 19:37:01.944364 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:01.944126    2577 scope.go:117] "RemoveContainer" containerID="8744c67524cbfd6b9e30a702082f7bd39c4fa80065cfb44381be4a20665243ca"
Sep 16 19:37:01.952941 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:01.952927    2577 scope.go:117] "RemoveContainer" containerID="b58ed1cefb35c4ce4ea643c272c02ed9a7e293ee8ba4dd7820bf40a792353d12"
Sep 16 19:37:01.960584 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:01.960558    2577 scope.go:117] "RemoveContainer" containerID="8744c67524cbfd6b9e30a702082f7bd39c4fa80065cfb44381be4a20665243ca"
Sep 16 19:37:01.960853 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:37:01.960834    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8744c67524cbfd6b9e30a702082f7bd39c4fa80065cfb44381be4a20665243ca\": container with ID starting with 8744c67524cbfd6b9e30a702082f7bd39c4fa80065cfb44381be4a20665243ca not found: ID does not exist" containerID="8744c67524cbfd6b9e30a702082f7bd39c4fa80065cfb44381be4a20665243ca"
Sep 16 19:37:01.960908 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:01.960860    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8744c67524cbfd6b9e30a702082f7bd39c4fa80065cfb44381be4a20665243ca"} err="failed to get container status \"8744c67524cbfd6b9e30a702082f7bd39c4fa80065cfb44381be4a20665243ca\": rpc error: code = NotFound desc = could not find container \"8744c67524cbfd6b9e30a702082f7bd39c4fa80065cfb44381be4a20665243ca\": container with ID starting with 8744c67524cbfd6b9e30a702082f7bd39c4fa80065cfb44381be4a20665243ca not found: ID does not exist"
Sep 16 19:37:01.960908 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:01.960877    2577 scope.go:117] "RemoveContainer" containerID="b58ed1cefb35c4ce4ea643c272c02ed9a7e293ee8ba4dd7820bf40a792353d12"
Sep 16 19:37:01.961149 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:37:01.961132    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b58ed1cefb35c4ce4ea643c272c02ed9a7e293ee8ba4dd7820bf40a792353d12\": container with ID starting with b58ed1cefb35c4ce4ea643c272c02ed9a7e293ee8ba4dd7820bf40a792353d12 not found: ID does not exist" containerID="b58ed1cefb35c4ce4ea643c272c02ed9a7e293ee8ba4dd7820bf40a792353d12"
Sep 16 19:37:01.961194 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:01.961155    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b58ed1cefb35c4ce4ea643c272c02ed9a7e293ee8ba4dd7820bf40a792353d12"} err="failed to get container status \"b58ed1cefb35c4ce4ea643c272c02ed9a7e293ee8ba4dd7820bf40a792353d12\": rpc error: code = NotFound desc = could not find container \"b58ed1cefb35c4ce4ea643c272c02ed9a7e293ee8ba4dd7820bf40a792353d12\": container with ID starting with b58ed1cefb35c4ce4ea643c272c02ed9a7e293ee8ba4dd7820bf40a792353d12 not found: ID does not exist"
Sep 16 19:37:01.968620 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:01.968597    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5"]
Sep 16 19:37:01.972935 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:01.972913    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-646f6cd48-fbmc5"]
Sep 16 19:37:02.672357 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:02.672329    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc"]
Sep 16 19:37:02.672679 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:02.672666    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8623810-ff6c-4bc5-b8b0-49e963a6aefc" containerName="main"
Sep 16 19:37:02.672734 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:02.672682    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8623810-ff6c-4bc5-b8b0-49e963a6aefc" containerName="main"
Sep 16 19:37:02.672734 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:02.672699    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8623810-ff6c-4bc5-b8b0-49e963a6aefc" containerName="storage-initializer"
Sep 16 19:37:02.672734 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:02.672705    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8623810-ff6c-4bc5-b8b0-49e963a6aefc" containerName="storage-initializer"
Sep 16 19:37:02.672835 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:02.672764    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8623810-ff6c-4bc5-b8b0-49e963a6aefc" containerName="main"
Sep 16 19:37:02.677472 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:02.677454    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc"
Sep 16 19:37:02.680847 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:02.680821    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"seaweedfs-s3-creds\""
Sep 16 19:37:02.680990 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:02.680889    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 16 19:37:02.680990 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:02.680906    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 16 19:37:02.680990 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:02.680896    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:37:02.680990 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:02.680965    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-2vrc2\""
Sep 16 19:37:02.681232 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:02.681059    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 16 19:37:02.692792 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:02.692768    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc"]
Sep 16 19:37:02.816390 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:02.816366    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0158535f-2b22-4f1a-b96e-60c24c38baff-dshm\") pod \"custom-route-timeout-test-kserve-966768f66-hc8cc\" (UID: \"0158535f-2b22-4f1a-b96e-60c24c38baff\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc"
Sep 16 19:37:02.816555 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:02.816397    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-945sx\" (UniqueName: \"kubernetes.io/projected/0158535f-2b22-4f1a-b96e-60c24c38baff-kube-api-access-945sx\") pod \"custom-route-timeout-test-kserve-966768f66-hc8cc\" (UID: \"0158535f-2b22-4f1a-b96e-60c24c38baff\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc"
Sep 16 19:37:02.816555 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:02.816429    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0158535f-2b22-4f1a-b96e-60c24c38baff-tmp-dir\") pod \"custom-route-timeout-test-kserve-966768f66-hc8cc\" (UID: \"0158535f-2b22-4f1a-b96e-60c24c38baff\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc"
Sep 16 19:37:02.816555 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:02.816512    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0158535f-2b22-4f1a-b96e-60c24c38baff-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-966768f66-hc8cc\" (UID: \"0158535f-2b22-4f1a-b96e-60c24c38baff\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc"
Sep 16 19:37:02.816670 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:02.816568    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0158535f-2b22-4f1a-b96e-60c24c38baff-model-cache\") pod \"custom-route-timeout-test-kserve-966768f66-hc8cc\" (UID: \"0158535f-2b22-4f1a-b96e-60c24c38baff\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc"
Sep 16 19:37:02.816670 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:02.816595    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0158535f-2b22-4f1a-b96e-60c24c38baff-tls-certs\") pod \"custom-route-timeout-test-kserve-966768f66-hc8cc\" (UID: \"0158535f-2b22-4f1a-b96e-60c24c38baff\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc"
Sep 16 19:37:02.816670 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:02.816615    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0158535f-2b22-4f1a-b96e-60c24c38baff-cabundle-cert\") pod \"custom-route-timeout-test-kserve-966768f66-hc8cc\" (UID: \"0158535f-2b22-4f1a-b96e-60c24c38baff\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc"
Sep 16 19:37:02.816768 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:02.816677    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0158535f-2b22-4f1a-b96e-60c24c38baff-home\") pod \"custom-route-timeout-test-kserve-966768f66-hc8cc\" (UID: \"0158535f-2b22-4f1a-b96e-60c24c38baff\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc"
Sep 16 19:37:02.917648 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:02.917614    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0158535f-2b22-4f1a-b96e-60c24c38baff-tmp-dir\") pod \"custom-route-timeout-test-kserve-966768f66-hc8cc\" (UID: \"0158535f-2b22-4f1a-b96e-60c24c38baff\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc"
Sep 16 19:37:02.917822 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:02.917661    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0158535f-2b22-4f1a-b96e-60c24c38baff-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-966768f66-hc8cc\" (UID: \"0158535f-2b22-4f1a-b96e-60c24c38baff\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc"
Sep 16 19:37:02.917822 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:02.917681    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0158535f-2b22-4f1a-b96e-60c24c38baff-model-cache\") pod \"custom-route-timeout-test-kserve-966768f66-hc8cc\" (UID: \"0158535f-2b22-4f1a-b96e-60c24c38baff\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc"
Sep 16 19:37:02.917954 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:02.917863    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0158535f-2b22-4f1a-b96e-60c24c38baff-tls-certs\") pod \"custom-route-timeout-test-kserve-966768f66-hc8cc\" (UID: \"0158535f-2b22-4f1a-b96e-60c24c38baff\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc"
Sep 16 19:37:02.917954 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:02.917899    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0158535f-2b22-4f1a-b96e-60c24c38baff-cabundle-cert\") pod \"custom-route-timeout-test-kserve-966768f66-hc8cc\" (UID: \"0158535f-2b22-4f1a-b96e-60c24c38baff\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc"
Sep 16 19:37:02.917954 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:02.917940    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0158535f-2b22-4f1a-b96e-60c24c38baff-home\") pod \"custom-route-timeout-test-kserve-966768f66-hc8cc\" (UID: \"0158535f-2b22-4f1a-b96e-60c24c38baff\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc"
Sep 16 19:37:02.918116 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:02.917983    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0158535f-2b22-4f1a-b96e-60c24c38baff-dshm\") pod \"custom-route-timeout-test-kserve-966768f66-hc8cc\" (UID: \"0158535f-2b22-4f1a-b96e-60c24c38baff\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc"
Sep 16 19:37:02.918116 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:02.918009    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-945sx\" (UniqueName: \"kubernetes.io/projected/0158535f-2b22-4f1a-b96e-60c24c38baff-kube-api-access-945sx\") pod \"custom-route-timeout-test-kserve-966768f66-hc8cc\" (UID: \"0158535f-2b22-4f1a-b96e-60c24c38baff\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc"
Sep 16 19:37:02.918116 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:02.918078    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0158535f-2b22-4f1a-b96e-60c24c38baff-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-966768f66-hc8cc\" (UID: \"0158535f-2b22-4f1a-b96e-60c24c38baff\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc"
Sep 16 19:37:02.918116 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:02.918101    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0158535f-2b22-4f1a-b96e-60c24c38baff-tmp-dir\") pod \"custom-route-timeout-test-kserve-966768f66-hc8cc\" (UID: \"0158535f-2b22-4f1a-b96e-60c24c38baff\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc"
Sep 16 19:37:02.918322 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:02.918120    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0158535f-2b22-4f1a-b96e-60c24c38baff-model-cache\") pod \"custom-route-timeout-test-kserve-966768f66-hc8cc\" (UID: \"0158535f-2b22-4f1a-b96e-60c24c38baff\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc"
Sep 16 19:37:02.918402 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:02.918382    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0158535f-2b22-4f1a-b96e-60c24c38baff-home\") pod \"custom-route-timeout-test-kserve-966768f66-hc8cc\" (UID: \"0158535f-2b22-4f1a-b96e-60c24c38baff\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc"
Sep 16 19:37:02.918808 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:02.918783    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0158535f-2b22-4f1a-b96e-60c24c38baff-cabundle-cert\") pod \"custom-route-timeout-test-kserve-966768f66-hc8cc\" (UID: \"0158535f-2b22-4f1a-b96e-60c24c38baff\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc"
Sep 16 19:37:02.920460 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:02.920440    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0158535f-2b22-4f1a-b96e-60c24c38baff-dshm\") pod \"custom-route-timeout-test-kserve-966768f66-hc8cc\" (UID: \"0158535f-2b22-4f1a-b96e-60c24c38baff\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc"
Sep 16 19:37:02.920771 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:02.920748    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0158535f-2b22-4f1a-b96e-60c24c38baff-tls-certs\") pod \"custom-route-timeout-test-kserve-966768f66-hc8cc\" (UID: \"0158535f-2b22-4f1a-b96e-60c24c38baff\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc"
Sep 16 19:37:02.932841 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:02.932781    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-945sx\" (UniqueName: \"kubernetes.io/projected/0158535f-2b22-4f1a-b96e-60c24c38baff-kube-api-access-945sx\") pod \"custom-route-timeout-test-kserve-966768f66-hc8cc\" (UID: \"0158535f-2b22-4f1a-b96e-60c24c38baff\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc"
Sep 16 19:37:02.987500 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:02.987475    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc"
Sep 16 19:37:03.160280 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:03.160245    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8623810-ff6c-4bc5-b8b0-49e963a6aefc" path="/var/lib/kubelet/pods/e8623810-ff6c-4bc5-b8b0-49e963a6aefc/volumes"
Sep 16 19:37:03.266281 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:03.266258    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc"]
Sep 16 19:37:03.268454 ip-10-0-128-247 kubenswrapper[2577]: W0916 19:37:03.268430    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0158535f_2b22_4f1a_b96e_60c24c38baff.slice/crio-74fcb5ca7b5ee642e77a9dd30509584f8242fdb2b17b396aecb882c7cae7d9aa WatchSource:0}: Error finding container 74fcb5ca7b5ee642e77a9dd30509584f8242fdb2b17b396aecb882c7cae7d9aa: Status 404 returned error can't find the container with id 74fcb5ca7b5ee642e77a9dd30509584f8242fdb2b17b396aecb882c7cae7d9aa
Sep 16 19:37:03.958091 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:03.958052    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc" event={"ID":"0158535f-2b22-4f1a-b96e-60c24c38baff","Type":"ContainerStarted","Data":"b2098f526fb7db441c4e7a43fe0def96437cabfa03e3bbb8ae83e2ddab3fc5cd"}
Sep 16 19:37:03.958320 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:03.958114    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc" event={"ID":"0158535f-2b22-4f1a-b96e-60c24c38baff","Type":"ContainerStarted","Data":"74fcb5ca7b5ee642e77a9dd30509584f8242fdb2b17b396aecb882c7cae7d9aa"}
Sep 16 19:37:05.968271 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:05.968239    2577 generic.go:356] "Generic (PLEG): container finished" podID="0158535f-2b22-4f1a-b96e-60c24c38baff" containerID="b2098f526fb7db441c4e7a43fe0def96437cabfa03e3bbb8ae83e2ddab3fc5cd" exitCode=0
Sep 16 19:37:05.968675 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:05.968319    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc" event={"ID":"0158535f-2b22-4f1a-b96e-60c24c38baff","Type":"ContainerDied","Data":"b2098f526fb7db441c4e7a43fe0def96437cabfa03e3bbb8ae83e2ddab3fc5cd"}
Sep 16 19:37:06.975491 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:06.975456    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc" event={"ID":"0158535f-2b22-4f1a-b96e-60c24c38baff","Type":"ContainerStarted","Data":"08d93fff60cdf4ef0a87d4530ec394ff757045f377bc01a04f190f1cc9ee012a"}
Sep 16 19:37:06.988286 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:06.988258    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc"
Sep 16 19:37:07.000466 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:07.000421    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc" podStartSLOduration=2.9449324040000002 podStartE2EDuration="5.000403954s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.05547155s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:37:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:37:06.998198569 +0000 UTC m=+3384.381087934" watchObservedRunningTime="2026-09-16 19:37:07.000403954 +0000 UTC m=+3384.383293322"
Sep 16 19:37:12.988030 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:12.987978    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc"
Sep 16 19:37:12.989646 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:12.989616    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc" podUID="0158535f-2b22-4f1a-b96e-60c24c38baff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 16 19:37:22.988678 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:22.988631    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc" podUID="0158535f-2b22-4f1a-b96e-60c24c38baff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 16 19:37:32.988089 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:32.988048    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc" podUID="0158535f-2b22-4f1a-b96e-60c24c38baff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 16 19:37:42.988366 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:42.988280    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc" podUID="0158535f-2b22-4f1a-b96e-60c24c38baff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 16 19:37:52.988462 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:37:52.988414    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc" podUID="0158535f-2b22-4f1a-b96e-60c24c38baff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 16 19:38:02.988211 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:38:02.988168    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc" podUID="0158535f-2b22-4f1a-b96e-60c24c38baff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 16 19:38:12.988786 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:38:12.988732    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc" podUID="0158535f-2b22-4f1a-b96e-60c24c38baff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 16 19:38:22.988336 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:38:22.988288    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc" podUID="0158535f-2b22-4f1a-b96e-60c24c38baff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 16 19:38:32.988236 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:38:32.988187    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc" podUID="0158535f-2b22-4f1a-b96e-60c24c38baff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 16 19:38:42.988298 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:38:42.988238    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc" podUID="0158535f-2b22-4f1a-b96e-60c24c38baff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 16 19:38:52.997972 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:38:52.997938    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc"
Sep 16 19:38:53.005734 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:38:53.005710    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc"
Sep 16 19:38:58.392666 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:38:58.392627    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc"]
Sep 16 19:39:13.415873 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:39:13.415767    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc" podUID="0158535f-2b22-4f1a-b96e-60c24c38baff" containerName="main" containerID="cri-o://08d93fff60cdf4ef0a87d4530ec394ff757045f377bc01a04f190f1cc9ee012a" gracePeriod=45
Sep 16 19:39:13.910427 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:39:13.910396    2577 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0158535f_2b22_4f1a_b96e_60c24c38baff.slice/crio-08d93fff60cdf4ef0a87d4530ec394ff757045f377bc01a04f190f1cc9ee012a.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 19:39:14.067171 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:39:14.067150    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc"
Sep 16 19:39:14.091570 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:39:14.091513    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-945sx\" (UniqueName: \"kubernetes.io/projected/0158535f-2b22-4f1a-b96e-60c24c38baff-kube-api-access-945sx\") pod \"0158535f-2b22-4f1a-b96e-60c24c38baff\" (UID: \"0158535f-2b22-4f1a-b96e-60c24c38baff\") "
Sep 16 19:39:14.091700 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:39:14.091585    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0158535f-2b22-4f1a-b96e-60c24c38baff-home\") pod \"0158535f-2b22-4f1a-b96e-60c24c38baff\" (UID: \"0158535f-2b22-4f1a-b96e-60c24c38baff\") "
Sep 16 19:39:14.091700 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:39:14.091633    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0158535f-2b22-4f1a-b96e-60c24c38baff-dshm\") pod \"0158535f-2b22-4f1a-b96e-60c24c38baff\" (UID: \"0158535f-2b22-4f1a-b96e-60c24c38baff\") "
Sep 16 19:39:14.091700 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:39:14.091670    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0158535f-2b22-4f1a-b96e-60c24c38baff-tmp-dir\") pod \"0158535f-2b22-4f1a-b96e-60c24c38baff\" (UID: \"0158535f-2b22-4f1a-b96e-60c24c38baff\") "
Sep 16 19:39:14.091890 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:39:14.091699    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0158535f-2b22-4f1a-b96e-60c24c38baff-kserve-provision-location\") pod \"0158535f-2b22-4f1a-b96e-60c24c38baff\" (UID: \"0158535f-2b22-4f1a-b96e-60c24c38baff\") "
Sep 16 19:39:14.091890 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:39:14.091730    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0158535f-2b22-4f1a-b96e-60c24c38baff-cabundle-cert\") pod \"0158535f-2b22-4f1a-b96e-60c24c38baff\" (UID: \"0158535f-2b22-4f1a-b96e-60c24c38baff\") "
Sep 16 19:39:14.091890 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:39:14.091809    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0158535f-2b22-4f1a-b96e-60c24c38baff-tls-certs\") pod \"0158535f-2b22-4f1a-b96e-60c24c38baff\" (UID: \"0158535f-2b22-4f1a-b96e-60c24c38baff\") "
Sep 16 19:39:14.091890 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:39:14.091841    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0158535f-2b22-4f1a-b96e-60c24c38baff-model-cache\") pod \"0158535f-2b22-4f1a-b96e-60c24c38baff\" (UID: \"0158535f-2b22-4f1a-b96e-60c24c38baff\") "
Sep 16 19:39:14.092349 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:39:14.092320    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0158535f-2b22-4f1a-b96e-60c24c38baff-home" (OuterVolumeSpecName: "home") pod "0158535f-2b22-4f1a-b96e-60c24c38baff" (UID: "0158535f-2b22-4f1a-b96e-60c24c38baff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:39:14.092684 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:39:14.092332    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0158535f-2b22-4f1a-b96e-60c24c38baff-model-cache" (OuterVolumeSpecName: "model-cache") pod "0158535f-2b22-4f1a-b96e-60c24c38baff" (UID: "0158535f-2b22-4f1a-b96e-60c24c38baff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:39:14.093178 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:39:14.093084    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0158535f-2b22-4f1a-b96e-60c24c38baff-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0158535f-2b22-4f1a-b96e-60c24c38baff" (UID: "0158535f-2b22-4f1a-b96e-60c24c38baff"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:39:14.094856 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:39:14.094827    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0158535f-2b22-4f1a-b96e-60c24c38baff-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0158535f-2b22-4f1a-b96e-60c24c38baff" (UID: "0158535f-2b22-4f1a-b96e-60c24c38baff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:39:14.094856 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:39:14.094834    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0158535f-2b22-4f1a-b96e-60c24c38baff-kube-api-access-945sx" (OuterVolumeSpecName: "kube-api-access-945sx") pod "0158535f-2b22-4f1a-b96e-60c24c38baff" (UID: "0158535f-2b22-4f1a-b96e-60c24c38baff"). InnerVolumeSpecName "kube-api-access-945sx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:39:14.095349 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:39:14.095324    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0158535f-2b22-4f1a-b96e-60c24c38baff-dshm" (OuterVolumeSpecName: "dshm") pod "0158535f-2b22-4f1a-b96e-60c24c38baff" (UID: "0158535f-2b22-4f1a-b96e-60c24c38baff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:39:14.110841 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:39:14.110811    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0158535f-2b22-4f1a-b96e-60c24c38baff-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0158535f-2b22-4f1a-b96e-60c24c38baff" (UID: "0158535f-2b22-4f1a-b96e-60c24c38baff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:39:14.123519 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:39:14.123493    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0158535f-2b22-4f1a-b96e-60c24c38baff-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0158535f-2b22-4f1a-b96e-60c24c38baff" (UID: "0158535f-2b22-4f1a-b96e-60c24c38baff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:39:14.192937 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:39:14.192866    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0158535f-2b22-4f1a-b96e-60c24c38baff-model-cache\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:39:14.192937 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:39:14.192896    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-945sx\" (UniqueName: \"kubernetes.io/projected/0158535f-2b22-4f1a-b96e-60c24c38baff-kube-api-access-945sx\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:39:14.192937 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:39:14.192907    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0158535f-2b22-4f1a-b96e-60c24c38baff-home\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:39:14.192937 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:39:14.192916    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0158535f-2b22-4f1a-b96e-60c24c38baff-dshm\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:39:14.192937 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:39:14.192924    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0158535f-2b22-4f1a-b96e-60c24c38baff-tmp-dir\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:39:14.192937 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:39:14.192933    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0158535f-2b22-4f1a-b96e-60c24c38baff-kserve-provision-location\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:39:14.192937 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:39:14.192941    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0158535f-2b22-4f1a-b96e-60c24c38baff-cabundle-cert\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:39:14.193221 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:39:14.192949    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0158535f-2b22-4f1a-b96e-60c24c38baff-tls-certs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:39:14.539860 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:39:14.539769    2577 generic.go:356] "Generic (PLEG): container finished" podID="0158535f-2b22-4f1a-b96e-60c24c38baff" containerID="08d93fff60cdf4ef0a87d4530ec394ff757045f377bc01a04f190f1cc9ee012a" exitCode=0
Sep 16 19:39:14.539860 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:39:14.539826    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc"
Sep 16 19:39:14.539860 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:39:14.539833    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc" event={"ID":"0158535f-2b22-4f1a-b96e-60c24c38baff","Type":"ContainerDied","Data":"08d93fff60cdf4ef0a87d4530ec394ff757045f377bc01a04f190f1cc9ee012a"}
Sep 16 19:39:14.540333 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:39:14.539878    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc" event={"ID":"0158535f-2b22-4f1a-b96e-60c24c38baff","Type":"ContainerDied","Data":"74fcb5ca7b5ee642e77a9dd30509584f8242fdb2b17b396aecb882c7cae7d9aa"}
Sep 16 19:39:14.540333 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:39:14.539900    2577 scope.go:117] "RemoveContainer" containerID="08d93fff60cdf4ef0a87d4530ec394ff757045f377bc01a04f190f1cc9ee012a"
Sep 16 19:39:14.549238 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:39:14.549223    2577 scope.go:117] "RemoveContainer" containerID="b2098f526fb7db441c4e7a43fe0def96437cabfa03e3bbb8ae83e2ddab3fc5cd"
Sep 16 19:39:14.556728 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:39:14.556709    2577 scope.go:117] "RemoveContainer" containerID="08d93fff60cdf4ef0a87d4530ec394ff757045f377bc01a04f190f1cc9ee012a"
Sep 16 19:39:14.556990 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:39:14.556975    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08d93fff60cdf4ef0a87d4530ec394ff757045f377bc01a04f190f1cc9ee012a\": container with ID starting with 08d93fff60cdf4ef0a87d4530ec394ff757045f377bc01a04f190f1cc9ee012a not found: ID does not exist" containerID="08d93fff60cdf4ef0a87d4530ec394ff757045f377bc01a04f190f1cc9ee012a"
Sep 16 19:39:14.557034 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:39:14.556997    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08d93fff60cdf4ef0a87d4530ec394ff757045f377bc01a04f190f1cc9ee012a"} err="failed to get container status \"08d93fff60cdf4ef0a87d4530ec394ff757045f377bc01a04f190f1cc9ee012a\": rpc error: code = NotFound desc = could not find container \"08d93fff60cdf4ef0a87d4530ec394ff757045f377bc01a04f190f1cc9ee012a\": container with ID starting with 08d93fff60cdf4ef0a87d4530ec394ff757045f377bc01a04f190f1cc9ee012a not found: ID does not exist"
Sep 16 19:39:14.557034 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:39:14.557011    2577 scope.go:117] "RemoveContainer" containerID="b2098f526fb7db441c4e7a43fe0def96437cabfa03e3bbb8ae83e2ddab3fc5cd"
Sep 16 19:39:14.557245 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:39:14.557226    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2098f526fb7db441c4e7a43fe0def96437cabfa03e3bbb8ae83e2ddab3fc5cd\": container with ID starting with b2098f526fb7db441c4e7a43fe0def96437cabfa03e3bbb8ae83e2ddab3fc5cd not found: ID does not exist" containerID="b2098f526fb7db441c4e7a43fe0def96437cabfa03e3bbb8ae83e2ddab3fc5cd"
Sep 16 19:39:14.557287 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:39:14.557252    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2098f526fb7db441c4e7a43fe0def96437cabfa03e3bbb8ae83e2ddab3fc5cd"} err="failed to get container status \"b2098f526fb7db441c4e7a43fe0def96437cabfa03e3bbb8ae83e2ddab3fc5cd\": rpc error: code = NotFound desc = could not find container \"b2098f526fb7db441c4e7a43fe0def96437cabfa03e3bbb8ae83e2ddab3fc5cd\": container with ID starting with b2098f526fb7db441c4e7a43fe0def96437cabfa03e3bbb8ae83e2ddab3fc5cd not found: ID does not exist"
Sep 16 19:39:14.565042 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:39:14.565021    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc"]
Sep 16 19:39:14.570480 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:39:14.570459    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc"]
Sep 16 19:39:14.999326 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:39:14.999288    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-966768f66-hc8cc" podUID="0158535f-2b22-4f1a-b96e-60c24c38baff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 16 19:39:15.155026 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:39:15.154995    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0158535f-2b22-4f1a-b96e-60c24c38baff" path="/var/lib/kubelet/pods/0158535f-2b22-4f1a-b96e-60c24c38baff/volumes"
Sep 16 19:40:02.073193 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:02.073154    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5"]
Sep 16 19:40:02.073688 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:02.073671    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0158535f-2b22-4f1a-b96e-60c24c38baff" containerName="storage-initializer"
Sep 16 19:40:02.073738 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:02.073691    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0158535f-2b22-4f1a-b96e-60c24c38baff" containerName="storage-initializer"
Sep 16 19:40:02.073738 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:02.073725    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0158535f-2b22-4f1a-b96e-60c24c38baff" containerName="main"
Sep 16 19:40:02.073738 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:02.073733    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0158535f-2b22-4f1a-b96e-60c24c38baff" containerName="main"
Sep 16 19:40:02.073838 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:02.073806    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0158535f-2b22-4f1a-b96e-60c24c38baff" containerName="main"
Sep 16 19:40:02.077100 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:02.077080    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5"
Sep 16 19:40:02.079652 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:02.079619    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\""
Sep 16 19:40:02.080014 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:02.079992    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:40:02.080142 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:02.080020    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 16 19:40:02.080450 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:02.080431    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 16 19:40:02.080577 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:02.080463    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 16 19:40:02.080577 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:02.080491    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-fmg6t\""
Sep 16 19:40:02.090712 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:02.090690    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5"]
Sep 16 19:40:02.144983 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:02.144959    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-model-cache\") pod \"router-with-refs-test-kserve-6dbd8fbf4-w2mq5\" (UID: \"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5"
Sep 16 19:40:02.145085 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:02.144987    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-kserve-provision-location\") pod \"router-with-refs-test-kserve-6dbd8fbf4-w2mq5\" (UID: \"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5"
Sep 16 19:40:02.145085 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:02.145005    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk8jm\" (UniqueName: \"kubernetes.io/projected/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-kube-api-access-kk8jm\") pod \"router-with-refs-test-kserve-6dbd8fbf4-w2mq5\" (UID: \"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5"
Sep 16 19:40:02.145085 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:02.145031    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-tls-certs\") pod \"router-with-refs-test-kserve-6dbd8fbf4-w2mq5\" (UID: \"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5"
Sep 16 19:40:02.145085 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:02.145083    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-cabundle-cert\") pod \"router-with-refs-test-kserve-6dbd8fbf4-w2mq5\" (UID: \"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5"
Sep 16 19:40:02.145237 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:02.145099    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-dshm\") pod \"router-with-refs-test-kserve-6dbd8fbf4-w2mq5\" (UID: \"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5"
Sep 16 19:40:02.145237 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:02.145131    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-tmp-dir\") pod \"router-with-refs-test-kserve-6dbd8fbf4-w2mq5\" (UID: \"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5"
Sep 16 19:40:02.145237 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:02.145191    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-home\") pod \"router-with-refs-test-kserve-6dbd8fbf4-w2mq5\" (UID: \"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5"
Sep 16 19:40:02.246458 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:02.246435    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-tls-certs\") pod \"router-with-refs-test-kserve-6dbd8fbf4-w2mq5\" (UID: \"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5"
Sep 16 19:40:02.246592 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:02.246467    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-cabundle-cert\") pod \"router-with-refs-test-kserve-6dbd8fbf4-w2mq5\" (UID: \"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5"
Sep 16 19:40:02.246592 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:02.246486    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-dshm\") pod \"router-with-refs-test-kserve-6dbd8fbf4-w2mq5\" (UID: \"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5"
Sep 16 19:40:02.246592 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:02.246509    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-tmp-dir\") pod \"router-with-refs-test-kserve-6dbd8fbf4-w2mq5\" (UID: \"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5"
Sep 16 19:40:02.246592 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:02.246590    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-home\") pod \"router-with-refs-test-kserve-6dbd8fbf4-w2mq5\" (UID: \"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5"
Sep 16 19:40:02.246802 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:02.246644    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-model-cache\") pod \"router-with-refs-test-kserve-6dbd8fbf4-w2mq5\" (UID: \"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5"
Sep 16 19:40:02.246802 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:02.246667    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-kserve-provision-location\") pod \"router-with-refs-test-kserve-6dbd8fbf4-w2mq5\" (UID: \"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5"
Sep 16 19:40:02.246802 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:02.246695    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk8jm\" (UniqueName: \"kubernetes.io/projected/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-kube-api-access-kk8jm\") pod \"router-with-refs-test-kserve-6dbd8fbf4-w2mq5\" (UID: \"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5"
Sep 16 19:40:02.247082 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:02.247047    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-home\") pod \"router-with-refs-test-kserve-6dbd8fbf4-w2mq5\" (UID: \"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5"
Sep 16 19:40:02.247082 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:02.247076    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-tmp-dir\") pod \"router-with-refs-test-kserve-6dbd8fbf4-w2mq5\" (UID: \"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5"
Sep 16 19:40:02.247262 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:02.247120    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-kserve-provision-location\") pod \"router-with-refs-test-kserve-6dbd8fbf4-w2mq5\" (UID: \"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5"
Sep 16 19:40:02.247325 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:02.247268    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-cabundle-cert\") pod \"router-with-refs-test-kserve-6dbd8fbf4-w2mq5\" (UID: \"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5"
Sep 16 19:40:02.247672 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:02.247649    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-model-cache\") pod \"router-with-refs-test-kserve-6dbd8fbf4-w2mq5\" (UID: \"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5"
Sep 16 19:40:02.249106 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:02.249073    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-dshm\") pod \"router-with-refs-test-kserve-6dbd8fbf4-w2mq5\" (UID: \"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5"
Sep 16 19:40:02.249384 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:02.249366    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-tls-certs\") pod \"router-with-refs-test-kserve-6dbd8fbf4-w2mq5\" (UID: \"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5"
Sep 16 19:40:02.256603 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:02.256578    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk8jm\" (UniqueName: \"kubernetes.io/projected/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-kube-api-access-kk8jm\") pod \"router-with-refs-test-kserve-6dbd8fbf4-w2mq5\" (UID: \"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5"
Sep 16 19:40:02.389968 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:02.389946    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5"
Sep 16 19:40:02.669934 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:02.669864    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5"]
Sep 16 19:40:02.673113 ip-10-0-128-247 kubenswrapper[2577]: W0916 19:40:02.673086    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2764bf1f_ba79_4d9e_a542_e6d6f1b52f8d.slice/crio-98d4e0c6361d16be046cadb175a5cb9c2370f736ef76e6ca6186f3308f228f89 WatchSource:0}: Error finding container 98d4e0c6361d16be046cadb175a5cb9c2370f736ef76e6ca6186f3308f228f89: Status 404 returned error can't find the container with id 98d4e0c6361d16be046cadb175a5cb9c2370f736ef76e6ca6186f3308f228f89
Sep 16 19:40:02.734202 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:02.734177    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5" event={"ID":"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d","Type":"ContainerStarted","Data":"98d4e0c6361d16be046cadb175a5cb9c2370f736ef76e6ca6186f3308f228f89"}
Sep 16 19:40:03.740826 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:03.740780    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5" event={"ID":"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d","Type":"ContainerStarted","Data":"54c324272bb9bb09ca5ab1826016c2c94fa594b9cfc3c3d01b0383edbae00967"}
Sep 16 19:40:05.751208 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:05.751171    2577 generic.go:356] "Generic (PLEG): container finished" podID="2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d" containerID="54c324272bb9bb09ca5ab1826016c2c94fa594b9cfc3c3d01b0383edbae00967" exitCode=0
Sep 16 19:40:05.751612 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:05.751242    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5" event={"ID":"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d","Type":"ContainerDied","Data":"54c324272bb9bb09ca5ab1826016c2c94fa594b9cfc3c3d01b0383edbae00967"}
Sep 16 19:40:06.757631 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:06.757596    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5" event={"ID":"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d","Type":"ContainerStarted","Data":"492f9c4676a1d244517938d60aa925cec91fd9f646fe5fb8c315902a970c0f75"}
Sep 16 19:40:06.787872 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:06.787822    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5" podStartSLOduration=2.7411644429999997 podStartE2EDuration="4.787807094s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.046642651s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:40:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:40:06.786136722 +0000 UTC m=+3564.169026087" watchObservedRunningTime="2026-09-16 19:40:06.787807094 +0000 UTC m=+3564.170696458"
Sep 16 19:40:07.390448 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:07.390413    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5"
Sep 16 19:40:12.390493 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:12.390456    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5"
Sep 16 19:40:12.392151 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:12.392116    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5" podUID="2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 16 19:40:22.391262 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:22.391221    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5" podUID="2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 16 19:40:27.446028 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:27.445996    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm"]
Sep 16 19:40:27.449338 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:27.449316    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm"
Sep 16 19:40:27.452590 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:27.452521    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Sep 16 19:40:27.452713 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:27.452617    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:40:27.452882 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:27.452854    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-vmhtc\""
Sep 16 19:40:27.453002 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:27.452883    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Sep 16 19:40:27.453603 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:27.453123    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Sep 16 19:40:27.453603 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:27.453279    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Sep 16 19:40:27.464222 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:27.464200    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm"]
Sep 16 19:40:27.553091 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:27.553053    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2179757f-1e83-45dd-8b65-19b57e471349-dshm\") pod \"lora-multiple-adapters-test-kserve-db6bd4769-w2dhm\" (UID: \"2179757f-1e83-45dd-8b65-19b57e471349\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm"
Sep 16 19:40:27.553267 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:27.553116    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2179757f-1e83-45dd-8b65-19b57e471349-home\") pod \"lora-multiple-adapters-test-kserve-db6bd4769-w2dhm\" (UID: \"2179757f-1e83-45dd-8b65-19b57e471349\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm"
Sep 16 19:40:27.553267 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:27.553185    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2179757f-1e83-45dd-8b65-19b57e471349-tls-certs\") pod \"lora-multiple-adapters-test-kserve-db6bd4769-w2dhm\" (UID: \"2179757f-1e83-45dd-8b65-19b57e471349\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm"
Sep 16 19:40:27.553267 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:27.553218    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q46vb\" (UniqueName: \"kubernetes.io/projected/2179757f-1e83-45dd-8b65-19b57e471349-kube-api-access-q46vb\") pod \"lora-multiple-adapters-test-kserve-db6bd4769-w2dhm\" (UID: \"2179757f-1e83-45dd-8b65-19b57e471349\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm"
Sep 16 19:40:27.553267 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:27.553260    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2179757f-1e83-45dd-8b65-19b57e471349-model-cache\") pod \"lora-multiple-adapters-test-kserve-db6bd4769-w2dhm\" (UID: \"2179757f-1e83-45dd-8b65-19b57e471349\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm"
Sep 16 19:40:27.553412 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:27.553279    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2179757f-1e83-45dd-8b65-19b57e471349-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-db6bd4769-w2dhm\" (UID: \"2179757f-1e83-45dd-8b65-19b57e471349\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm"
Sep 16 19:40:27.553412 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:27.553330    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2179757f-1e83-45dd-8b65-19b57e471349-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-db6bd4769-w2dhm\" (UID: \"2179757f-1e83-45dd-8b65-19b57e471349\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm"
Sep 16 19:40:27.553412 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:27.553353    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2179757f-1e83-45dd-8b65-19b57e471349-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-db6bd4769-w2dhm\" (UID: \"2179757f-1e83-45dd-8b65-19b57e471349\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm"
Sep 16 19:40:27.654568 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:27.654496    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2179757f-1e83-45dd-8b65-19b57e471349-home\") pod \"lora-multiple-adapters-test-kserve-db6bd4769-w2dhm\" (UID: \"2179757f-1e83-45dd-8b65-19b57e471349\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm"
Sep 16 19:40:27.654744 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:27.654597    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2179757f-1e83-45dd-8b65-19b57e471349-tls-certs\") pod \"lora-multiple-adapters-test-kserve-db6bd4769-w2dhm\" (UID: \"2179757f-1e83-45dd-8b65-19b57e471349\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm"
Sep 16 19:40:27.654744 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:27.654621    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q46vb\" (UniqueName: \"kubernetes.io/projected/2179757f-1e83-45dd-8b65-19b57e471349-kube-api-access-q46vb\") pod \"lora-multiple-adapters-test-kserve-db6bd4769-w2dhm\" (UID: \"2179757f-1e83-45dd-8b65-19b57e471349\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm"
Sep 16 19:40:27.654744 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:27.654657    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2179757f-1e83-45dd-8b65-19b57e471349-model-cache\") pod \"lora-multiple-adapters-test-kserve-db6bd4769-w2dhm\" (UID: \"2179757f-1e83-45dd-8b65-19b57e471349\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm"
Sep 16 19:40:27.654744 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:27.654681    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2179757f-1e83-45dd-8b65-19b57e471349-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-db6bd4769-w2dhm\" (UID: \"2179757f-1e83-45dd-8b65-19b57e471349\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm"
Sep 16 19:40:27.654963 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:27.654815    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2179757f-1e83-45dd-8b65-19b57e471349-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-db6bd4769-w2dhm\" (UID: \"2179757f-1e83-45dd-8b65-19b57e471349\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm"
Sep 16 19:40:27.654963 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:27.654876    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2179757f-1e83-45dd-8b65-19b57e471349-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-db6bd4769-w2dhm\" (UID: \"2179757f-1e83-45dd-8b65-19b57e471349\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm"
Sep 16 19:40:27.655071 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:27.654970    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2179757f-1e83-45dd-8b65-19b57e471349-home\") pod \"lora-multiple-adapters-test-kserve-db6bd4769-w2dhm\" (UID: \"2179757f-1e83-45dd-8b65-19b57e471349\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm"
Sep 16 19:40:27.655071 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:27.654975    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2179757f-1e83-45dd-8b65-19b57e471349-dshm\") pod \"lora-multiple-adapters-test-kserve-db6bd4769-w2dhm\" (UID: \"2179757f-1e83-45dd-8b65-19b57e471349\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm"
Sep 16 19:40:27.655071 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:27.655002    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2179757f-1e83-45dd-8b65-19b57e471349-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-db6bd4769-w2dhm\" (UID: \"2179757f-1e83-45dd-8b65-19b57e471349\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm"
Sep 16 19:40:27.655071 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:27.655059    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2179757f-1e83-45dd-8b65-19b57e471349-model-cache\") pod \"lora-multiple-adapters-test-kserve-db6bd4769-w2dhm\" (UID: \"2179757f-1e83-45dd-8b65-19b57e471349\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm"
Sep 16 19:40:27.655258 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:27.655230    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2179757f-1e83-45dd-8b65-19b57e471349-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-db6bd4769-w2dhm\" (UID: \"2179757f-1e83-45dd-8b65-19b57e471349\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm"
Sep 16 19:40:27.655693 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:27.655627    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2179757f-1e83-45dd-8b65-19b57e471349-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-db6bd4769-w2dhm\" (UID: \"2179757f-1e83-45dd-8b65-19b57e471349\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm"
Sep 16 19:40:27.657252 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:27.657223    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2179757f-1e83-45dd-8b65-19b57e471349-dshm\") pod \"lora-multiple-adapters-test-kserve-db6bd4769-w2dhm\" (UID: \"2179757f-1e83-45dd-8b65-19b57e471349\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm"
Sep 16 19:40:27.657402 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:27.657383    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2179757f-1e83-45dd-8b65-19b57e471349-tls-certs\") pod \"lora-multiple-adapters-test-kserve-db6bd4769-w2dhm\" (UID: \"2179757f-1e83-45dd-8b65-19b57e471349\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm"
Sep 16 19:40:27.666283 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:27.666262    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q46vb\" (UniqueName: \"kubernetes.io/projected/2179757f-1e83-45dd-8b65-19b57e471349-kube-api-access-q46vb\") pod \"lora-multiple-adapters-test-kserve-db6bd4769-w2dhm\" (UID: \"2179757f-1e83-45dd-8b65-19b57e471349\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm"
Sep 16 19:40:27.762017 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:27.761947    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm"
Sep 16 19:40:28.043825 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:28.043708    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm"]
Sep 16 19:40:28.046581 ip-10-0-128-247 kubenswrapper[2577]: W0916 19:40:28.046516    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2179757f_1e83_45dd_8b65_19b57e471349.slice/crio-dfd6f7c81275175639972670f11909ed31b46806d2c59ba6e950c97b7bd06674 WatchSource:0}: Error finding container dfd6f7c81275175639972670f11909ed31b46806d2c59ba6e950c97b7bd06674: Status 404 returned error can't find the container with id dfd6f7c81275175639972670f11909ed31b46806d2c59ba6e950c97b7bd06674
Sep 16 19:40:28.857275 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:28.857230    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm" event={"ID":"2179757f-1e83-45dd-8b65-19b57e471349","Type":"ContainerStarted","Data":"0151baffa6e316cdfbb2637325d2c8bf0b0b6f658aa19698cdd5f38bee98fd14"}
Sep 16 19:40:28.857698 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:28.857344    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm" event={"ID":"2179757f-1e83-45dd-8b65-19b57e471349","Type":"ContainerStarted","Data":"dfd6f7c81275175639972670f11909ed31b46806d2c59ba6e950c97b7bd06674"}
Sep 16 19:40:31.872605 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:31.872569    2577 generic.go:356] "Generic (PLEG): container finished" podID="2179757f-1e83-45dd-8b65-19b57e471349" containerID="0151baffa6e316cdfbb2637325d2c8bf0b0b6f658aa19698cdd5f38bee98fd14" exitCode=0
Sep 16 19:40:31.872605 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:31.872615    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm" event={"ID":"2179757f-1e83-45dd-8b65-19b57e471349","Type":"ContainerDied","Data":"0151baffa6e316cdfbb2637325d2c8bf0b0b6f658aa19698cdd5f38bee98fd14"}
Sep 16 19:40:32.390600 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:32.390517    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5" podUID="2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 16 19:40:32.880194 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:32.880099    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm" event={"ID":"2179757f-1e83-45dd-8b65-19b57e471349","Type":"ContainerStarted","Data":"4251dd106a9ec266b8a5bfca2d3afbaf36f4b2ff2f47316f4ed497e41c32ee18"}
Sep 16 19:40:32.906827 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:32.906775    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm" podStartSLOduration=2.972424298 podStartE2EDuration="5.906759425s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.934335127s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:40:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:40:32.904155631 +0000 UTC m=+3590.287044999" watchObservedRunningTime="2026-09-16 19:40:32.906759425 +0000 UTC m=+3590.289648790"
Sep 16 19:40:33.762953 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:33.762919    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm"
Sep 16 19:40:37.762761 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:37.762659    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm"
Sep 16 19:40:37.764261 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:37.764224    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm" podUID="2179757f-1e83-45dd-8b65-19b57e471349" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 19:40:42.390631 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:42.390581    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5" podUID="2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 16 19:40:47.762971 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:47.762922    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm" podUID="2179757f-1e83-45dd-8b65-19b57e471349" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 19:40:52.390713 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:52.390661    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5" podUID="2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 16 19:40:57.762884 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:40:57.762835    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm" podUID="2179757f-1e83-45dd-8b65-19b57e471349" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 19:41:02.390395 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:41:02.390337    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5" podUID="2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 16 19:41:07.762646 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:41:07.762590    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm" podUID="2179757f-1e83-45dd-8b65-19b57e471349" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 19:41:12.390919 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:41:12.390869    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5" podUID="2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 16 19:41:17.763299 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:41:17.763252    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm" podUID="2179757f-1e83-45dd-8b65-19b57e471349" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 19:41:22.390900 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:41:22.390844    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5" podUID="2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 16 19:41:27.762739 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:41:27.762698    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm" podUID="2179757f-1e83-45dd-8b65-19b57e471349" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 19:41:32.390612 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:41:32.390564    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5" podUID="2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 16 19:41:37.762399 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:41:37.762352    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm" podUID="2179757f-1e83-45dd-8b65-19b57e471349" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 19:41:42.390417 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:41:42.390362    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5" podUID="2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 16 19:41:47.772502 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:41:47.772471    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm"
Sep 16 19:41:47.780324 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:41:47.780301    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm"
Sep 16 19:41:52.390456 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:41:52.390412    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5" podUID="2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 16 19:42:02.390964 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:02.390912    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5" podUID="2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 16 19:42:12.400886 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:12.400796    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5"
Sep 16 19:42:12.408730 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:12.408703    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5"
Sep 16 19:42:19.771160 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:19.771124    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5"]
Sep 16 19:42:21.035926 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:21.035898    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-db6bd4769-w2dhm_2179757f-1e83-45dd-8b65-19b57e471349/storage-initializer/0.log"
Sep 16 19:42:21.046758 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:21.046735    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-db6bd4769-w2dhm_2179757f-1e83-45dd-8b65-19b57e471349/main/0.log"
Sep 16 19:42:22.561177 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:22.561146    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm"]
Sep 16 19:42:34.794050 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:34.794009    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5" podUID="2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d" containerName="main" containerID="cri-o://492f9c4676a1d244517938d60aa925cec91fd9f646fe5fb8c315902a970c0f75" gracePeriod=45
Sep 16 19:42:35.401932 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:35.401890    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5" podUID="2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 16 19:42:35.466034 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:35.466004    2577 generic.go:356] "Generic (PLEG): container finished" podID="2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d" containerID="492f9c4676a1d244517938d60aa925cec91fd9f646fe5fb8c315902a970c0f75" exitCode=0
Sep 16 19:42:35.466201 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:35.466060    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5" event={"ID":"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d","Type":"ContainerDied","Data":"492f9c4676a1d244517938d60aa925cec91fd9f646fe5fb8c315902a970c0f75"}
Sep 16 19:42:35.545042 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:35.545018    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5"
Sep 16 19:42:35.681830 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:35.681800    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-home\") pod \"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d\" (UID: \"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d\") "
Sep 16 19:42:35.681998 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:35.681888    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-tls-certs\") pod \"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d\" (UID: \"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d\") "
Sep 16 19:42:35.681998 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:35.681905    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-cabundle-cert\") pod \"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d\" (UID: \"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d\") "
Sep 16 19:42:35.681998 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:35.681922    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-dshm\") pod \"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d\" (UID: \"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d\") "
Sep 16 19:42:35.681998 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:35.681970    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-model-cache\") pod \"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d\" (UID: \"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d\") "
Sep 16 19:42:35.681998 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:35.681988    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-tmp-dir\") pod \"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d\" (UID: \"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d\") "
Sep 16 19:42:35.682283 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:35.682010    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-kserve-provision-location\") pod \"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d\" (UID: \"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d\") "
Sep 16 19:42:35.682283 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:35.682038    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kk8jm\" (UniqueName: \"kubernetes.io/projected/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-kube-api-access-kk8jm\") pod \"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d\" (UID: \"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d\") "
Sep 16 19:42:35.682283 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:35.682237    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-model-cache" (OuterVolumeSpecName: "model-cache") pod "2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d" (UID: "2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:42:35.682283 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:35.682253    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d" (UID: "2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:42:35.682450 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:35.682371    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-home" (OuterVolumeSpecName: "home") pod "2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d" (UID: "2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:42:35.684774 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:35.684736    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d" (UID: "2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:42:35.684885 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:35.684796    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-dshm" (OuterVolumeSpecName: "dshm") pod "2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d" (UID: "2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:42:35.685497 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:35.685470    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-kube-api-access-kk8jm" (OuterVolumeSpecName: "kube-api-access-kk8jm") pod "2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d" (UID: "2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d"). InnerVolumeSpecName "kube-api-access-kk8jm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:42:35.693295 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:35.693269    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d" (UID: "2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:42:35.712091 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:35.712068    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d" (UID: "2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:42:35.783313 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:35.783291    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-tls-certs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:42:35.783313 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:35.783310    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-cabundle-cert\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:42:35.783443 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:35.783319    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-dshm\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:42:35.783443 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:35.783327    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-model-cache\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:42:35.783443 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:35.783335    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-tmp-dir\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:42:35.783443 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:35.783343    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-kserve-provision-location\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:42:35.783443 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:35.783352    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk8jm\" (UniqueName: \"kubernetes.io/projected/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-kube-api-access-kk8jm\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:42:35.783443 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:35.783359    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d-home\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:42:36.472181 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:36.472151    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5"
Sep 16 19:42:36.472181 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:36.472181    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5" event={"ID":"2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d","Type":"ContainerDied","Data":"98d4e0c6361d16be046cadb175a5cb9c2370f736ef76e6ca6186f3308f228f89"}
Sep 16 19:42:36.472774 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:36.472247    2577 scope.go:117] "RemoveContainer" containerID="492f9c4676a1d244517938d60aa925cec91fd9f646fe5fb8c315902a970c0f75"
Sep 16 19:42:36.481974 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:36.481956    2577 scope.go:117] "RemoveContainer" containerID="54c324272bb9bb09ca5ab1826016c2c94fa594b9cfc3c3d01b0383edbae00967"
Sep 16 19:42:36.501903 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:36.501880    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5"]
Sep 16 19:42:36.507601 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:36.507577    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6dbd8fbf4-w2mq5"]
Sep 16 19:42:37.155594 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:37.155563    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d" path="/var/lib/kubelet/pods/2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d/volumes"
Sep 16 19:42:37.583406 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:37.583314    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm" podUID="2179757f-1e83-45dd-8b65-19b57e471349" containerName="main" containerID="cri-o://4251dd106a9ec266b8a5bfca2d3afbaf36f4b2ff2f47316f4ed497e41c32ee18" gracePeriod=45
Sep 16 19:42:37.773360 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:37.773316    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm" podUID="2179757f-1e83-45dd-8b65-19b57e471349" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 19:42:38.237061 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:38.237037    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm"
Sep 16 19:42:38.403852 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:38.403827    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2179757f-1e83-45dd-8b65-19b57e471349-dshm\") pod \"2179757f-1e83-45dd-8b65-19b57e471349\" (UID: \"2179757f-1e83-45dd-8b65-19b57e471349\") "
Sep 16 19:42:38.403963 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:38.403855    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2179757f-1e83-45dd-8b65-19b57e471349-home\") pod \"2179757f-1e83-45dd-8b65-19b57e471349\" (UID: \"2179757f-1e83-45dd-8b65-19b57e471349\") "
Sep 16 19:42:38.403963 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:38.403873    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2179757f-1e83-45dd-8b65-19b57e471349-tmp-dir\") pod \"2179757f-1e83-45dd-8b65-19b57e471349\" (UID: \"2179757f-1e83-45dd-8b65-19b57e471349\") "
Sep 16 19:42:38.403963 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:38.403893    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q46vb\" (UniqueName: \"kubernetes.io/projected/2179757f-1e83-45dd-8b65-19b57e471349-kube-api-access-q46vb\") pod \"2179757f-1e83-45dd-8b65-19b57e471349\" (UID: \"2179757f-1e83-45dd-8b65-19b57e471349\") "
Sep 16 19:42:38.403963 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:38.403916    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2179757f-1e83-45dd-8b65-19b57e471349-model-cache\") pod \"2179757f-1e83-45dd-8b65-19b57e471349\" (UID: \"2179757f-1e83-45dd-8b65-19b57e471349\") "
Sep 16 19:42:38.404145 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:38.403969    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2179757f-1e83-45dd-8b65-19b57e471349-cabundle-cert\") pod \"2179757f-1e83-45dd-8b65-19b57e471349\" (UID: \"2179757f-1e83-45dd-8b65-19b57e471349\") "
Sep 16 19:42:38.404145 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:38.404011    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2179757f-1e83-45dd-8b65-19b57e471349-kserve-provision-location\") pod \"2179757f-1e83-45dd-8b65-19b57e471349\" (UID: \"2179757f-1e83-45dd-8b65-19b57e471349\") "
Sep 16 19:42:38.404145 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:38.404049    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2179757f-1e83-45dd-8b65-19b57e471349-tls-certs\") pod \"2179757f-1e83-45dd-8b65-19b57e471349\" (UID: \"2179757f-1e83-45dd-8b65-19b57e471349\") "
Sep 16 19:42:38.404360 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:38.404317    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2179757f-1e83-45dd-8b65-19b57e471349-home" (OuterVolumeSpecName: "home") pod "2179757f-1e83-45dd-8b65-19b57e471349" (UID: "2179757f-1e83-45dd-8b65-19b57e471349"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:42:38.404488 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:38.404352    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2179757f-1e83-45dd-8b65-19b57e471349-model-cache" (OuterVolumeSpecName: "model-cache") pod "2179757f-1e83-45dd-8b65-19b57e471349" (UID: "2179757f-1e83-45dd-8b65-19b57e471349"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:42:38.404488 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:38.404474    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2179757f-1e83-45dd-8b65-19b57e471349-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2179757f-1e83-45dd-8b65-19b57e471349" (UID: "2179757f-1e83-45dd-8b65-19b57e471349"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:42:38.406359 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:38.406337    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2179757f-1e83-45dd-8b65-19b57e471349-kube-api-access-q46vb" (OuterVolumeSpecName: "kube-api-access-q46vb") pod "2179757f-1e83-45dd-8b65-19b57e471349" (UID: "2179757f-1e83-45dd-8b65-19b57e471349"). InnerVolumeSpecName "kube-api-access-q46vb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:42:38.406461 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:38.406389    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2179757f-1e83-45dd-8b65-19b57e471349-dshm" (OuterVolumeSpecName: "dshm") pod "2179757f-1e83-45dd-8b65-19b57e471349" (UID: "2179757f-1e83-45dd-8b65-19b57e471349"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:42:38.406720 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:38.406697    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2179757f-1e83-45dd-8b65-19b57e471349-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2179757f-1e83-45dd-8b65-19b57e471349" (UID: "2179757f-1e83-45dd-8b65-19b57e471349"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:42:38.415852 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:38.415831    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2179757f-1e83-45dd-8b65-19b57e471349-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2179757f-1e83-45dd-8b65-19b57e471349" (UID: "2179757f-1e83-45dd-8b65-19b57e471349"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:42:38.427713 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:38.427688    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2179757f-1e83-45dd-8b65-19b57e471349-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2179757f-1e83-45dd-8b65-19b57e471349" (UID: "2179757f-1e83-45dd-8b65-19b57e471349"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:42:38.483243 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:38.483215    2577 generic.go:356] "Generic (PLEG): container finished" podID="2179757f-1e83-45dd-8b65-19b57e471349" containerID="4251dd106a9ec266b8a5bfca2d3afbaf36f4b2ff2f47316f4ed497e41c32ee18" exitCode=0
Sep 16 19:42:38.483323 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:38.483271    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm"
Sep 16 19:42:38.483323 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:38.483293    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm" event={"ID":"2179757f-1e83-45dd-8b65-19b57e471349","Type":"ContainerDied","Data":"4251dd106a9ec266b8a5bfca2d3afbaf36f4b2ff2f47316f4ed497e41c32ee18"}
Sep 16 19:42:38.483424 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:38.483337    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm" event={"ID":"2179757f-1e83-45dd-8b65-19b57e471349","Type":"ContainerDied","Data":"dfd6f7c81275175639972670f11909ed31b46806d2c59ba6e950c97b7bd06674"}
Sep 16 19:42:38.483424 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:38.483359    2577 scope.go:117] "RemoveContainer" containerID="4251dd106a9ec266b8a5bfca2d3afbaf36f4b2ff2f47316f4ed497e41c32ee18"
Sep 16 19:42:38.492407 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:38.492388    2577 scope.go:117] "RemoveContainer" containerID="0151baffa6e316cdfbb2637325d2c8bf0b0b6f658aa19698cdd5f38bee98fd14"
Sep 16 19:42:38.500357 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:38.500336    2577 scope.go:117] "RemoveContainer" containerID="4251dd106a9ec266b8a5bfca2d3afbaf36f4b2ff2f47316f4ed497e41c32ee18"
Sep 16 19:42:38.500672 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:42:38.500653    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4251dd106a9ec266b8a5bfca2d3afbaf36f4b2ff2f47316f4ed497e41c32ee18\": container with ID starting with 4251dd106a9ec266b8a5bfca2d3afbaf36f4b2ff2f47316f4ed497e41c32ee18 not found: ID does not exist" containerID="4251dd106a9ec266b8a5bfca2d3afbaf36f4b2ff2f47316f4ed497e41c32ee18"
Sep 16 19:42:38.500731 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:38.500679    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4251dd106a9ec266b8a5bfca2d3afbaf36f4b2ff2f47316f4ed497e41c32ee18"} err="failed to get container status \"4251dd106a9ec266b8a5bfca2d3afbaf36f4b2ff2f47316f4ed497e41c32ee18\": rpc error: code = NotFound desc = could not find container \"4251dd106a9ec266b8a5bfca2d3afbaf36f4b2ff2f47316f4ed497e41c32ee18\": container with ID starting with 4251dd106a9ec266b8a5bfca2d3afbaf36f4b2ff2f47316f4ed497e41c32ee18 not found: ID does not exist"
Sep 16 19:42:38.500731 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:38.500695    2577 scope.go:117] "RemoveContainer" containerID="0151baffa6e316cdfbb2637325d2c8bf0b0b6f658aa19698cdd5f38bee98fd14"
Sep 16 19:42:38.500960 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:42:38.500943    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0151baffa6e316cdfbb2637325d2c8bf0b0b6f658aa19698cdd5f38bee98fd14\": container with ID starting with 0151baffa6e316cdfbb2637325d2c8bf0b0b6f658aa19698cdd5f38bee98fd14 not found: ID does not exist" containerID="0151baffa6e316cdfbb2637325d2c8bf0b0b6f658aa19698cdd5f38bee98fd14"
Sep 16 19:42:38.501033 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:38.500964    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0151baffa6e316cdfbb2637325d2c8bf0b0b6f658aa19698cdd5f38bee98fd14"} err="failed to get container status \"0151baffa6e316cdfbb2637325d2c8bf0b0b6f658aa19698cdd5f38bee98fd14\": rpc error: code = NotFound desc = could not find container \"0151baffa6e316cdfbb2637325d2c8bf0b0b6f658aa19698cdd5f38bee98fd14\": container with ID starting with 0151baffa6e316cdfbb2637325d2c8bf0b0b6f658aa19698cdd5f38bee98fd14 not found: ID does not exist"
Sep 16 19:42:38.505331 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:38.505311    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2179757f-1e83-45dd-8b65-19b57e471349-tls-certs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:42:38.505423 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:38.505334    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2179757f-1e83-45dd-8b65-19b57e471349-dshm\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:42:38.505423 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:38.505345    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2179757f-1e83-45dd-8b65-19b57e471349-home\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:42:38.505423 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:38.505356    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2179757f-1e83-45dd-8b65-19b57e471349-tmp-dir\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:42:38.505423 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:38.505369    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q46vb\" (UniqueName: \"kubernetes.io/projected/2179757f-1e83-45dd-8b65-19b57e471349-kube-api-access-q46vb\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:42:38.505423 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:38.505383    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2179757f-1e83-45dd-8b65-19b57e471349-model-cache\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:42:38.505423 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:38.505395    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2179757f-1e83-45dd-8b65-19b57e471349-cabundle-cert\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:42:38.505423 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:38.505408    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2179757f-1e83-45dd-8b65-19b57e471349-kserve-provision-location\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:42:38.509561 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:38.509517    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm"]
Sep 16 19:42:38.517326 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:38.517305    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-db6bd4769-w2dhm"]
Sep 16 19:42:39.155169 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:39.155132    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2179757f-1e83-45dd-8b65-19b57e471349" path="/var/lib/kubelet/pods/2179757f-1e83-45dd-8b65-19b57e471349/volumes"
Sep 16 19:42:45.952681 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:45.952647    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fzzpp"]
Sep 16 19:42:45.953051 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:45.952992    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d" containerName="main"
Sep 16 19:42:45.953051 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:45.953004    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d" containerName="main"
Sep 16 19:42:45.953051 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:45.953049    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d" containerName="storage-initializer"
Sep 16 19:42:45.953177 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:45.953056    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d" containerName="storage-initializer"
Sep 16 19:42:45.953177 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:45.953069    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2179757f-1e83-45dd-8b65-19b57e471349" containerName="main"
Sep 16 19:42:45.953177 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:45.953076    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2179757f-1e83-45dd-8b65-19b57e471349" containerName="main"
Sep 16 19:42:45.953177 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:45.953083    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2179757f-1e83-45dd-8b65-19b57e471349" containerName="storage-initializer"
Sep 16 19:42:45.953177 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:45.953090    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2179757f-1e83-45dd-8b65-19b57e471349" containerName="storage-initializer"
Sep 16 19:42:45.953177 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:45.953148    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2764bf1f-ba79-4d9e-a542-e6d6f1b52f8d" containerName="main"
Sep 16 19:42:45.953177 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:45.953159    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2179757f-1e83-45dd-8b65-19b57e471349" containerName="main"
Sep 16 19:42:45.957647 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:45.957615    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fzzpp"
Sep 16 19:42:45.960876 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:45.960853    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Sep 16 19:42:45.962356 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:45.961840    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Sep 16 19:42:45.962356 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:45.962037    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Sep 16 19:42:45.962356 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:45.962186    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-mcz2w\""
Sep 16 19:42:45.964219 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:45.964192    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fzzpp"]
Sep 16 19:42:46.065011 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:46.064983    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcjgk\" (UniqueName: \"kubernetes.io/projected/2bfcaf50-33db-49da-b6c2-c9f256d3003a-kube-api-access-lcjgk\") pod \"e2e-pvc-lora-download-fzzpp\" (UID: \"2bfcaf50-33db-49da-b6c2-c9f256d3003a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fzzpp"
Sep 16 19:42:46.165919 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:46.165885    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcjgk\" (UniqueName: \"kubernetes.io/projected/2bfcaf50-33db-49da-b6c2-c9f256d3003a-kube-api-access-lcjgk\") pod \"e2e-pvc-lora-download-fzzpp\" (UID: \"2bfcaf50-33db-49da-b6c2-c9f256d3003a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fzzpp"
Sep 16 19:42:46.175735 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:46.175710    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcjgk\" (UniqueName: \"kubernetes.io/projected/2bfcaf50-33db-49da-b6c2-c9f256d3003a-kube-api-access-lcjgk\") pod \"e2e-pvc-lora-download-fzzpp\" (UID: \"2bfcaf50-33db-49da-b6c2-c9f256d3003a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fzzpp"
Sep 16 19:42:55.338823 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:55.338791    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-2fe947a3-5c1e-4794-abe9-094caaa71ac1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08f0c901b6a67c9df\") pod \"e2e-pvc-lora-download-fzzpp\" (UID: \"2bfcaf50-33db-49da-b6c2-c9f256d3003a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fzzpp"
Sep 16 19:42:55.345850 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:55.345824    2577 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-2fe947a3-5c1e-4794-abe9-094caaa71ac1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08f0c901b6a67c9df\") pod \"e2e-pvc-lora-download-fzzpp\" (UID: \"2bfcaf50-33db-49da-b6c2-c9f256d3003a\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fzzpp"
Sep 16 19:42:55.440109 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:55.440082    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-2fe947a3-5c1e-4794-abe9-094caaa71ac1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08f0c901b6a67c9df\") pod \"e2e-pvc-lora-download-fzzpp\" (UID: \"2bfcaf50-33db-49da-b6c2-c9f256d3003a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fzzpp"
Sep 16 19:42:55.440214 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:55.440199    2577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-2fe947a3-5c1e-4794-abe9-094caaa71ac1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08f0c901b6a67c9df\") pod \"e2e-pvc-lora-download-fzzpp\" (UID: \"2bfcaf50-33db-49da-b6c2-c9f256d3003a\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fzzpp"
Sep 16 19:42:55.443422 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:55.443404    2577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-2fe947a3-5c1e-4794-abe9-094caaa71ac1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08f0c901b6a67c9df\") pod \"e2e-pvc-lora-download-fzzpp\" (UID: \"2bfcaf50-33db-49da-b6c2-c9f256d3003a\") DevicePath \"csi-4363e22c73ac0a714f654589ee88d65bf183440c3b2e66b5d4d89e0404af6395\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fzzpp"
Sep 16 19:42:55.659247 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:55.659219    2577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-2fe947a3-5c1e-4794-abe9-094caaa71ac1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08f0c901b6a67c9df\") pod \"e2e-pvc-lora-download-fzzpp\" (UID: \"2bfcaf50-33db-49da-b6c2-c9f256d3003a\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/ca81d73a49359f683bb5af066e02a187d39aac3d5a85ce42751b56bc44f1d64a/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fzzpp"
Sep 16 19:42:55.672727 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:55.672703    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-2fe947a3-5c1e-4794-abe9-094caaa71ac1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08f0c901b6a67c9df\") pod \"e2e-pvc-lora-download-fzzpp\" (UID: \"2bfcaf50-33db-49da-b6c2-c9f256d3003a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fzzpp"
Sep 16 19:42:55.867277 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:55.867249    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fzzpp"
Sep 16 19:42:56.142359 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:56.142281    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fzzpp"]
Sep 16 19:42:56.144924 ip-10-0-128-247 kubenswrapper[2577]: W0916 19:42:56.144895    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2bfcaf50_33db_49da_b6c2_c9f256d3003a.slice/crio-5524faca6544309495fa7659ae6619df64199785533d5f463f027bbfaea3eefc WatchSource:0}: Error finding container 5524faca6544309495fa7659ae6619df64199785533d5f463f027bbfaea3eefc: Status 404 returned error can't find the container with id 5524faca6544309495fa7659ae6619df64199785533d5f463f027bbfaea3eefc
Sep 16 19:42:56.567695 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:56.567651    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fzzpp" event={"ID":"2bfcaf50-33db-49da-b6c2-c9f256d3003a","Type":"ContainerStarted","Data":"e4b18f71be231a546a89bf66e0471bce3f9e7e1bceb573f3b64e98c1fbeef3de"}
Sep 16 19:42:56.568069 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:56.567707    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fzzpp" event={"ID":"2bfcaf50-33db-49da-b6c2-c9f256d3003a","Type":"ContainerStarted","Data":"5524faca6544309495fa7659ae6619df64199785533d5f463f027bbfaea3eefc"}
Sep 16 19:42:56.586725 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:56.586667    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fzzpp" podStartSLOduration=14.586647689 podStartE2EDuration="14.586647689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:42:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:42:56.585201617 +0000 UTC m=+3733.968090982" watchObservedRunningTime="2026-09-16 19:42:56.586647689 +0000 UTC m=+3733.969537055"
Sep 16 19:42:58.577806 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:58.577771    2577 generic.go:356] "Generic (PLEG): container finished" podID="2bfcaf50-33db-49da-b6c2-c9f256d3003a" containerID="e4b18f71be231a546a89bf66e0471bce3f9e7e1bceb573f3b64e98c1fbeef3de" exitCode=0
Sep 16 19:42:58.578188 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:58.577855    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fzzpp" event={"ID":"2bfcaf50-33db-49da-b6c2-c9f256d3003a","Type":"ContainerDied","Data":"e4b18f71be231a546a89bf66e0471bce3f9e7e1bceb573f3b64e98c1fbeef3de"}
Sep 16 19:42:59.718241 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:59.718217    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fzzpp"
Sep 16 19:42:59.874685 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:59.874619    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcjgk\" (UniqueName: \"kubernetes.io/projected/2bfcaf50-33db-49da-b6c2-c9f256d3003a-kube-api-access-lcjgk\") pod \"2bfcaf50-33db-49da-b6c2-c9f256d3003a\" (UID: \"2bfcaf50-33db-49da-b6c2-c9f256d3003a\") "
Sep 16 19:42:59.874797 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:59.874768    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08f0c901b6a67c9df\") pod \"2bfcaf50-33db-49da-b6c2-c9f256d3003a\" (UID: \"2bfcaf50-33db-49da-b6c2-c9f256d3003a\") "
Sep 16 19:42:59.876995 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:59.876969    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bfcaf50-33db-49da-b6c2-c9f256d3003a-kube-api-access-lcjgk" (OuterVolumeSpecName: "kube-api-access-lcjgk") pod "2bfcaf50-33db-49da-b6c2-c9f256d3003a" (UID: "2bfcaf50-33db-49da-b6c2-c9f256d3003a"). InnerVolumeSpecName "kube-api-access-lcjgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:42:59.878820 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:59.878795    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-08f0c901b6a67c9df" (OuterVolumeSpecName: "model-storage") pod "2bfcaf50-33db-49da-b6c2-c9f256d3003a" (UID: "2bfcaf50-33db-49da-b6c2-c9f256d3003a"). InnerVolumeSpecName "pvc-2fe947a3-5c1e-4794-abe9-094caaa71ac1". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 16 19:42:59.975953 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:59.975933    2577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-2fe947a3-5c1e-4794-abe9-094caaa71ac1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08f0c901b6a67c9df\") on node \"ip-10-0-128-247.ec2.internal\" "
Sep 16 19:42:59.975953 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:42:59.975953    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcjgk\" (UniqueName: \"kubernetes.io/projected/2bfcaf50-33db-49da-b6c2-c9f256d3003a-kube-api-access-lcjgk\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:43:00.587900 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:00.587864    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fzzpp" event={"ID":"2bfcaf50-33db-49da-b6c2-c9f256d3003a","Type":"ContainerDied","Data":"5524faca6544309495fa7659ae6619df64199785533d5f463f027bbfaea3eefc"}
Sep 16 19:43:00.587900 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:00.587903    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5524faca6544309495fa7659ae6619df64199785533d5f463f027bbfaea3eefc"
Sep 16 19:43:00.588105 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:00.587908    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fzzpp"
Sep 16 19:43:00.946679 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:00.946654    2577 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-2fe947a3-5c1e-4794-abe9-094caaa71ac1" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-08f0c901b6a67c9df") on node "ip-10-0-128-247.ec2.internal"
Sep 16 19:43:00.982980 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:00.982950    2577 reconciler_common.go:299] "Volume detached for volume \"pvc-2fe947a3-5c1e-4794-abe9-094caaa71ac1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08f0c901b6a67c9df\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"csi-4363e22c73ac0a714f654589ee88d65bf183440c3b2e66b5d4d89e0404af6395\""
Sep 16 19:43:10.862154 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:10.862115    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"]
Sep 16 19:43:10.862669 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:10.862452    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bfcaf50-33db-49da-b6c2-c9f256d3003a" containerName="storage-initializer"
Sep 16 19:43:10.862669 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:10.862464    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bfcaf50-33db-49da-b6c2-c9f256d3003a" containerName="storage-initializer"
Sep 16 19:43:10.862669 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:10.862551    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bfcaf50-33db-49da-b6c2-c9f256d3003a" containerName="storage-initializer"
Sep 16 19:43:10.870091 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:10.866381    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"
Sep 16 19:43:10.870779 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:10.870754    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-mcz2w\""
Sep 16 19:43:10.870929 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:10.870790    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:43:10.870929 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:10.870860    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Sep 16 19:43:10.871164 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:10.871148    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Sep 16 19:43:10.871277 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:10.871148    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Sep 16 19:43:10.871342 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:10.871283    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"lora-pvc-single-adapter-test-kserve-self-signed-certs\""
Sep 16 19:43:10.882289 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:10.882267    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"]
Sep 16 19:43:10.954093 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:10.954063    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-f74755dd-msk9m\" (UID: \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"
Sep 16 19:43:10.954244 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:10.954097    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-f74755dd-msk9m\" (UID: \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"
Sep 16 19:43:10.954244 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:10.954165    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-home\") pod \"lora-pvc-single-adapter-test-kserve-f74755dd-msk9m\" (UID: \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"
Sep 16 19:43:10.954244 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:10.954183    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-f74755dd-msk9m\" (UID: \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"
Sep 16 19:43:10.954244 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:10.954207    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-dshm\") pod \"lora-pvc-single-adapter-test-kserve-f74755dd-msk9m\" (UID: \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"
Sep 16 19:43:10.954244 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:10.954243    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-f74755dd-msk9m\" (UID: \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"
Sep 16 19:43:10.954455 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:10.954260    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-f74755dd-msk9m\" (UID: \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"
Sep 16 19:43:11.054690 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:11.054659    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-home\") pod \"lora-pvc-single-adapter-test-kserve-f74755dd-msk9m\" (UID: \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"
Sep 16 19:43:11.054876 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:11.054701    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-f74755dd-msk9m\" (UID: \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"
Sep 16 19:43:11.054876 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:11.054744    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-dshm\") pod \"lora-pvc-single-adapter-test-kserve-f74755dd-msk9m\" (UID: \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"
Sep 16 19:43:11.054876 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:11.054783    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-f74755dd-msk9m\" (UID: \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"
Sep 16 19:43:11.054876 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:11.054809    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-f74755dd-msk9m\" (UID: \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"
Sep 16 19:43:11.054876 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:11.054847    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4wbg\" (UniqueName: \"kubernetes.io/projected/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-kube-api-access-f4wbg\") pod \"lora-pvc-single-adapter-test-kserve-f74755dd-msk9m\" (UID: \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"
Sep 16 19:43:11.055156 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:11.054902    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-f74755dd-msk9m\" (UID: \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"
Sep 16 19:43:11.055156 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:11.054938    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-f74755dd-msk9m\" (UID: \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"
Sep 16 19:43:11.055156 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:11.055115    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-home\") pod \"lora-pvc-single-adapter-test-kserve-f74755dd-msk9m\" (UID: \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"
Sep 16 19:43:11.055350 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:11.055326    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-f74755dd-msk9m\" (UID: \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"
Sep 16 19:43:11.055437 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:11.055354    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-f74755dd-msk9m\" (UID: \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"
Sep 16 19:43:11.055503 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:11.055479    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-f74755dd-msk9m\" (UID: \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"
Sep 16 19:43:11.055736 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:11.055714    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-f74755dd-msk9m\" (UID: \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"
Sep 16 19:43:11.057210 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:11.057194    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-dshm\") pod \"lora-pvc-single-adapter-test-kserve-f74755dd-msk9m\" (UID: \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"
Sep 16 19:43:11.057568 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:11.057550    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-f74755dd-msk9m\" (UID: \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"
Sep 16 19:43:11.155821 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:11.155789    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4wbg\" (UniqueName: \"kubernetes.io/projected/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-kube-api-access-f4wbg\") pod \"lora-pvc-single-adapter-test-kserve-f74755dd-msk9m\" (UID: \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"
Sep 16 19:43:11.169494 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:11.169458    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4wbg\" (UniqueName: \"kubernetes.io/projected/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-kube-api-access-f4wbg\") pod \"lora-pvc-single-adapter-test-kserve-f74755dd-msk9m\" (UID: \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"
Sep 16 19:43:15.586970 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:15.586934    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-2fe947a3-5c1e-4794-abe9-094caaa71ac1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08f0c901b6a67c9df\") pod \"lora-pvc-single-adapter-test-kserve-f74755dd-msk9m\" (UID: \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"
Sep 16 19:43:15.593997 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:15.593973    2577 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-2fe947a3-5c1e-4794-abe9-094caaa71ac1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08f0c901b6a67c9df\") pod \"lora-pvc-single-adapter-test-kserve-f74755dd-msk9m\" (UID: \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"
Sep 16 19:43:15.688106 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:15.688077    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-2fe947a3-5c1e-4794-abe9-094caaa71ac1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08f0c901b6a67c9df\") pod \"lora-pvc-single-adapter-test-kserve-f74755dd-msk9m\" (UID: \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"
Sep 16 19:43:15.688232 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:15.688183    2577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-2fe947a3-5c1e-4794-abe9-094caaa71ac1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08f0c901b6a67c9df\") pod \"lora-pvc-single-adapter-test-kserve-f74755dd-msk9m\" (UID: \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"
Sep 16 19:43:15.691545 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:15.691507    2577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-2fe947a3-5c1e-4794-abe9-094caaa71ac1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08f0c901b6a67c9df\") pod \"lora-pvc-single-adapter-test-kserve-f74755dd-msk9m\" (UID: \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\") DevicePath \"csi-4363e22c73ac0a714f654589ee88d65bf183440c3b2e66b5d4d89e0404af6395\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"
Sep 16 19:43:15.781888 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:15.781850    2577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-2fe947a3-5c1e-4794-abe9-094caaa71ac1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08f0c901b6a67c9df\") pod \"lora-pvc-single-adapter-test-kserve-f74755dd-msk9m\" (UID: \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/ca81d73a49359f683bb5af066e02a187d39aac3d5a85ce42751b56bc44f1d64a/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"
Sep 16 19:43:15.804339 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:15.804312    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-2fe947a3-5c1e-4794-abe9-094caaa71ac1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08f0c901b6a67c9df\") pod \"lora-pvc-single-adapter-test-kserve-f74755dd-msk9m\" (UID: \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"
Sep 16 19:43:15.978889 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:15.978860    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"
Sep 16 19:43:16.259167 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:16.259144    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"]
Sep 16 19:43:16.259907 ip-10-0-128-247 kubenswrapper[2577]: W0916 19:43:16.259882    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b7a83a_41fe_4e97_98cf_3e0ba372ecc3.slice/crio-c63893b8107f61705b9fdad29e8310eb162f0ce6550b04a80d1f55b9317cbe17 WatchSource:0}: Error finding container c63893b8107f61705b9fdad29e8310eb162f0ce6550b04a80d1f55b9317cbe17: Status 404 returned error can't find the container with id c63893b8107f61705b9fdad29e8310eb162f0ce6550b04a80d1f55b9317cbe17
Sep 16 19:43:16.655493 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:16.655457    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m" event={"ID":"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3","Type":"ContainerStarted","Data":"288ce7e3d19ebac01d27d3156f5dd3711f5dd20e5e63f01af407f9a335169a91"}
Sep 16 19:43:16.655981 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:16.655521    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m" event={"ID":"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3","Type":"ContainerStarted","Data":"c63893b8107f61705b9fdad29e8310eb162f0ce6550b04a80d1f55b9317cbe17"}
Sep 16 19:43:18.667620 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:18.667588    2577 generic.go:356] "Generic (PLEG): container finished" podID="f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3" containerID="288ce7e3d19ebac01d27d3156f5dd3711f5dd20e5e63f01af407f9a335169a91" exitCode=0
Sep 16 19:43:18.667980 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:18.667663    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m" event={"ID":"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3","Type":"ContainerDied","Data":"288ce7e3d19ebac01d27d3156f5dd3711f5dd20e5e63f01af407f9a335169a91"}
Sep 16 19:43:19.673819 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:19.673783    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m" event={"ID":"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3","Type":"ContainerStarted","Data":"fcc7bcbbf97d85c88903635d1094ac712f9212e885af14be7c850e3be6be08be"}
Sep 16 19:43:19.700734 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:19.700676    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m" podStartSLOduration=7.680327513 podStartE2EDuration="9.700655725s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.020328212s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:43:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:43:19.700487659 +0000 UTC m=+3757.083377026" watchObservedRunningTime="2026-09-16 19:43:19.700655725 +0000 UTC m=+3757.083545090"
Sep 16 19:43:19.979206 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:19.979130    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"
Sep 16 19:43:25.979706 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:25.979668    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"
Sep 16 19:43:25.981140 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:25.981107    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m" podUID="f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 16 19:43:35.979886 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:35.979785    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m" podUID="f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 16 19:43:45.979936 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:45.979891    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m" podUID="f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 16 19:43:51.555947 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:51.555904    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk"]
Sep 16 19:43:51.559771 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:51.559742    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk"
Sep 16 19:43:51.562609 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:51.562549    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 16 19:43:51.563080 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:51.563038    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:43:51.563602 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:51.563572    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 16 19:43:51.563938 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:51.563613    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 16 19:43:51.564378 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:51.564344    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-mknvd\""
Sep 16 19:43:51.564707 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:51.563665    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 16 19:43:51.577274 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:51.577247    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk"]
Sep 16 19:43:51.590297 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:51.590273    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl8jl\" (UniqueName: \"kubernetes.io/projected/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-kube-api-access-fl8jl\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk\" (UID: \"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk"
Sep 16 19:43:51.590419 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:51.590325    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk\" (UID: \"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk"
Sep 16 19:43:51.590419 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:51.590356    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk\" (UID: \"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk"
Sep 16 19:43:51.590419 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:51.590386    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk\" (UID: \"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk"
Sep 16 19:43:51.590589 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:51.590444    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk\" (UID: \"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk"
Sep 16 19:43:51.590589 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:51.590569    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk\" (UID: \"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk"
Sep 16 19:43:51.590692 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:51.590615    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk\" (UID: \"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk"
Sep 16 19:43:51.590692 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:51.590674    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk\" (UID: \"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk"
Sep 16 19:43:51.691027 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:51.690998    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl8jl\" (UniqueName: \"kubernetes.io/projected/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-kube-api-access-fl8jl\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk\" (UID: \"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk"
Sep 16 19:43:51.691188 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:51.691039    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk\" (UID: \"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk"
Sep 16 19:43:51.691188 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:51.691060    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk\" (UID: \"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk"
Sep 16 19:43:51.691188 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:51.691076    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk\" (UID: \"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk"
Sep 16 19:43:51.691188 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:51.691106    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk\" (UID: \"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk"
Sep 16 19:43:51.691423 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:51.691271    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk\" (UID: \"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk"
Sep 16 19:43:51.691423 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:51.691337    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk\" (UID: \"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk"
Sep 16 19:43:51.691423 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:51.691372    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk\" (UID: \"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk"
Sep 16 19:43:51.691615 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:51.691503    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk\" (UID: \"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk"
Sep 16 19:43:51.691615 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:51.691522    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk\" (UID: \"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk"
Sep 16 19:43:51.691741 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:51.691720    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk\" (UID: \"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk"
Sep 16 19:43:51.691942 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:51.691913    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk\" (UID: \"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk"
Sep 16 19:43:51.692268 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:51.692213    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk\" (UID: \"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk"
Sep 16 19:43:51.693897 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:51.693872    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk\" (UID: \"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk"
Sep 16 19:43:51.694708 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:51.694678    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk\" (UID: \"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk"
Sep 16 19:43:51.700858 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:51.700832    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl8jl\" (UniqueName: \"kubernetes.io/projected/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-kube-api-access-fl8jl\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk\" (UID: \"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk"
Sep 16 19:43:51.876420 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:51.876325    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk"
Sep 16 19:43:52.155873 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:52.155761    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk"]
Sep 16 19:43:52.822198 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:52.822100    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk" event={"ID":"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0","Type":"ContainerStarted","Data":"05d44b82a0354614efa8bf0efd24486ce2a867078a1f83b9e07f8c8d3f280ed5"}
Sep 16 19:43:52.822198 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:52.822157    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk" event={"ID":"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0","Type":"ContainerStarted","Data":"ee82795229ebac5550dd1ff927f0a4a6e04fd97ee5915a3c9e43fde4d2fadc3b"}
Sep 16 19:43:54.834835 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:54.834750    2577 generic.go:356] "Generic (PLEG): container finished" podID="ffe28a99-65d0-4c5e-ae54-6e36a621a3c0" containerID="05d44b82a0354614efa8bf0efd24486ce2a867078a1f83b9e07f8c8d3f280ed5" exitCode=0
Sep 16 19:43:54.835290 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:54.834830    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk" event={"ID":"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0","Type":"ContainerDied","Data":"05d44b82a0354614efa8bf0efd24486ce2a867078a1f83b9e07f8c8d3f280ed5"}
Sep 16 19:43:55.841559 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:55.841501    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk" event={"ID":"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0","Type":"ContainerStarted","Data":"fd27d0b428623639ebd7851e55e9994615156a7de0201365ee47e3ab7f515ed0"}
Sep 16 19:43:55.867490 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:55.867441    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk" podStartSLOduration=2.661606192 podStartE2EDuration="4.867424687s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.205818495s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:43:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:43:55.865369054 +0000 UTC m=+3793.248258420" watchObservedRunningTime="2026-09-16 19:43:55.867424687 +0000 UTC m=+3793.250314052"
Sep 16 19:43:55.980071 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:43:55.980018    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m" podUID="f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 16 19:44:01.876730 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:44:01.876691    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk"
Sep 16 19:44:01.877158 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:44:01.876778    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk"
Sep 16 19:44:01.878573 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:44:01.878515    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk" podUID="ffe28a99-65d0-4c5e-ae54-6e36a621a3c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Sep 16 19:44:05.979981 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:44:05.979926    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m" podUID="f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 16 19:44:11.877429 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:44:11.877375    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk" podUID="ffe28a99-65d0-4c5e-ae54-6e36a621a3c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Sep 16 19:44:15.980038 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:44:15.979988    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m" podUID="f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 16 19:44:21.877509 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:44:21.877458    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk" podUID="ffe28a99-65d0-4c5e-ae54-6e36a621a3c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Sep 16 19:44:25.979764 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:44:25.979712    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m" podUID="f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 16 19:44:31.876824 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:44:31.876780    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk" podUID="ffe28a99-65d0-4c5e-ae54-6e36a621a3c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Sep 16 19:44:35.989471 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:44:35.989440    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"
Sep 16 19:44:35.997546 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:44:35.997498    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"
Sep 16 19:44:41.877454 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:44:41.877406    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk" podUID="ffe28a99-65d0-4c5e-ae54-6e36a621a3c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Sep 16 19:44:51.876758 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:44:51.876699    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk" podUID="ffe28a99-65d0-4c5e-ae54-6e36a621a3c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Sep 16 19:45:00.892680 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:00.892645    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-f74755dd-msk9m_f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3/storage-initializer/0.log"
Sep 16 19:45:00.904033 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:00.903986    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-f74755dd-msk9m_f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3/main/0.log"
Sep 16 19:45:01.876948 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:01.876889    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk" podUID="ffe28a99-65d0-4c5e-ae54-6e36a621a3c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Sep 16 19:45:02.084567 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:02.084510    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"]
Sep 16 19:45:11.877546 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:11.877431    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk" podUID="ffe28a99-65d0-4c5e-ae54-6e36a621a3c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Sep 16 19:45:17.107550 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:17.107496    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m" podUID="f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3" containerName="main" containerID="cri-o://fcc7bcbbf97d85c88903635d1094ac712f9212e885af14be7c850e3be6be08be" gracePeriod=45
Sep 16 19:45:17.962827 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:17.962797    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"
Sep 16 19:45:18.088319 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:18.088248    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-model-cache\") pod \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\" (UID: \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\") "
Sep 16 19:45:18.088461 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:18.088319    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-cabundle-cert\") pod \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\" (UID: \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\") "
Sep 16 19:45:18.088461 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:18.088422    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"lora-pvc-lora-adapter-1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08f0c901b6a67c9df\") pod \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\" (UID: \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\") "
Sep 16 19:45:18.088461 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:18.088452    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4wbg\" (UniqueName: \"kubernetes.io/projected/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-kube-api-access-f4wbg\") pod \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\" (UID: \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\") "
Sep 16 19:45:18.088903 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:18.088506    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-tmp-dir\") pod \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\" (UID: \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\") "
Sep 16 19:45:18.088903 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:18.088556    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-model-cache" (OuterVolumeSpecName: "model-cache") pod "f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3" (UID: "f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:45:18.088903 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:18.088560    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-kserve-provision-location\") pod \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\" (UID: \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\") "
Sep 16 19:45:18.088903 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:18.088611    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-home\") pod \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\" (UID: \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\") "
Sep 16 19:45:18.088903 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:18.088648    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-tls-certs\") pod \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\" (UID: \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\") "
Sep 16 19:45:18.088903 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:18.088686    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-dshm\") pod \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\" (UID: \"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3\") "
Sep 16 19:45:18.088903 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:18.088713    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3" (UID: "f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:45:18.089428 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:18.089051    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-home" (OuterVolumeSpecName: "home") pod "f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3" (UID: "f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:45:18.089428 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:18.089347    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-model-cache\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:45:18.089428 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:18.089365    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-cabundle-cert\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:45:18.089428 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:18.089379    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-home\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:45:18.091843 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:18.091807    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-kube-api-access-f4wbg" (OuterVolumeSpecName: "kube-api-access-f4wbg") pod "f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3" (UID: "f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3"). InnerVolumeSpecName "kube-api-access-f4wbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:45:18.091971 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:18.091925    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-dshm" (OuterVolumeSpecName: "dshm") pod "f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3" (UID: "f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:45:18.092796 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:18.092770    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3" (UID: "f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:45:18.093832 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:18.093812    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-08f0c901b6a67c9df" (OuterVolumeSpecName: "lora-pvc-lora-adapter-1") pod "f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3" (UID: "f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3"). InnerVolumeSpecName "pvc-2fe947a3-5c1e-4794-abe9-094caaa71ac1". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 16 19:45:18.106330 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:18.106297    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3" (UID: "f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:45:18.114720 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:18.114696    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3" (UID: "f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:45:18.189891 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:18.189861    2577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-2fe947a3-5c1e-4794-abe9-094caaa71ac1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08f0c901b6a67c9df\") on node \"ip-10-0-128-247.ec2.internal\" "
Sep 16 19:45:18.189891 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:18.189888    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4wbg\" (UniqueName: \"kubernetes.io/projected/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-kube-api-access-f4wbg\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:45:18.190050 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:18.189902    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-tmp-dir\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:45:18.190050 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:18.189913    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-kserve-provision-location\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:45:18.190050 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:18.189921    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-tls-certs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:45:18.190050 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:18.189930    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3-dshm\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:45:18.199375 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:18.199359    2577 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-2fe947a3-5c1e-4794-abe9-094caaa71ac1" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-08f0c901b6a67c9df") on node "ip-10-0-128-247.ec2.internal"
Sep 16 19:45:18.255304 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:18.255272    2577 generic.go:356] "Generic (PLEG): container finished" podID="f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3" containerID="fcc7bcbbf97d85c88903635d1094ac712f9212e885af14be7c850e3be6be08be" exitCode=0
Sep 16 19:45:18.255447 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:18.255340    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"
Sep 16 19:45:18.255447 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:18.255349    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m" event={"ID":"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3","Type":"ContainerDied","Data":"fcc7bcbbf97d85c88903635d1094ac712f9212e885af14be7c850e3be6be08be"}
Sep 16 19:45:18.255447 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:18.255428    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m" event={"ID":"f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3","Type":"ContainerDied","Data":"c63893b8107f61705b9fdad29e8310eb162f0ce6550b04a80d1f55b9317cbe17"}
Sep 16 19:45:18.255650 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:18.255450    2577 scope.go:117] "RemoveContainer" containerID="fcc7bcbbf97d85c88903635d1094ac712f9212e885af14be7c850e3be6be08be"
Sep 16 19:45:18.264573 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:18.264553    2577 scope.go:117] "RemoveContainer" containerID="288ce7e3d19ebac01d27d3156f5dd3711f5dd20e5e63f01af407f9a335169a91"
Sep 16 19:45:18.272615 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:18.272597    2577 scope.go:117] "RemoveContainer" containerID="fcc7bcbbf97d85c88903635d1094ac712f9212e885af14be7c850e3be6be08be"
Sep 16 19:45:18.272888 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:45:18.272870    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcc7bcbbf97d85c88903635d1094ac712f9212e885af14be7c850e3be6be08be\": container with ID starting with fcc7bcbbf97d85c88903635d1094ac712f9212e885af14be7c850e3be6be08be not found: ID does not exist" containerID="fcc7bcbbf97d85c88903635d1094ac712f9212e885af14be7c850e3be6be08be"
Sep 16 19:45:18.272943 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:18.272897    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcc7bcbbf97d85c88903635d1094ac712f9212e885af14be7c850e3be6be08be"} err="failed to get container status \"fcc7bcbbf97d85c88903635d1094ac712f9212e885af14be7c850e3be6be08be\": rpc error: code = NotFound desc = could not find container \"fcc7bcbbf97d85c88903635d1094ac712f9212e885af14be7c850e3be6be08be\": container with ID starting with fcc7bcbbf97d85c88903635d1094ac712f9212e885af14be7c850e3be6be08be not found: ID does not exist"
Sep 16 19:45:18.272943 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:18.272914    2577 scope.go:117] "RemoveContainer" containerID="288ce7e3d19ebac01d27d3156f5dd3711f5dd20e5e63f01af407f9a335169a91"
Sep 16 19:45:18.273143 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:45:18.273125    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"288ce7e3d19ebac01d27d3156f5dd3711f5dd20e5e63f01af407f9a335169a91\": container with ID starting with 288ce7e3d19ebac01d27d3156f5dd3711f5dd20e5e63f01af407f9a335169a91 not found: ID does not exist" containerID="288ce7e3d19ebac01d27d3156f5dd3711f5dd20e5e63f01af407f9a335169a91"
Sep 16 19:45:18.273183 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:18.273150    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"288ce7e3d19ebac01d27d3156f5dd3711f5dd20e5e63f01af407f9a335169a91"} err="failed to get container status \"288ce7e3d19ebac01d27d3156f5dd3711f5dd20e5e63f01af407f9a335169a91\": rpc error: code = NotFound desc = could not find container \"288ce7e3d19ebac01d27d3156f5dd3711f5dd20e5e63f01af407f9a335169a91\": container with ID starting with 288ce7e3d19ebac01d27d3156f5dd3711f5dd20e5e63f01af407f9a335169a91 not found: ID does not exist"
Sep 16 19:45:18.281961 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:18.281940    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"]
Sep 16 19:45:18.290748 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:18.290723    2577 reconciler_common.go:299] "Volume detached for volume \"pvc-2fe947a3-5c1e-4794-abe9-094caaa71ac1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08f0c901b6a67c9df\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"csi-4363e22c73ac0a714f654589ee88d65bf183440c3b2e66b5d4d89e0404af6395\""
Sep 16 19:45:18.290906 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:18.290888    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-f74755dd-msk9m"]
Sep 16 19:45:19.155425 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:19.155383    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3" path="/var/lib/kubelet/pods/f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3/volumes"
Sep 16 19:45:21.877522 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:21.877472    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk" podUID="ffe28a99-65d0-4c5e-ae54-6e36a621a3c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Sep 16 19:45:25.630428 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:25.630387    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-6prc7"]
Sep 16 19:45:25.630915 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:25.630879    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3" containerName="storage-initializer"
Sep 16 19:45:25.630915 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:25.630898    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3" containerName="storage-initializer"
Sep 16 19:45:25.631040 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:25.630943    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3" containerName="main"
Sep 16 19:45:25.631040 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:25.630953    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3" containerName="main"
Sep 16 19:45:25.631040 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:25.631028    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4b7a83a-41fe-4e97-98cf-3e0ba372ecc3" containerName="main"
Sep 16 19:45:25.635593 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:25.635574    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-6prc7"
Sep 16 19:45:25.638152 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:25.638128    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 16 19:45:25.638292 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:25.638273    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"seaweedfs-s3-creds\""
Sep 16 19:45:25.639498 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:25.639472    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 16 19:45:25.639498 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:25.639501    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-cp97w\""
Sep 16 19:45:25.641625 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:25.641579    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-6prc7"]
Sep 16 19:45:25.760079 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:25.760049    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfmf8\" (UniqueName: \"kubernetes.io/projected/d9f182a2-6394-488b-9a26-489578ea4818-kube-api-access-lfmf8\") pod \"e2e-pvc-lora-download-6prc7\" (UID: \"d9f182a2-6394-488b-9a26-489578ea4818\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-6prc7"
Sep 16 19:45:25.860569 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:25.860546    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfmf8\" (UniqueName: \"kubernetes.io/projected/d9f182a2-6394-488b-9a26-489578ea4818-kube-api-access-lfmf8\") pod \"e2e-pvc-lora-download-6prc7\" (UID: \"d9f182a2-6394-488b-9a26-489578ea4818\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-6prc7"
Sep 16 19:45:25.877968 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:25.877951    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfmf8\" (UniqueName: \"kubernetes.io/projected/d9f182a2-6394-488b-9a26-489578ea4818-kube-api-access-lfmf8\") pod \"e2e-pvc-lora-download-6prc7\" (UID: \"d9f182a2-6394-488b-9a26-489578ea4818\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-6prc7"
Sep 16 19:45:27.281031 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:27.280996    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fzzpp"]
Sep 16 19:45:27.283793 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:27.283767    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fzzpp"]
Sep 16 19:45:28.080635 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:28.080589    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-86b39b6d-ce90-4025-823c-c0244a45ed72\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09b88cffeeadd7a3d\") pod \"e2e-pvc-lora-download-6prc7\" (UID: \"d9f182a2-6394-488b-9a26-489578ea4818\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-6prc7"
Sep 16 19:45:28.089135 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:28.089106    2577 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-86b39b6d-ce90-4025-823c-c0244a45ed72\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09b88cffeeadd7a3d\") pod \"e2e-pvc-lora-download-6prc7\" (UID: \"d9f182a2-6394-488b-9a26-489578ea4818\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-6prc7"
Sep 16 19:45:28.181670 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:28.181640    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-86b39b6d-ce90-4025-823c-c0244a45ed72\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09b88cffeeadd7a3d\") pod \"e2e-pvc-lora-download-6prc7\" (UID: \"d9f182a2-6394-488b-9a26-489578ea4818\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-6prc7"
Sep 16 19:45:28.181834 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:28.181775    2577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-86b39b6d-ce90-4025-823c-c0244a45ed72\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09b88cffeeadd7a3d\") pod \"e2e-pvc-lora-download-6prc7\" (UID: \"d9f182a2-6394-488b-9a26-489578ea4818\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-6prc7"
Sep 16 19:45:28.185153 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:28.185131    2577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-86b39b6d-ce90-4025-823c-c0244a45ed72\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09b88cffeeadd7a3d\") pod \"e2e-pvc-lora-download-6prc7\" (UID: \"d9f182a2-6394-488b-9a26-489578ea4818\") DevicePath \"csi-12d115259df15a592ba58bae550057b0ca43f4237efbef2b18d5f34f7237271b\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-6prc7"
Sep 16 19:45:28.405156 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:28.405124    2577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-86b39b6d-ce90-4025-823c-c0244a45ed72\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09b88cffeeadd7a3d\") pod \"e2e-pvc-lora-download-6prc7\" (UID: \"d9f182a2-6394-488b-9a26-489578ea4818\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/6c63b96b19d1080af6156134d13ea2983e81b0ca8abb7c134acde6f58b7b04b2/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-6prc7"
Sep 16 19:45:28.419479 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:28.419454    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-86b39b6d-ce90-4025-823c-c0244a45ed72\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09b88cffeeadd7a3d\") pod \"e2e-pvc-lora-download-6prc7\" (UID: \"d9f182a2-6394-488b-9a26-489578ea4818\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-6prc7"
Sep 16 19:45:28.645978 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:28.645903    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-6prc7"
Sep 16 19:45:28.947214 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:28.947188    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-6prc7"]
Sep 16 19:45:28.950650 ip-10-0-128-247 kubenswrapper[2577]: W0916 19:45:28.950619    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9f182a2_6394_488b_9a26_489578ea4818.slice/crio-5bb14a1428f3da6f8a9790bf0c9a460424cef199b04806d74d96d5d9ababbd87 WatchSource:0}: Error finding container 5bb14a1428f3da6f8a9790bf0c9a460424cef199b04806d74d96d5d9ababbd87: Status 404 returned error can't find the container with id 5bb14a1428f3da6f8a9790bf0c9a460424cef199b04806d74d96d5d9ababbd87
Sep 16 19:45:29.156058 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:29.156016    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bfcaf50-33db-49da-b6c2-c9f256d3003a" path="/var/lib/kubelet/pods/2bfcaf50-33db-49da-b6c2-c9f256d3003a/volumes"
Sep 16 19:45:29.306750 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:29.306666    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-6prc7" event={"ID":"d9f182a2-6394-488b-9a26-489578ea4818","Type":"ContainerStarted","Data":"c2f730332235fa2b19709646d87e9e60e83021f4465d1eae34f88b22595a56dc"}
Sep 16 19:45:29.306750 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:29.306715    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-6prc7" event={"ID":"d9f182a2-6394-488b-9a26-489578ea4818","Type":"ContainerStarted","Data":"5bb14a1428f3da6f8a9790bf0c9a460424cef199b04806d74d96d5d9ababbd87"}
Sep 16 19:45:29.324343 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:29.324293    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-6prc7" podStartSLOduration=7.324276499 podStartE2EDuration="7.324276499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:45:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:45:29.32337587 +0000 UTC m=+3886.706265234" watchObservedRunningTime="2026-09-16 19:45:29.324276499 +0000 UTC m=+3886.707165864"
Sep 16 19:45:31.319290 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:31.319202    2577 generic.go:356] "Generic (PLEG): container finished" podID="d9f182a2-6394-488b-9a26-489578ea4818" containerID="c2f730332235fa2b19709646d87e9e60e83021f4465d1eae34f88b22595a56dc" exitCode=0
Sep 16 19:45:31.319290 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:31.319270    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-6prc7" event={"ID":"d9f182a2-6394-488b-9a26-489578ea4818","Type":"ContainerDied","Data":"c2f730332235fa2b19709646d87e9e60e83021f4465d1eae34f88b22595a56dc"}
Sep 16 19:45:31.877413 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:31.877370    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk" podUID="ffe28a99-65d0-4c5e-ae54-6e36a621a3c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Sep 16 19:45:32.469060 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:32.469034    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-6prc7"
Sep 16 19:45:32.625037 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:32.624955    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09b88cffeeadd7a3d\") pod \"d9f182a2-6394-488b-9a26-489578ea4818\" (UID: \"d9f182a2-6394-488b-9a26-489578ea4818\") "
Sep 16 19:45:32.625037 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:32.625005    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfmf8\" (UniqueName: \"kubernetes.io/projected/d9f182a2-6394-488b-9a26-489578ea4818-kube-api-access-lfmf8\") pod \"d9f182a2-6394-488b-9a26-489578ea4818\" (UID: \"d9f182a2-6394-488b-9a26-489578ea4818\") "
Sep 16 19:45:32.627580 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:32.627521    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9f182a2-6394-488b-9a26-489578ea4818-kube-api-access-lfmf8" (OuterVolumeSpecName: "kube-api-access-lfmf8") pod "d9f182a2-6394-488b-9a26-489578ea4818" (UID: "d9f182a2-6394-488b-9a26-489578ea4818"). InnerVolumeSpecName "kube-api-access-lfmf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:45:32.627877 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:32.627857    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-09b88cffeeadd7a3d" (OuterVolumeSpecName: "model-storage") pod "d9f182a2-6394-488b-9a26-489578ea4818" (UID: "d9f182a2-6394-488b-9a26-489578ea4818"). InnerVolumeSpecName "pvc-86b39b6d-ce90-4025-823c-c0244a45ed72". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 16 19:45:32.726509 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:32.726465    2577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-86b39b6d-ce90-4025-823c-c0244a45ed72\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09b88cffeeadd7a3d\") on node \"ip-10-0-128-247.ec2.internal\" "
Sep 16 19:45:32.726509 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:32.726503    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfmf8\" (UniqueName: \"kubernetes.io/projected/d9f182a2-6394-488b-9a26-489578ea4818-kube-api-access-lfmf8\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:45:33.330509 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:33.330474    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-6prc7"
Sep 16 19:45:33.330683 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:33.330477    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-6prc7" event={"ID":"d9f182a2-6394-488b-9a26-489578ea4818","Type":"ContainerDied","Data":"5bb14a1428f3da6f8a9790bf0c9a460424cef199b04806d74d96d5d9ababbd87"}
Sep 16 19:45:33.330683 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:33.330575    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5bb14a1428f3da6f8a9790bf0c9a460424cef199b04806d74d96d5d9ababbd87"
Sep 16 19:45:33.696388 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:33.696368    2577 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-86b39b6d-ce90-4025-823c-c0244a45ed72" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-09b88cffeeadd7a3d") on node "ip-10-0-128-247.ec2.internal"
Sep 16 19:45:33.735827 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:33.735794    2577 reconciler_common.go:299] "Volume detached for volume \"pvc-86b39b6d-ce90-4025-823c-c0244a45ed72\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09b88cffeeadd7a3d\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"csi-12d115259df15a592ba58bae550057b0ca43f4237efbef2b18d5f34f7237271b\""
Sep 16 19:45:41.876684 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:41.876642    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk" podUID="ffe28a99-65d0-4c5e-ae54-6e36a621a3c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Sep 16 19:45:51.886950 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:51.886913    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk"
Sep 16 19:45:51.895239 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:45:51.895201    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk"
Sep 16 19:46:11.487786 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:46:11.487753    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk"]
Sep 16 19:46:26.511636 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:46:26.511572    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk" podUID="ffe28a99-65d0-4c5e-ae54-6e36a621a3c0" containerName="main" containerID="cri-o://fd27d0b428623639ebd7851e55e9994615156a7de0201365ee47e3ab7f515ed0" gracePeriod=45
Sep 16 19:46:27.252099 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:46:27.252073    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk"
Sep 16 19:46:27.350857 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:46:27.350789    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-tmp-dir\") pod \"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0\" (UID: \"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0\") "
Sep 16 19:46:27.350857 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:46:27.350827    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-home\") pod \"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0\" (UID: \"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0\") "
Sep 16 19:46:27.350857 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:46:27.350845    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-dshm\") pod \"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0\" (UID: \"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0\") "
Sep 16 19:46:27.351111 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:46:27.350876    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-kserve-provision-location\") pod \"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0\" (UID: \"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0\") "
Sep 16 19:46:27.351111 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:46:27.350930    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-model-cache\") pod \"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0\" (UID: \"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0\") "
Sep 16 19:46:27.351111 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:46:27.350974    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fl8jl\" (UniqueName: \"kubernetes.io/projected/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-kube-api-access-fl8jl\") pod \"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0\" (UID: \"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0\") "
Sep 16 19:46:27.351111 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:46:27.350990    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-tls-certs\") pod \"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0\" (UID: \"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0\") "
Sep 16 19:46:27.351111 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:46:27.351010    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-cabundle-cert\") pod \"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0\" (UID: \"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0\") "
Sep 16 19:46:27.351380 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:46:27.351313    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-model-cache" (OuterVolumeSpecName: "model-cache") pod "ffe28a99-65d0-4c5e-ae54-6e36a621a3c0" (UID: "ffe28a99-65d0-4c5e-ae54-6e36a621a3c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:46:27.351606 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:46:27.351585    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-model-cache\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:46:27.351692 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:46:27.351621    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-home" (OuterVolumeSpecName: "home") pod "ffe28a99-65d0-4c5e-ae54-6e36a621a3c0" (UID: "ffe28a99-65d0-4c5e-ae54-6e36a621a3c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:46:27.351963 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:46:27.351940    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ffe28a99-65d0-4c5e-ae54-6e36a621a3c0" (UID: "ffe28a99-65d0-4c5e-ae54-6e36a621a3c0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:46:27.353492 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:46:27.353458    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ffe28a99-65d0-4c5e-ae54-6e36a621a3c0" (UID: "ffe28a99-65d0-4c5e-ae54-6e36a621a3c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:46:27.353786 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:46:27.353763    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-dshm" (OuterVolumeSpecName: "dshm") pod "ffe28a99-65d0-4c5e-ae54-6e36a621a3c0" (UID: "ffe28a99-65d0-4c5e-ae54-6e36a621a3c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:46:27.354137 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:46:27.354115    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-kube-api-access-fl8jl" (OuterVolumeSpecName: "kube-api-access-fl8jl") pod "ffe28a99-65d0-4c5e-ae54-6e36a621a3c0" (UID: "ffe28a99-65d0-4c5e-ae54-6e36a621a3c0"). InnerVolumeSpecName "kube-api-access-fl8jl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:46:27.372304 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:46:27.372253    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ffe28a99-65d0-4c5e-ae54-6e36a621a3c0" (UID: "ffe28a99-65d0-4c5e-ae54-6e36a621a3c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:46:27.381471 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:46:27.381446    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ffe28a99-65d0-4c5e-ae54-6e36a621a3c0" (UID: "ffe28a99-65d0-4c5e-ae54-6e36a621a3c0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:46:27.452438 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:46:27.452417    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl8jl\" (UniqueName: \"kubernetes.io/projected/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-kube-api-access-fl8jl\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:46:27.452438 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:46:27.452438    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-tls-certs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:46:27.452598 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:46:27.452448    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-cabundle-cert\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:46:27.452598 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:46:27.452457    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-tmp-dir\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:46:27.452598 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:46:27.452464    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-home\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:46:27.452598 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:46:27.452471    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-dshm\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:46:27.452598 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:46:27.452478    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0-kserve-provision-location\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:46:27.580172 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:46:27.580144    2577 generic.go:356] "Generic (PLEG): container finished" podID="ffe28a99-65d0-4c5e-ae54-6e36a621a3c0" containerID="fd27d0b428623639ebd7851e55e9994615156a7de0201365ee47e3ab7f515ed0" exitCode=0
Sep 16 19:46:27.580481 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:46:27.580192    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk"
Sep 16 19:46:27.580481 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:46:27.580216    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk" event={"ID":"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0","Type":"ContainerDied","Data":"fd27d0b428623639ebd7851e55e9994615156a7de0201365ee47e3ab7f515ed0"}
Sep 16 19:46:27.580481 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:46:27.580266    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk" event={"ID":"ffe28a99-65d0-4c5e-ae54-6e36a621a3c0","Type":"ContainerDied","Data":"ee82795229ebac5550dd1ff927f0a4a6e04fd97ee5915a3c9e43fde4d2fadc3b"}
Sep 16 19:46:27.580481 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:46:27.580290    2577 scope.go:117] "RemoveContainer" containerID="fd27d0b428623639ebd7851e55e9994615156a7de0201365ee47e3ab7f515ed0"
Sep 16 19:46:27.589479 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:46:27.589463    2577 scope.go:117] "RemoveContainer" containerID="05d44b82a0354614efa8bf0efd24486ce2a867078a1f83b9e07f8c8d3f280ed5"
Sep 16 19:46:27.597355 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:46:27.597339    2577 scope.go:117] "RemoveContainer" containerID="fd27d0b428623639ebd7851e55e9994615156a7de0201365ee47e3ab7f515ed0"
Sep 16 19:46:27.597622 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:46:27.597604    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd27d0b428623639ebd7851e55e9994615156a7de0201365ee47e3ab7f515ed0\": container with ID starting with fd27d0b428623639ebd7851e55e9994615156a7de0201365ee47e3ab7f515ed0 not found: ID does not exist" containerID="fd27d0b428623639ebd7851e55e9994615156a7de0201365ee47e3ab7f515ed0"
Sep 16 19:46:27.597690 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:46:27.597628    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd27d0b428623639ebd7851e55e9994615156a7de0201365ee47e3ab7f515ed0"} err="failed to get container status \"fd27d0b428623639ebd7851e55e9994615156a7de0201365ee47e3ab7f515ed0\": rpc error: code = NotFound desc = could not find container \"fd27d0b428623639ebd7851e55e9994615156a7de0201365ee47e3ab7f515ed0\": container with ID starting with fd27d0b428623639ebd7851e55e9994615156a7de0201365ee47e3ab7f515ed0 not found: ID does not exist"
Sep 16 19:46:27.597690 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:46:27.597644    2577 scope.go:117] "RemoveContainer" containerID="05d44b82a0354614efa8bf0efd24486ce2a867078a1f83b9e07f8c8d3f280ed5"
Sep 16 19:46:27.597912 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:46:27.597893    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05d44b82a0354614efa8bf0efd24486ce2a867078a1f83b9e07f8c8d3f280ed5\": container with ID starting with 05d44b82a0354614efa8bf0efd24486ce2a867078a1f83b9e07f8c8d3f280ed5 not found: ID does not exist" containerID="05d44b82a0354614efa8bf0efd24486ce2a867078a1f83b9e07f8c8d3f280ed5"
Sep 16 19:46:27.597957 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:46:27.597919    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05d44b82a0354614efa8bf0efd24486ce2a867078a1f83b9e07f8c8d3f280ed5"} err="failed to get container status \"05d44b82a0354614efa8bf0efd24486ce2a867078a1f83b9e07f8c8d3f280ed5\": rpc error: code = NotFound desc = could not find container \"05d44b82a0354614efa8bf0efd24486ce2a867078a1f83b9e07f8c8d3f280ed5\": container with ID starting with 05d44b82a0354614efa8bf0efd24486ce2a867078a1f83b9e07f8c8d3f280ed5 not found: ID does not exist"
Sep 16 19:46:27.605957 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:46:27.605910    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk"]
Sep 16 19:46:27.610336 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:46:27.610309    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5bqbsfk"]
Sep 16 19:46:29.155240 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:46:29.155206    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffe28a99-65d0-4c5e-ae54-6e36a621a3c0" path="/var/lib/kubelet/pods/ffe28a99-65d0-4c5e-ae54-6e36a621a3c0/volumes"
Sep 16 19:47:05.070170 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.070130    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84"]
Sep 16 19:47:05.070780 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.070641    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9f182a2-6394-488b-9a26-489578ea4818" containerName="storage-initializer"
Sep 16 19:47:05.070780 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.070661    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9f182a2-6394-488b-9a26-489578ea4818" containerName="storage-initializer"
Sep 16 19:47:05.070780 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.070677    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffe28a99-65d0-4c5e-ae54-6e36a621a3c0" containerName="storage-initializer"
Sep 16 19:47:05.070780 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.070686    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffe28a99-65d0-4c5e-ae54-6e36a621a3c0" containerName="storage-initializer"
Sep 16 19:47:05.070780 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.070712    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffe28a99-65d0-4c5e-ae54-6e36a621a3c0" containerName="main"
Sep 16 19:47:05.070780 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.070722    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffe28a99-65d0-4c5e-ae54-6e36a621a3c0" containerName="main"
Sep 16 19:47:05.071100 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.070797    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffe28a99-65d0-4c5e-ae54-6e36a621a3c0" containerName="main"
Sep 16 19:47:05.071100 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.070808    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9f182a2-6394-488b-9a26-489578ea4818" containerName="storage-initializer"
Sep 16 19:47:05.074213 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.074192    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84"
Sep 16 19:47:05.079989 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.079893    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 16 19:47:05.079989 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.079912    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 16 19:47:05.079989 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.079953    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 16 19:47:05.080227 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.080025    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:47:05.083094 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.083074    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 16 19:47:05.083454 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.083424    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-cc52p\""
Sep 16 19:47:05.093129 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.093108    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84"]
Sep 16 19:47:05.133572 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.133547    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86bc1462-0270-4d35-b169-058d3332fb6a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84\" (UID: \"86bc1462-0270-4d35-b169-058d3332fb6a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84"
Sep 16 19:47:05.133663 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.133575    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86bc1462-0270-4d35-b169-058d3332fb6a-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84\" (UID: \"86bc1462-0270-4d35-b169-058d3332fb6a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84"
Sep 16 19:47:05.133663 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.133641    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpdtb\" (UniqueName: \"kubernetes.io/projected/86bc1462-0270-4d35-b169-058d3332fb6a-kube-api-access-dpdtb\") pod \"custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84\" (UID: \"86bc1462-0270-4d35-b169-058d3332fb6a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84"
Sep 16 19:47:05.133737 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.133666    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86bc1462-0270-4d35-b169-058d3332fb6a-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84\" (UID: \"86bc1462-0270-4d35-b169-058d3332fb6a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84"
Sep 16 19:47:05.133737 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.133711    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86bc1462-0270-4d35-b169-058d3332fb6a-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84\" (UID: \"86bc1462-0270-4d35-b169-058d3332fb6a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84"
Sep 16 19:47:05.133806 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.133736    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86bc1462-0270-4d35-b169-058d3332fb6a-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84\" (UID: \"86bc1462-0270-4d35-b169-058d3332fb6a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84"
Sep 16 19:47:05.133806 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.133787    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86bc1462-0270-4d35-b169-058d3332fb6a-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84\" (UID: \"86bc1462-0270-4d35-b169-058d3332fb6a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84"
Sep 16 19:47:05.133874 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.133809    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86bc1462-0270-4d35-b169-058d3332fb6a-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84\" (UID: \"86bc1462-0270-4d35-b169-058d3332fb6a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84"
Sep 16 19:47:05.234604 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.234580    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86bc1462-0270-4d35-b169-058d3332fb6a-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84\" (UID: \"86bc1462-0270-4d35-b169-058d3332fb6a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84"
Sep 16 19:47:05.234711 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.234613    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86bc1462-0270-4d35-b169-058d3332fb6a-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84\" (UID: \"86bc1462-0270-4d35-b169-058d3332fb6a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84"
Sep 16 19:47:05.234711 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.234632    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86bc1462-0270-4d35-b169-058d3332fb6a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84\" (UID: \"86bc1462-0270-4d35-b169-058d3332fb6a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84"
Sep 16 19:47:05.234711 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.234648    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86bc1462-0270-4d35-b169-058d3332fb6a-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84\" (UID: \"86bc1462-0270-4d35-b169-058d3332fb6a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84"
Sep 16 19:47:05.234870 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.234780    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpdtb\" (UniqueName: \"kubernetes.io/projected/86bc1462-0270-4d35-b169-058d3332fb6a-kube-api-access-dpdtb\") pod \"custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84\" (UID: \"86bc1462-0270-4d35-b169-058d3332fb6a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84"
Sep 16 19:47:05.234870 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.234818    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86bc1462-0270-4d35-b169-058d3332fb6a-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84\" (UID: \"86bc1462-0270-4d35-b169-058d3332fb6a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84"
Sep 16 19:47:05.234870 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.234862    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86bc1462-0270-4d35-b169-058d3332fb6a-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84\" (UID: \"86bc1462-0270-4d35-b169-058d3332fb6a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84"
Sep 16 19:47:05.235035 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.234900    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86bc1462-0270-4d35-b169-058d3332fb6a-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84\" (UID: \"86bc1462-0270-4d35-b169-058d3332fb6a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84"
Sep 16 19:47:05.235221 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.235108    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86bc1462-0270-4d35-b169-058d3332fb6a-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84\" (UID: \"86bc1462-0270-4d35-b169-058d3332fb6a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84"
Sep 16 19:47:05.235221 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.235176    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86bc1462-0270-4d35-b169-058d3332fb6a-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84\" (UID: \"86bc1462-0270-4d35-b169-058d3332fb6a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84"
Sep 16 19:47:05.235354 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.235235    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86bc1462-0270-4d35-b169-058d3332fb6a-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84\" (UID: \"86bc1462-0270-4d35-b169-058d3332fb6a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84"
Sep 16 19:47:05.235354 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.235312    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86bc1462-0270-4d35-b169-058d3332fb6a-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84\" (UID: \"86bc1462-0270-4d35-b169-058d3332fb6a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84"
Sep 16 19:47:05.235439 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.235388    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86bc1462-0270-4d35-b169-058d3332fb6a-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84\" (UID: \"86bc1462-0270-4d35-b169-058d3332fb6a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84"
Sep 16 19:47:05.237044 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.237016    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86bc1462-0270-4d35-b169-058d3332fb6a-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84\" (UID: \"86bc1462-0270-4d35-b169-058d3332fb6a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84"
Sep 16 19:47:05.237320 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.237301    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86bc1462-0270-4d35-b169-058d3332fb6a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84\" (UID: \"86bc1462-0270-4d35-b169-058d3332fb6a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84"
Sep 16 19:47:05.244716 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.244694    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpdtb\" (UniqueName: \"kubernetes.io/projected/86bc1462-0270-4d35-b169-058d3332fb6a-kube-api-access-dpdtb\") pod \"custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84\" (UID: \"86bc1462-0270-4d35-b169-058d3332fb6a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84"
Sep 16 19:47:05.384791 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.384766    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84"
Sep 16 19:47:05.663877 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.663849    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84"]
Sep 16 19:47:05.666249 ip-10-0-128-247 kubenswrapper[2577]: W0916 19:47:05.666226    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86bc1462_0270_4d35_b169_058d3332fb6a.slice/crio-205bc724c0bc78d3547cf0ede879fc3eea440c161360942d5a83debbd001f46c WatchSource:0}: Error finding container 205bc724c0bc78d3547cf0ede879fc3eea440c161360942d5a83debbd001f46c: Status 404 returned error can't find the container with id 205bc724c0bc78d3547cf0ede879fc3eea440c161360942d5a83debbd001f46c
Sep 16 19:47:05.752712 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.752675    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84" event={"ID":"86bc1462-0270-4d35-b169-058d3332fb6a","Type":"ContainerStarted","Data":"f6b32d7e8805d20fb14a0a85f62a4e111ad59c3674310c8cb7611f1f60b870b4"}
Sep 16 19:47:05.752846 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:05.752731    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84" event={"ID":"86bc1462-0270-4d35-b169-058d3332fb6a","Type":"ContainerStarted","Data":"205bc724c0bc78d3547cf0ede879fc3eea440c161360942d5a83debbd001f46c"}
Sep 16 19:47:08.768290 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:08.768250    2577 generic.go:356] "Generic (PLEG): container finished" podID="86bc1462-0270-4d35-b169-058d3332fb6a" containerID="f6b32d7e8805d20fb14a0a85f62a4e111ad59c3674310c8cb7611f1f60b870b4" exitCode=0
Sep 16 19:47:08.768857 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:08.768332    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84" event={"ID":"86bc1462-0270-4d35-b169-058d3332fb6a","Type":"ContainerDied","Data":"f6b32d7e8805d20fb14a0a85f62a4e111ad59c3674310c8cb7611f1f60b870b4"}
Sep 16 19:47:09.775901 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:09.775861    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84" event={"ID":"86bc1462-0270-4d35-b169-058d3332fb6a","Type":"ContainerStarted","Data":"b417857b439a17f9419598ba7ade223e838cf5fe02ac2577efd2b82dec5abd19"}
Sep 16 19:47:09.810028 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:09.809978    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84" podStartSLOduration=2.775348557 podStartE2EDuration="4.809962539s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.034613982s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:47:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:47:09.809179124 +0000 UTC m=+3987.192068493" watchObservedRunningTime="2026-09-16 19:47:09.809962539 +0000 UTC m=+3987.192851903"
Sep 16 19:47:15.385817 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:15.385781    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84"
Sep 16 19:47:15.386349 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:15.385841    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84"
Sep 16 19:47:15.387704 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:15.387670    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84" podUID="86bc1462-0270-4d35-b169-058d3332fb6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 16 19:47:25.385828 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:25.385774    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84" podUID="86bc1462-0270-4d35-b169-058d3332fb6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 16 19:47:35.386130 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:35.386087    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84" podUID="86bc1462-0270-4d35-b169-058d3332fb6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 16 19:47:45.386068 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:45.386020    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84" podUID="86bc1462-0270-4d35-b169-058d3332fb6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 16 19:47:55.385793 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:47:55.385743    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84" podUID="86bc1462-0270-4d35-b169-058d3332fb6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 16 19:48:05.385878 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:05.385788    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84" podUID="86bc1462-0270-4d35-b169-058d3332fb6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 16 19:48:15.386057 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:15.386016    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84" podUID="86bc1462-0270-4d35-b169-058d3332fb6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 16 19:48:19.852638 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:19.852602    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-6prc7"]
Sep 16 19:48:19.854138 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:19.854117    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-6prc7"]
Sep 16 19:48:21.155575 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:21.155509    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9f182a2-6394-488b-9a26-489578ea4818" path="/var/lib/kubelet/pods/d9f182a2-6394-488b-9a26-489578ea4818/volumes"
Sep 16 19:48:25.385361 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:25.385316    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84" podUID="86bc1462-0270-4d35-b169-058d3332fb6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 16 19:48:35.386105 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:35.386055    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84" podUID="86bc1462-0270-4d35-b169-058d3332fb6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 16 19:48:45.385433 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:45.385384    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84" podUID="86bc1462-0270-4d35-b169-058d3332fb6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 16 19:48:48.496650 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:48.496612    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb"]
Sep 16 19:48:48.502893 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:48.502871    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb"
Sep 16 19:48:48.505669 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:48.505640    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:48:48.505789 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:48.505643    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 16 19:48:48.505876 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:48.505855    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-d6svp\""
Sep 16 19:48:48.505943 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:48.505860    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"seaweedfs-s3-creds\""
Sep 16 19:48:48.506072 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:48.506053    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 16 19:48:48.506201 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:48.506187    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 16 19:48:48.513770 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:48.513747    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb"]
Sep 16 19:48:48.587718 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:48.587685    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/200472d0-f6f0-4234-8888-dd0f65ed8b01-model-cache\") pod \"tls-verification-test-kserve-98c9d6556-4ttdb\" (UID: \"200472d0-f6f0-4234-8888-dd0f65ed8b01\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb"
Sep 16 19:48:48.587921 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:48.587731    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/200472d0-f6f0-4234-8888-dd0f65ed8b01-dshm\") pod \"tls-verification-test-kserve-98c9d6556-4ttdb\" (UID: \"200472d0-f6f0-4234-8888-dd0f65ed8b01\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb"
Sep 16 19:48:48.587921 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:48.587821    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/200472d0-f6f0-4234-8888-dd0f65ed8b01-kserve-provision-location\") pod \"tls-verification-test-kserve-98c9d6556-4ttdb\" (UID: \"200472d0-f6f0-4234-8888-dd0f65ed8b01\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb"
Sep 16 19:48:48.587921 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:48.587882    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsxv6\" (UniqueName: \"kubernetes.io/projected/200472d0-f6f0-4234-8888-dd0f65ed8b01-kube-api-access-xsxv6\") pod \"tls-verification-test-kserve-98c9d6556-4ttdb\" (UID: \"200472d0-f6f0-4234-8888-dd0f65ed8b01\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb"
Sep 16 19:48:48.587921 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:48.587905    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/200472d0-f6f0-4234-8888-dd0f65ed8b01-tls-certs\") pod \"tls-verification-test-kserve-98c9d6556-4ttdb\" (UID: \"200472d0-f6f0-4234-8888-dd0f65ed8b01\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb"
Sep 16 19:48:48.588084 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:48.587974    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/200472d0-f6f0-4234-8888-dd0f65ed8b01-cabundle-cert\") pod \"tls-verification-test-kserve-98c9d6556-4ttdb\" (UID: \"200472d0-f6f0-4234-8888-dd0f65ed8b01\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb"
Sep 16 19:48:48.588084 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:48.588008    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/200472d0-f6f0-4234-8888-dd0f65ed8b01-tmp-dir\") pod \"tls-verification-test-kserve-98c9d6556-4ttdb\" (UID: \"200472d0-f6f0-4234-8888-dd0f65ed8b01\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb"
Sep 16 19:48:48.588084 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:48.588045    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/200472d0-f6f0-4234-8888-dd0f65ed8b01-home\") pod \"tls-verification-test-kserve-98c9d6556-4ttdb\" (UID: \"200472d0-f6f0-4234-8888-dd0f65ed8b01\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb"
Sep 16 19:48:48.688475 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:48.688430    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/200472d0-f6f0-4234-8888-dd0f65ed8b01-kserve-provision-location\") pod \"tls-verification-test-kserve-98c9d6556-4ttdb\" (UID: \"200472d0-f6f0-4234-8888-dd0f65ed8b01\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb"
Sep 16 19:48:48.688679 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:48.688489    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsxv6\" (UniqueName: \"kubernetes.io/projected/200472d0-f6f0-4234-8888-dd0f65ed8b01-kube-api-access-xsxv6\") pod \"tls-verification-test-kserve-98c9d6556-4ttdb\" (UID: \"200472d0-f6f0-4234-8888-dd0f65ed8b01\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb"
Sep 16 19:48:48.688679 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:48.688613    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/200472d0-f6f0-4234-8888-dd0f65ed8b01-tls-certs\") pod \"tls-verification-test-kserve-98c9d6556-4ttdb\" (UID: \"200472d0-f6f0-4234-8888-dd0f65ed8b01\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb"
Sep 16 19:48:48.688784 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:48.688682    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/200472d0-f6f0-4234-8888-dd0f65ed8b01-cabundle-cert\") pod \"tls-verification-test-kserve-98c9d6556-4ttdb\" (UID: \"200472d0-f6f0-4234-8888-dd0f65ed8b01\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb"
Sep 16 19:48:48.688784 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:48.688717    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/200472d0-f6f0-4234-8888-dd0f65ed8b01-tmp-dir\") pod \"tls-verification-test-kserve-98c9d6556-4ttdb\" (UID: \"200472d0-f6f0-4234-8888-dd0f65ed8b01\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb"
Sep 16 19:48:48.688784 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:48.688760    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/200472d0-f6f0-4234-8888-dd0f65ed8b01-home\") pod \"tls-verification-test-kserve-98c9d6556-4ttdb\" (UID: \"200472d0-f6f0-4234-8888-dd0f65ed8b01\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb"
Sep 16 19:48:48.688932 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:48.688790    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/200472d0-f6f0-4234-8888-dd0f65ed8b01-model-cache\") pod \"tls-verification-test-kserve-98c9d6556-4ttdb\" (UID: \"200472d0-f6f0-4234-8888-dd0f65ed8b01\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb"
Sep 16 19:48:48.688932 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:48.688854    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/200472d0-f6f0-4234-8888-dd0f65ed8b01-dshm\") pod \"tls-verification-test-kserve-98c9d6556-4ttdb\" (UID: \"200472d0-f6f0-4234-8888-dd0f65ed8b01\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb"
Sep 16 19:48:48.689042 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:48.688957    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/200472d0-f6f0-4234-8888-dd0f65ed8b01-kserve-provision-location\") pod \"tls-verification-test-kserve-98c9d6556-4ttdb\" (UID: \"200472d0-f6f0-4234-8888-dd0f65ed8b01\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb"
Sep 16 19:48:48.689144 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:48.689116    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/200472d0-f6f0-4234-8888-dd0f65ed8b01-home\") pod \"tls-verification-test-kserve-98c9d6556-4ttdb\" (UID: \"200472d0-f6f0-4234-8888-dd0f65ed8b01\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb"
Sep 16 19:48:48.689310 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:48.689182    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/200472d0-f6f0-4234-8888-dd0f65ed8b01-tmp-dir\") pod \"tls-verification-test-kserve-98c9d6556-4ttdb\" (UID: \"200472d0-f6f0-4234-8888-dd0f65ed8b01\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb"
Sep 16 19:48:48.689386 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:48.689345    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/200472d0-f6f0-4234-8888-dd0f65ed8b01-model-cache\") pod \"tls-verification-test-kserve-98c9d6556-4ttdb\" (UID: \"200472d0-f6f0-4234-8888-dd0f65ed8b01\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb"
Sep 16 19:48:48.689564 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:48.689478    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/200472d0-f6f0-4234-8888-dd0f65ed8b01-cabundle-cert\") pod \"tls-verification-test-kserve-98c9d6556-4ttdb\" (UID: \"200472d0-f6f0-4234-8888-dd0f65ed8b01\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb"
Sep 16 19:48:48.691266 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:48.691246    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/200472d0-f6f0-4234-8888-dd0f65ed8b01-dshm\") pod \"tls-verification-test-kserve-98c9d6556-4ttdb\" (UID: \"200472d0-f6f0-4234-8888-dd0f65ed8b01\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb"
Sep 16 19:48:48.691725 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:48.691703    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/200472d0-f6f0-4234-8888-dd0f65ed8b01-tls-certs\") pod \"tls-verification-test-kserve-98c9d6556-4ttdb\" (UID: \"200472d0-f6f0-4234-8888-dd0f65ed8b01\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb"
Sep 16 19:48:48.698855 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:48.698830    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsxv6\" (UniqueName: \"kubernetes.io/projected/200472d0-f6f0-4234-8888-dd0f65ed8b01-kube-api-access-xsxv6\") pod \"tls-verification-test-kserve-98c9d6556-4ttdb\" (UID: \"200472d0-f6f0-4234-8888-dd0f65ed8b01\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb"
Sep 16 19:48:48.816281 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:48.816184    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb"
Sep 16 19:48:49.098017 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:49.097936    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb"]
Sep 16 19:48:49.101244 ip-10-0-128-247 kubenswrapper[2577]: W0916 19:48:49.101214    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod200472d0_f6f0_4234_8888_dd0f65ed8b01.slice/crio-28576f2b773ea217578f0b65ec98a04ab0a16112e58ff55b4f4285a999118b85 WatchSource:0}: Error finding container 28576f2b773ea217578f0b65ec98a04ab0a16112e58ff55b4f4285a999118b85: Status 404 returned error can't find the container with id 28576f2b773ea217578f0b65ec98a04ab0a16112e58ff55b4f4285a999118b85
Sep 16 19:48:49.247993 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:49.247951    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb" event={"ID":"200472d0-f6f0-4234-8888-dd0f65ed8b01","Type":"ContainerStarted","Data":"dd0db65de924843cedcb0ea5f09d698928620dff77a307731cbc9cdef7489fd5"}
Sep 16 19:48:49.248185 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:49.248008    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb" event={"ID":"200472d0-f6f0-4234-8888-dd0f65ed8b01","Type":"ContainerStarted","Data":"28576f2b773ea217578f0b65ec98a04ab0a16112e58ff55b4f4285a999118b85"}
Sep 16 19:48:52.264930 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:52.264895    2577 generic.go:356] "Generic (PLEG): container finished" podID="200472d0-f6f0-4234-8888-dd0f65ed8b01" containerID="dd0db65de924843cedcb0ea5f09d698928620dff77a307731cbc9cdef7489fd5" exitCode=0
Sep 16 19:48:52.265357 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:52.264978    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb" event={"ID":"200472d0-f6f0-4234-8888-dd0f65ed8b01","Type":"ContainerDied","Data":"dd0db65de924843cedcb0ea5f09d698928620dff77a307731cbc9cdef7489fd5"}
Sep 16 19:48:53.273625 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:53.273590    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb" event={"ID":"200472d0-f6f0-4234-8888-dd0f65ed8b01","Type":"ContainerStarted","Data":"d8aad399fd8587d392f0b4a119c6f03104bb762ec5d0175a3562fbaf6f41d32b"}
Sep 16 19:48:53.303936 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:53.303866    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb" podStartSLOduration=3.098448379 podStartE2EDuration="5.3038408s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.205392421s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:48:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:48:53.300896409 +0000 UTC m=+4090.683785783" watchObservedRunningTime="2026-09-16 19:48:53.3038408 +0000 UTC m=+4090.686730182"
Sep 16 19:48:53.816543 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:53.816496    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb"
Sep 16 19:48:55.385796 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:55.385737    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84" podUID="86bc1462-0270-4d35-b169-058d3332fb6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 16 19:48:58.817291 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:58.817258    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb"
Sep 16 19:48:58.818826 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:48:58.818795    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb" podUID="200472d0-f6f0-4234-8888-dd0f65ed8b01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 16 19:49:05.395931 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:05.395897    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84"
Sep 16 19:49:05.403871 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:05.403838    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84"
Sep 16 19:49:08.817231 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:08.817185    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb" podUID="200472d0-f6f0-4234-8888-dd0f65ed8b01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 16 19:49:15.455591 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:15.455557    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84"]
Sep 16 19:49:18.817173 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:18.817131    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb" podUID="200472d0-f6f0-4234-8888-dd0f65ed8b01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 16 19:49:28.817426 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:28.817380    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb" podUID="200472d0-f6f0-4234-8888-dd0f65ed8b01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 16 19:49:30.478250 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:30.478210    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84" podUID="86bc1462-0270-4d35-b169-058d3332fb6a" containerName="main" containerID="cri-o://b417857b439a17f9419598ba7ade223e838cf5fe02ac2577efd2b82dec5abd19" gracePeriod=45
Sep 16 19:49:31.240249 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:31.240220    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84"
Sep 16 19:49:31.347668 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:31.347576    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86bc1462-0270-4d35-b169-058d3332fb6a-tls-certs\") pod \"86bc1462-0270-4d35-b169-058d3332fb6a\" (UID: \"86bc1462-0270-4d35-b169-058d3332fb6a\") "
Sep 16 19:49:31.347668 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:31.347626    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86bc1462-0270-4d35-b169-058d3332fb6a-cabundle-cert\") pod \"86bc1462-0270-4d35-b169-058d3332fb6a\" (UID: \"86bc1462-0270-4d35-b169-058d3332fb6a\") "
Sep 16 19:49:31.347668 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:31.347666    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86bc1462-0270-4d35-b169-058d3332fb6a-home\") pod \"86bc1462-0270-4d35-b169-058d3332fb6a\" (UID: \"86bc1462-0270-4d35-b169-058d3332fb6a\") "
Sep 16 19:49:31.347961 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:31.347682    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86bc1462-0270-4d35-b169-058d3332fb6a-dshm\") pod \"86bc1462-0270-4d35-b169-058d3332fb6a\" (UID: \"86bc1462-0270-4d35-b169-058d3332fb6a\") "
Sep 16 19:49:31.347961 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:31.347724    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86bc1462-0270-4d35-b169-058d3332fb6a-model-cache\") pod \"86bc1462-0270-4d35-b169-058d3332fb6a\" (UID: \"86bc1462-0270-4d35-b169-058d3332fb6a\") "
Sep 16 19:49:31.347961 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:31.347747    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86bc1462-0270-4d35-b169-058d3332fb6a-kserve-provision-location\") pod \"86bc1462-0270-4d35-b169-058d3332fb6a\" (UID: \"86bc1462-0270-4d35-b169-058d3332fb6a\") "
Sep 16 19:49:31.347961 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:31.347787    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpdtb\" (UniqueName: \"kubernetes.io/projected/86bc1462-0270-4d35-b169-058d3332fb6a-kube-api-access-dpdtb\") pod \"86bc1462-0270-4d35-b169-058d3332fb6a\" (UID: \"86bc1462-0270-4d35-b169-058d3332fb6a\") "
Sep 16 19:49:31.347961 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:31.347845    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86bc1462-0270-4d35-b169-058d3332fb6a-tmp-dir\") pod \"86bc1462-0270-4d35-b169-058d3332fb6a\" (UID: \"86bc1462-0270-4d35-b169-058d3332fb6a\") "
Sep 16 19:49:31.348244 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:31.347963    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86bc1462-0270-4d35-b169-058d3332fb6a-model-cache" (OuterVolumeSpecName: "model-cache") pod "86bc1462-0270-4d35-b169-058d3332fb6a" (UID: "86bc1462-0270-4d35-b169-058d3332fb6a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:49:31.348244 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:31.348086    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86bc1462-0270-4d35-b169-058d3332fb6a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "86bc1462-0270-4d35-b169-058d3332fb6a" (UID: "86bc1462-0270-4d35-b169-058d3332fb6a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:49:31.348244 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:31.348148    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86bc1462-0270-4d35-b169-058d3332fb6a-model-cache\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:49:31.348444 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:31.348418    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86bc1462-0270-4d35-b169-058d3332fb6a-home" (OuterVolumeSpecName: "home") pod "86bc1462-0270-4d35-b169-058d3332fb6a" (UID: "86bc1462-0270-4d35-b169-058d3332fb6a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:49:31.350166 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:31.350140    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86bc1462-0270-4d35-b169-058d3332fb6a-dshm" (OuterVolumeSpecName: "dshm") pod "86bc1462-0270-4d35-b169-058d3332fb6a" (UID: "86bc1462-0270-4d35-b169-058d3332fb6a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:49:31.350628 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:31.350588    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86bc1462-0270-4d35-b169-058d3332fb6a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "86bc1462-0270-4d35-b169-058d3332fb6a" (UID: "86bc1462-0270-4d35-b169-058d3332fb6a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:49:31.350751 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:31.350681    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86bc1462-0270-4d35-b169-058d3332fb6a-kube-api-access-dpdtb" (OuterVolumeSpecName: "kube-api-access-dpdtb") pod "86bc1462-0270-4d35-b169-058d3332fb6a" (UID: "86bc1462-0270-4d35-b169-058d3332fb6a"). InnerVolumeSpecName "kube-api-access-dpdtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:49:31.367457 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:31.367420    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86bc1462-0270-4d35-b169-058d3332fb6a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "86bc1462-0270-4d35-b169-058d3332fb6a" (UID: "86bc1462-0270-4d35-b169-058d3332fb6a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:49:31.378739 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:31.378711    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86bc1462-0270-4d35-b169-058d3332fb6a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "86bc1462-0270-4d35-b169-058d3332fb6a" (UID: "86bc1462-0270-4d35-b169-058d3332fb6a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:49:31.449436 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:31.449393    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86bc1462-0270-4d35-b169-058d3332fb6a-cabundle-cert\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:49:31.449436 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:31.449430    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86bc1462-0270-4d35-b169-058d3332fb6a-home\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:49:31.449436 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:31.449439    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86bc1462-0270-4d35-b169-058d3332fb6a-dshm\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:49:31.449790 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:31.449448    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86bc1462-0270-4d35-b169-058d3332fb6a-kserve-provision-location\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:49:31.449790 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:31.449457    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpdtb\" (UniqueName: \"kubernetes.io/projected/86bc1462-0270-4d35-b169-058d3332fb6a-kube-api-access-dpdtb\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:49:31.449790 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:31.449480    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86bc1462-0270-4d35-b169-058d3332fb6a-tmp-dir\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:49:31.449790 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:31.449489    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86bc1462-0270-4d35-b169-058d3332fb6a-tls-certs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:49:31.474400 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:31.474359    2577 generic.go:356] "Generic (PLEG): container finished" podID="86bc1462-0270-4d35-b169-058d3332fb6a" containerID="b417857b439a17f9419598ba7ade223e838cf5fe02ac2577efd2b82dec5abd19" exitCode=0
Sep 16 19:49:31.474577 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:31.474420    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84"
Sep 16 19:49:31.474577 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:31.474446    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84" event={"ID":"86bc1462-0270-4d35-b169-058d3332fb6a","Type":"ContainerDied","Data":"b417857b439a17f9419598ba7ade223e838cf5fe02ac2577efd2b82dec5abd19"}
Sep 16 19:49:31.474577 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:31.474503    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84" event={"ID":"86bc1462-0270-4d35-b169-058d3332fb6a","Type":"ContainerDied","Data":"205bc724c0bc78d3547cf0ede879fc3eea440c161360942d5a83debbd001f46c"}
Sep 16 19:49:31.474577 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:31.474543    2577 scope.go:117] "RemoveContainer" containerID="b417857b439a17f9419598ba7ade223e838cf5fe02ac2577efd2b82dec5abd19"
Sep 16 19:49:31.484320 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:31.484141    2577 scope.go:117] "RemoveContainer" containerID="f6b32d7e8805d20fb14a0a85f62a4e111ad59c3674310c8cb7611f1f60b870b4"
Sep 16 19:49:31.491920 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:31.491899    2577 scope.go:117] "RemoveContainer" containerID="b417857b439a17f9419598ba7ade223e838cf5fe02ac2577efd2b82dec5abd19"
Sep 16 19:49:31.492167 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:49:31.492150    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b417857b439a17f9419598ba7ade223e838cf5fe02ac2577efd2b82dec5abd19\": container with ID starting with b417857b439a17f9419598ba7ade223e838cf5fe02ac2577efd2b82dec5abd19 not found: ID does not exist" containerID="b417857b439a17f9419598ba7ade223e838cf5fe02ac2577efd2b82dec5abd19"
Sep 16 19:49:31.492244 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:31.492176    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b417857b439a17f9419598ba7ade223e838cf5fe02ac2577efd2b82dec5abd19"} err="failed to get container status \"b417857b439a17f9419598ba7ade223e838cf5fe02ac2577efd2b82dec5abd19\": rpc error: code = NotFound desc = could not find container \"b417857b439a17f9419598ba7ade223e838cf5fe02ac2577efd2b82dec5abd19\": container with ID starting with b417857b439a17f9419598ba7ade223e838cf5fe02ac2577efd2b82dec5abd19 not found: ID does not exist"
Sep 16 19:49:31.492244 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:31.492192    2577 scope.go:117] "RemoveContainer" containerID="f6b32d7e8805d20fb14a0a85f62a4e111ad59c3674310c8cb7611f1f60b870b4"
Sep 16 19:49:31.492446 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:49:31.492422    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6b32d7e8805d20fb14a0a85f62a4e111ad59c3674310c8cb7611f1f60b870b4\": container with ID starting with f6b32d7e8805d20fb14a0a85f62a4e111ad59c3674310c8cb7611f1f60b870b4 not found: ID does not exist" containerID="f6b32d7e8805d20fb14a0a85f62a4e111ad59c3674310c8cb7611f1f60b870b4"
Sep 16 19:49:31.492489 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:31.492458    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6b32d7e8805d20fb14a0a85f62a4e111ad59c3674310c8cb7611f1f60b870b4"} err="failed to get container status \"f6b32d7e8805d20fb14a0a85f62a4e111ad59c3674310c8cb7611f1f60b870b4\": rpc error: code = NotFound desc = could not find container \"f6b32d7e8805d20fb14a0a85f62a4e111ad59c3674310c8cb7611f1f60b870b4\": container with ID starting with f6b32d7e8805d20fb14a0a85f62a4e111ad59c3674310c8cb7611f1f60b870b4 not found: ID does not exist"
Sep 16 19:49:31.514635 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:31.514603    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84"]
Sep 16 19:49:31.531039 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:31.531014    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-8678696b9-45t84"]
Sep 16 19:49:33.155190 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:33.155149    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86bc1462-0270-4d35-b169-058d3332fb6a" path="/var/lib/kubelet/pods/86bc1462-0270-4d35-b169-058d3332fb6a/volumes"
Sep 16 19:49:38.817762 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:38.817675    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb" podUID="200472d0-f6f0-4234-8888-dd0f65ed8b01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 16 19:49:43.632765 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:43.632722    2577 scope.go:117] "RemoveContainer" containerID="e4b18f71be231a546a89bf66e0471bce3f9e7e1bceb573f3b64e98c1fbeef3de"
Sep 16 19:49:48.817051 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:48.817008    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb" podUID="200472d0-f6f0-4234-8888-dd0f65ed8b01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 16 19:49:58.816822 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:49:58.816776    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb" podUID="200472d0-f6f0-4234-8888-dd0f65ed8b01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 16 19:50:08.816753 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:08.816701    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb" podUID="200472d0-f6f0-4234-8888-dd0f65ed8b01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 16 19:50:14.567515 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:14.567478    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz"]
Sep 16 19:50:14.567915 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:14.567866    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86bc1462-0270-4d35-b169-058d3332fb6a" containerName="storage-initializer"
Sep 16 19:50:14.567915 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:14.567885    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="86bc1462-0270-4d35-b169-058d3332fb6a" containerName="storage-initializer"
Sep 16 19:50:14.567915 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:14.567909    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86bc1462-0270-4d35-b169-058d3332fb6a" containerName="main"
Sep 16 19:50:14.567915 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:14.567914    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="86bc1462-0270-4d35-b169-058d3332fb6a" containerName="main"
Sep 16 19:50:14.568054 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:14.567997    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="86bc1462-0270-4d35-b169-058d3332fb6a" containerName="main"
Sep 16 19:50:14.570171 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:14.570153    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz"
Sep 16 19:50:14.573200 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:14.573177    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 16 19:50:14.573338 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:14.573215    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 16 19:50:14.573523 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:14.573509    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:50:14.573770 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:14.573754    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 16 19:50:14.574385 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:14.574345    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 16 19:50:14.574561 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:14.574349    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-f7z8n\""
Sep 16 19:50:14.585832 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:14.585812    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz"]
Sep 16 19:50:14.706762 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:14.706729    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e86b081c-0927-44f2-acea-c66ea38ec59c-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz\" (UID: \"e86b081c-0927-44f2-acea-c66ea38ec59c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz"
Sep 16 19:50:14.706762 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:14.706765    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e86b081c-0927-44f2-acea-c66ea38ec59c-home\") pod \"router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz\" (UID: \"e86b081c-0927-44f2-acea-c66ea38ec59c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz"
Sep 16 19:50:14.706969 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:14.706785    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e86b081c-0927-44f2-acea-c66ea38ec59c-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz\" (UID: \"e86b081c-0927-44f2-acea-c66ea38ec59c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz"
Sep 16 19:50:14.706969 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:14.706852    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e86b081c-0927-44f2-acea-c66ea38ec59c-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz\" (UID: \"e86b081c-0927-44f2-acea-c66ea38ec59c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz"
Sep 16 19:50:14.706969 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:14.706891    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e86b081c-0927-44f2-acea-c66ea38ec59c-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz\" (UID: \"e86b081c-0927-44f2-acea-c66ea38ec59c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz"
Sep 16 19:50:14.706969 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:14.706938    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e86b081c-0927-44f2-acea-c66ea38ec59c-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz\" (UID: \"e86b081c-0927-44f2-acea-c66ea38ec59c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz"
Sep 16 19:50:14.707104 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:14.707027    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnwnk\" (UniqueName: \"kubernetes.io/projected/e86b081c-0927-44f2-acea-c66ea38ec59c-kube-api-access-jnwnk\") pod \"router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz\" (UID: \"e86b081c-0927-44f2-acea-c66ea38ec59c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz"
Sep 16 19:50:14.707104 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:14.707080    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e86b081c-0927-44f2-acea-c66ea38ec59c-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz\" (UID: \"e86b081c-0927-44f2-acea-c66ea38ec59c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz"
Sep 16 19:50:14.808486 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:14.807904    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e86b081c-0927-44f2-acea-c66ea38ec59c-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz\" (UID: \"e86b081c-0927-44f2-acea-c66ea38ec59c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz"
Sep 16 19:50:14.808486 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:14.807980    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnwnk\" (UniqueName: \"kubernetes.io/projected/e86b081c-0927-44f2-acea-c66ea38ec59c-kube-api-access-jnwnk\") pod \"router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz\" (UID: \"e86b081c-0927-44f2-acea-c66ea38ec59c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz"
Sep 16 19:50:14.808486 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:14.808013    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e86b081c-0927-44f2-acea-c66ea38ec59c-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz\" (UID: \"e86b081c-0927-44f2-acea-c66ea38ec59c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz"
Sep 16 19:50:14.808486 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:14.808037    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e86b081c-0927-44f2-acea-c66ea38ec59c-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz\" (UID: \"e86b081c-0927-44f2-acea-c66ea38ec59c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz"
Sep 16 19:50:14.808486 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:14.808073    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e86b081c-0927-44f2-acea-c66ea38ec59c-home\") pod \"router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz\" (UID: \"e86b081c-0927-44f2-acea-c66ea38ec59c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz"
Sep 16 19:50:14.808486 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:14.808095    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e86b081c-0927-44f2-acea-c66ea38ec59c-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz\" (UID: \"e86b081c-0927-44f2-acea-c66ea38ec59c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz"
Sep 16 19:50:14.808486 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:14.808127    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e86b081c-0927-44f2-acea-c66ea38ec59c-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz\" (UID: \"e86b081c-0927-44f2-acea-c66ea38ec59c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz"
Sep 16 19:50:14.808486 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:14.808158    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e86b081c-0927-44f2-acea-c66ea38ec59c-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz\" (UID: \"e86b081c-0927-44f2-acea-c66ea38ec59c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz"
Sep 16 19:50:14.809071 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:14.808794    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e86b081c-0927-44f2-acea-c66ea38ec59c-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz\" (UID: \"e86b081c-0927-44f2-acea-c66ea38ec59c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz"
Sep 16 19:50:14.809071 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:14.808841    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e86b081c-0927-44f2-acea-c66ea38ec59c-home\") pod \"router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz\" (UID: \"e86b081c-0927-44f2-acea-c66ea38ec59c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz"
Sep 16 19:50:14.809184 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:14.809108    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e86b081c-0927-44f2-acea-c66ea38ec59c-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz\" (UID: \"e86b081c-0927-44f2-acea-c66ea38ec59c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz"
Sep 16 19:50:14.809434 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:14.809408    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e86b081c-0927-44f2-acea-c66ea38ec59c-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz\" (UID: \"e86b081c-0927-44f2-acea-c66ea38ec59c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz"
Sep 16 19:50:14.809565 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:14.809495    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e86b081c-0927-44f2-acea-c66ea38ec59c-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz\" (UID: \"e86b081c-0927-44f2-acea-c66ea38ec59c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz"
Sep 16 19:50:14.811131 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:14.811107    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e86b081c-0927-44f2-acea-c66ea38ec59c-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz\" (UID: \"e86b081c-0927-44f2-acea-c66ea38ec59c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz"
Sep 16 19:50:14.811391 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:14.811370    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e86b081c-0927-44f2-acea-c66ea38ec59c-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz\" (UID: \"e86b081c-0927-44f2-acea-c66ea38ec59c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz"
Sep 16 19:50:14.824519 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:14.824464    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnwnk\" (UniqueName: \"kubernetes.io/projected/e86b081c-0927-44f2-acea-c66ea38ec59c-kube-api-access-jnwnk\") pod \"router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz\" (UID: \"e86b081c-0927-44f2-acea-c66ea38ec59c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz"
Sep 16 19:50:14.881458 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:14.881436    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz"
Sep 16 19:50:15.160710 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:15.160672    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz"]
Sep 16 19:50:15.163935 ip-10-0-128-247 kubenswrapper[2577]: W0916 19:50:15.163906    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode86b081c_0927_44f2_acea_c66ea38ec59c.slice/crio-2ea1b90ccea08ba445528e0218f8fcb650fd0aaeff83f3edbca4a2dfc48093ba WatchSource:0}: Error finding container 2ea1b90ccea08ba445528e0218f8fcb650fd0aaeff83f3edbca4a2dfc48093ba: Status 404 returned error can't find the container with id 2ea1b90ccea08ba445528e0218f8fcb650fd0aaeff83f3edbca4a2dfc48093ba
Sep 16 19:50:15.675923 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:15.675887    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz" event={"ID":"e86b081c-0927-44f2-acea-c66ea38ec59c","Type":"ContainerStarted","Data":"0269f53aaf16c0ebd4e05f41324c024377aa82e38fa85e1ddf326037523671e5"}
Sep 16 19:50:15.676297 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:15.675945    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz" event={"ID":"e86b081c-0927-44f2-acea-c66ea38ec59c","Type":"ContainerStarted","Data":"2ea1b90ccea08ba445528e0218f8fcb650fd0aaeff83f3edbca4a2dfc48093ba"}
Sep 16 19:50:17.687867 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:17.687778    2577 generic.go:356] "Generic (PLEG): container finished" podID="e86b081c-0927-44f2-acea-c66ea38ec59c" containerID="0269f53aaf16c0ebd4e05f41324c024377aa82e38fa85e1ddf326037523671e5" exitCode=0
Sep 16 19:50:17.688225 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:17.687865    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz" event={"ID":"e86b081c-0927-44f2-acea-c66ea38ec59c","Type":"ContainerDied","Data":"0269f53aaf16c0ebd4e05f41324c024377aa82e38fa85e1ddf326037523671e5"}
Sep 16 19:50:18.696073 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:18.696040    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz" event={"ID":"e86b081c-0927-44f2-acea-c66ea38ec59c","Type":"ContainerStarted","Data":"84f7438391ed7cf0fdab56d427f919928e512ca9651ea9599baff280f3dadc4a"}
Sep 16 19:50:18.722943 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:18.722891    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz" podStartSLOduration=2.5040680379999998 podStartE2EDuration="4.722871851s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.218803813s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:50:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:50:18.721888196 +0000 UTC m=+4176.104777572" watchObservedRunningTime="2026-09-16 19:50:18.722871851 +0000 UTC m=+4176.105761216"
Sep 16 19:50:18.817731 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:18.817681    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb" podUID="200472d0-f6f0-4234-8888-dd0f65ed8b01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 16 19:50:24.881735 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:24.881694    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz"
Sep 16 19:50:24.882132 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:24.881773    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz"
Sep 16 19:50:24.883595 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:24.883560    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz" podUID="e86b081c-0927-44f2-acea-c66ea38ec59c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Sep 16 19:50:28.816994 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:28.816938    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb" podUID="200472d0-f6f0-4234-8888-dd0f65ed8b01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 16 19:50:34.882510 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:34.882462    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz" podUID="e86b081c-0927-44f2-acea-c66ea38ec59c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Sep 16 19:50:38.816882 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:38.816828    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb" podUID="200472d0-f6f0-4234-8888-dd0f65ed8b01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 16 19:50:44.882090 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:44.882040    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz" podUID="e86b081c-0927-44f2-acea-c66ea38ec59c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Sep 16 19:50:48.826813 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:48.826781    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb"
Sep 16 19:50:48.834508 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:48.834483    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb"
Sep 16 19:50:53.917431 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:53.917395    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb"]
Sep 16 19:50:54.882582 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:50:54.882513    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz" podUID="e86b081c-0927-44f2-acea-c66ea38ec59c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Sep 16 19:51:04.882992 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:04.882899    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz" podUID="e86b081c-0927-44f2-acea-c66ea38ec59c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Sep 16 19:51:08.939881 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:08.939821    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb" podUID="200472d0-f6f0-4234-8888-dd0f65ed8b01" containerName="main" containerID="cri-o://d8aad399fd8587d392f0b4a119c6f03104bb762ec5d0175a3562fbaf6f41d32b" gracePeriod=45
Sep 16 19:51:09.687280 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:09.687252    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb"
Sep 16 19:51:09.781242 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:09.781152    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/200472d0-f6f0-4234-8888-dd0f65ed8b01-tls-certs\") pod \"200472d0-f6f0-4234-8888-dd0f65ed8b01\" (UID: \"200472d0-f6f0-4234-8888-dd0f65ed8b01\") "
Sep 16 19:51:09.781242 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:09.781189    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/200472d0-f6f0-4234-8888-dd0f65ed8b01-tmp-dir\") pod \"200472d0-f6f0-4234-8888-dd0f65ed8b01\" (UID: \"200472d0-f6f0-4234-8888-dd0f65ed8b01\") "
Sep 16 19:51:09.781242 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:09.781223    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/200472d0-f6f0-4234-8888-dd0f65ed8b01-home\") pod \"200472d0-f6f0-4234-8888-dd0f65ed8b01\" (UID: \"200472d0-f6f0-4234-8888-dd0f65ed8b01\") "
Sep 16 19:51:09.781562 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:09.781296    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/200472d0-f6f0-4234-8888-dd0f65ed8b01-model-cache\") pod \"200472d0-f6f0-4234-8888-dd0f65ed8b01\" (UID: \"200472d0-f6f0-4234-8888-dd0f65ed8b01\") "
Sep 16 19:51:09.781562 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:09.781341    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/200472d0-f6f0-4234-8888-dd0f65ed8b01-cabundle-cert\") pod \"200472d0-f6f0-4234-8888-dd0f65ed8b01\" (UID: \"200472d0-f6f0-4234-8888-dd0f65ed8b01\") "
Sep 16 19:51:09.781562 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:09.781377    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/200472d0-f6f0-4234-8888-dd0f65ed8b01-kserve-provision-location\") pod \"200472d0-f6f0-4234-8888-dd0f65ed8b01\" (UID: \"200472d0-f6f0-4234-8888-dd0f65ed8b01\") "
Sep 16 19:51:09.781562 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:09.781403    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/200472d0-f6f0-4234-8888-dd0f65ed8b01-dshm\") pod \"200472d0-f6f0-4234-8888-dd0f65ed8b01\" (UID: \"200472d0-f6f0-4234-8888-dd0f65ed8b01\") "
Sep 16 19:51:09.781562 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:09.781435    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsxv6\" (UniqueName: \"kubernetes.io/projected/200472d0-f6f0-4234-8888-dd0f65ed8b01-kube-api-access-xsxv6\") pod \"200472d0-f6f0-4234-8888-dd0f65ed8b01\" (UID: \"200472d0-f6f0-4234-8888-dd0f65ed8b01\") "
Sep 16 19:51:09.781836 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:09.781610    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/200472d0-f6f0-4234-8888-dd0f65ed8b01-model-cache" (OuterVolumeSpecName: "model-cache") pod "200472d0-f6f0-4234-8888-dd0f65ed8b01" (UID: "200472d0-f6f0-4234-8888-dd0f65ed8b01"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:51:09.781836 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:09.781713    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/200472d0-f6f0-4234-8888-dd0f65ed8b01-model-cache\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:51:09.781836 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:09.781754    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/200472d0-f6f0-4234-8888-dd0f65ed8b01-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "200472d0-f6f0-4234-8888-dd0f65ed8b01" (UID: "200472d0-f6f0-4234-8888-dd0f65ed8b01"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:51:09.782270 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:09.782248    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/200472d0-f6f0-4234-8888-dd0f65ed8b01-home" (OuterVolumeSpecName: "home") pod "200472d0-f6f0-4234-8888-dd0f65ed8b01" (UID: "200472d0-f6f0-4234-8888-dd0f65ed8b01"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:51:09.784462 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:09.784425    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/200472d0-f6f0-4234-8888-dd0f65ed8b01-kube-api-access-xsxv6" (OuterVolumeSpecName: "kube-api-access-xsxv6") pod "200472d0-f6f0-4234-8888-dd0f65ed8b01" (UID: "200472d0-f6f0-4234-8888-dd0f65ed8b01"). InnerVolumeSpecName "kube-api-access-xsxv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:51:09.784601 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:09.784466    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/200472d0-f6f0-4234-8888-dd0f65ed8b01-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "200472d0-f6f0-4234-8888-dd0f65ed8b01" (UID: "200472d0-f6f0-4234-8888-dd0f65ed8b01"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:51:09.784765 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:09.784736    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/200472d0-f6f0-4234-8888-dd0f65ed8b01-dshm" (OuterVolumeSpecName: "dshm") pod "200472d0-f6f0-4234-8888-dd0f65ed8b01" (UID: "200472d0-f6f0-4234-8888-dd0f65ed8b01"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:51:09.799538 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:09.799500    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/200472d0-f6f0-4234-8888-dd0f65ed8b01-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "200472d0-f6f0-4234-8888-dd0f65ed8b01" (UID: "200472d0-f6f0-4234-8888-dd0f65ed8b01"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:51:09.808369 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:09.808341    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/200472d0-f6f0-4234-8888-dd0f65ed8b01-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "200472d0-f6f0-4234-8888-dd0f65ed8b01" (UID: "200472d0-f6f0-4234-8888-dd0f65ed8b01"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:51:09.882274 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:09.882235    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/200472d0-f6f0-4234-8888-dd0f65ed8b01-cabundle-cert\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:51:09.882274 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:09.882267    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/200472d0-f6f0-4234-8888-dd0f65ed8b01-kserve-provision-location\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:51:09.882274 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:09.882278    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/200472d0-f6f0-4234-8888-dd0f65ed8b01-dshm\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:51:09.882511 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:09.882287    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsxv6\" (UniqueName: \"kubernetes.io/projected/200472d0-f6f0-4234-8888-dd0f65ed8b01-kube-api-access-xsxv6\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:51:09.882511 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:09.882296    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/200472d0-f6f0-4234-8888-dd0f65ed8b01-tls-certs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:51:09.882511 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:09.882305    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/200472d0-f6f0-4234-8888-dd0f65ed8b01-tmp-dir\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:51:09.882511 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:09.882311    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/200472d0-f6f0-4234-8888-dd0f65ed8b01-home\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:51:09.957155 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:09.957090    2577 generic.go:356] "Generic (PLEG): container finished" podID="200472d0-f6f0-4234-8888-dd0f65ed8b01" containerID="d8aad399fd8587d392f0b4a119c6f03104bb762ec5d0175a3562fbaf6f41d32b" exitCode=0
Sep 16 19:51:09.957155 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:09.957144    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb"
Sep 16 19:51:09.957155 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:09.957147    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb" event={"ID":"200472d0-f6f0-4234-8888-dd0f65ed8b01","Type":"ContainerDied","Data":"d8aad399fd8587d392f0b4a119c6f03104bb762ec5d0175a3562fbaf6f41d32b"}
Sep 16 19:51:09.957741 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:09.957214    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb" event={"ID":"200472d0-f6f0-4234-8888-dd0f65ed8b01","Type":"ContainerDied","Data":"28576f2b773ea217578f0b65ec98a04ab0a16112e58ff55b4f4285a999118b85"}
Sep 16 19:51:09.957741 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:09.957250    2577 scope.go:117] "RemoveContainer" containerID="d8aad399fd8587d392f0b4a119c6f03104bb762ec5d0175a3562fbaf6f41d32b"
Sep 16 19:51:09.967427 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:09.967400    2577 scope.go:117] "RemoveContainer" containerID="dd0db65de924843cedcb0ea5f09d698928620dff77a307731cbc9cdef7489fd5"
Sep 16 19:51:09.975570 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:09.975548    2577 scope.go:117] "RemoveContainer" containerID="d8aad399fd8587d392f0b4a119c6f03104bb762ec5d0175a3562fbaf6f41d32b"
Sep 16 19:51:09.975871 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:51:09.975849    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8aad399fd8587d392f0b4a119c6f03104bb762ec5d0175a3562fbaf6f41d32b\": container with ID starting with d8aad399fd8587d392f0b4a119c6f03104bb762ec5d0175a3562fbaf6f41d32b not found: ID does not exist" containerID="d8aad399fd8587d392f0b4a119c6f03104bb762ec5d0175a3562fbaf6f41d32b"
Sep 16 19:51:09.975947 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:09.975880    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8aad399fd8587d392f0b4a119c6f03104bb762ec5d0175a3562fbaf6f41d32b"} err="failed to get container status \"d8aad399fd8587d392f0b4a119c6f03104bb762ec5d0175a3562fbaf6f41d32b\": rpc error: code = NotFound desc = could not find container \"d8aad399fd8587d392f0b4a119c6f03104bb762ec5d0175a3562fbaf6f41d32b\": container with ID starting with d8aad399fd8587d392f0b4a119c6f03104bb762ec5d0175a3562fbaf6f41d32b not found: ID does not exist"
Sep 16 19:51:09.975947 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:09.975899    2577 scope.go:117] "RemoveContainer" containerID="dd0db65de924843cedcb0ea5f09d698928620dff77a307731cbc9cdef7489fd5"
Sep 16 19:51:09.976167 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:51:09.976148    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd0db65de924843cedcb0ea5f09d698928620dff77a307731cbc9cdef7489fd5\": container with ID starting with dd0db65de924843cedcb0ea5f09d698928620dff77a307731cbc9cdef7489fd5 not found: ID does not exist" containerID="dd0db65de924843cedcb0ea5f09d698928620dff77a307731cbc9cdef7489fd5"
Sep 16 19:51:09.976217 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:09.976172    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd0db65de924843cedcb0ea5f09d698928620dff77a307731cbc9cdef7489fd5"} err="failed to get container status \"dd0db65de924843cedcb0ea5f09d698928620dff77a307731cbc9cdef7489fd5\": rpc error: code = NotFound desc = could not find container \"dd0db65de924843cedcb0ea5f09d698928620dff77a307731cbc9cdef7489fd5\": container with ID starting with dd0db65de924843cedcb0ea5f09d698928620dff77a307731cbc9cdef7489fd5 not found: ID does not exist"
Sep 16 19:51:09.983543 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:09.983509    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb"]
Sep 16 19:51:09.989485 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:09.989463    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-98c9d6556-4ttdb"]
Sep 16 19:51:11.157729 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:11.157690    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="200472d0-f6f0-4234-8888-dd0f65ed8b01" path="/var/lib/kubelet/pods/200472d0-f6f0-4234-8888-dd0f65ed8b01/volumes"
Sep 16 19:51:14.882522 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:14.882474    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz" podUID="e86b081c-0927-44f2-acea-c66ea38ec59c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Sep 16 19:51:24.882566 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:24.882499    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz" podUID="e86b081c-0927-44f2-acea-c66ea38ec59c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Sep 16 19:51:34.882665 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:34.882623    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz" podUID="e86b081c-0927-44f2-acea-c66ea38ec59c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Sep 16 19:51:43.666974 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:43.666936    2577 scope.go:117] "RemoveContainer" containerID="c2f730332235fa2b19709646d87e9e60e83021f4465d1eae34f88b22595a56dc"
Sep 16 19:51:44.881947 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:44.881907    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz" podUID="e86b081c-0927-44f2-acea-c66ea38ec59c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Sep 16 19:51:48.881277 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:48.881245    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv"]
Sep 16 19:51:48.881787 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:48.881766    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="200472d0-f6f0-4234-8888-dd0f65ed8b01" containerName="main"
Sep 16 19:51:48.881863 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:48.881791    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="200472d0-f6f0-4234-8888-dd0f65ed8b01" containerName="main"
Sep 16 19:51:48.881863 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:48.881827    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="200472d0-f6f0-4234-8888-dd0f65ed8b01" containerName="storage-initializer"
Sep 16 19:51:48.881863 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:48.881836    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="200472d0-f6f0-4234-8888-dd0f65ed8b01" containerName="storage-initializer"
Sep 16 19:51:48.882026 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:48.881918    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="200472d0-f6f0-4234-8888-dd0f65ed8b01" containerName="main"
Sep 16 19:51:48.886579 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:48.886558    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv"
Sep 16 19:51:48.890086 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:48.890063    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 16 19:51:48.890216 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:48.890069    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 16 19:51:48.890216 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:48.890112    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 16 19:51:48.890340 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:48.890266    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:51:48.891180 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:48.891161    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 16 19:51:48.891285 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:48.891207    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-jr7t7\""
Sep 16 19:51:48.903110 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:48.903085    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22be106b-a8a0-447a-b8c5-ee22d46a19ea-dshm\") pod \"prestop-hook-test-kserve-87c8d84b4-677mv\" (UID: \"22be106b-a8a0-447a-b8c5-ee22d46a19ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv"
Sep 16 19:51:48.903211 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:48.903120    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22be106b-a8a0-447a-b8c5-ee22d46a19ea-tmp-dir\") pod \"prestop-hook-test-kserve-87c8d84b4-677mv\" (UID: \"22be106b-a8a0-447a-b8c5-ee22d46a19ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv"
Sep 16 19:51:48.903211 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:48.903140    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22be106b-a8a0-447a-b8c5-ee22d46a19ea-tls-certs\") pod \"prestop-hook-test-kserve-87c8d84b4-677mv\" (UID: \"22be106b-a8a0-447a-b8c5-ee22d46a19ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv"
Sep 16 19:51:48.903211 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:48.903158    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22be106b-a8a0-447a-b8c5-ee22d46a19ea-cabundle-cert\") pod \"prestop-hook-test-kserve-87c8d84b4-677mv\" (UID: \"22be106b-a8a0-447a-b8c5-ee22d46a19ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv"
Sep 16 19:51:48.903211 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:48.903191    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5xct\" (UniqueName: \"kubernetes.io/projected/22be106b-a8a0-447a-b8c5-ee22d46a19ea-kube-api-access-b5xct\") pod \"prestop-hook-test-kserve-87c8d84b4-677mv\" (UID: \"22be106b-a8a0-447a-b8c5-ee22d46a19ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv"
Sep 16 19:51:48.903425 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:48.903223    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22be106b-a8a0-447a-b8c5-ee22d46a19ea-model-cache\") pod \"prestop-hook-test-kserve-87c8d84b4-677mv\" (UID: \"22be106b-a8a0-447a-b8c5-ee22d46a19ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv"
Sep 16 19:51:48.903425 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:48.903271    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22be106b-a8a0-447a-b8c5-ee22d46a19ea-home\") pod \"prestop-hook-test-kserve-87c8d84b4-677mv\" (UID: \"22be106b-a8a0-447a-b8c5-ee22d46a19ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv"
Sep 16 19:51:48.903425 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:48.903287    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22be106b-a8a0-447a-b8c5-ee22d46a19ea-kserve-provision-location\") pod \"prestop-hook-test-kserve-87c8d84b4-677mv\" (UID: \"22be106b-a8a0-447a-b8c5-ee22d46a19ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv"
Sep 16 19:51:48.905981 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:48.905956    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv"]
Sep 16 19:51:49.003872 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:49.003834    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5xct\" (UniqueName: \"kubernetes.io/projected/22be106b-a8a0-447a-b8c5-ee22d46a19ea-kube-api-access-b5xct\") pod \"prestop-hook-test-kserve-87c8d84b4-677mv\" (UID: \"22be106b-a8a0-447a-b8c5-ee22d46a19ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv"
Sep 16 19:51:49.004044 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:49.003899    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22be106b-a8a0-447a-b8c5-ee22d46a19ea-model-cache\") pod \"prestop-hook-test-kserve-87c8d84b4-677mv\" (UID: \"22be106b-a8a0-447a-b8c5-ee22d46a19ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv"
Sep 16 19:51:49.004044 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:49.003946    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22be106b-a8a0-447a-b8c5-ee22d46a19ea-home\") pod \"prestop-hook-test-kserve-87c8d84b4-677mv\" (UID: \"22be106b-a8a0-447a-b8c5-ee22d46a19ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv"
Sep 16 19:51:49.004044 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:49.003970    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22be106b-a8a0-447a-b8c5-ee22d46a19ea-kserve-provision-location\") pod \"prestop-hook-test-kserve-87c8d84b4-677mv\" (UID: \"22be106b-a8a0-447a-b8c5-ee22d46a19ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv"
Sep 16 19:51:49.004044 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:49.004004    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22be106b-a8a0-447a-b8c5-ee22d46a19ea-dshm\") pod \"prestop-hook-test-kserve-87c8d84b4-677mv\" (UID: \"22be106b-a8a0-447a-b8c5-ee22d46a19ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv"
Sep 16 19:51:49.004044 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:49.004034    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22be106b-a8a0-447a-b8c5-ee22d46a19ea-tmp-dir\") pod \"prestop-hook-test-kserve-87c8d84b4-677mv\" (UID: \"22be106b-a8a0-447a-b8c5-ee22d46a19ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv"
Sep 16 19:51:49.004281 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:49.004065    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22be106b-a8a0-447a-b8c5-ee22d46a19ea-tls-certs\") pod \"prestop-hook-test-kserve-87c8d84b4-677mv\" (UID: \"22be106b-a8a0-447a-b8c5-ee22d46a19ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv"
Sep 16 19:51:49.004281 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:49.004095    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22be106b-a8a0-447a-b8c5-ee22d46a19ea-cabundle-cert\") pod \"prestop-hook-test-kserve-87c8d84b4-677mv\" (UID: \"22be106b-a8a0-447a-b8c5-ee22d46a19ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv"
Sep 16 19:51:49.004460 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:49.004424    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22be106b-a8a0-447a-b8c5-ee22d46a19ea-model-cache\") pod \"prestop-hook-test-kserve-87c8d84b4-677mv\" (UID: \"22be106b-a8a0-447a-b8c5-ee22d46a19ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv"
Sep 16 19:51:49.004549 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:49.004483    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22be106b-a8a0-447a-b8c5-ee22d46a19ea-home\") pod \"prestop-hook-test-kserve-87c8d84b4-677mv\" (UID: \"22be106b-a8a0-447a-b8c5-ee22d46a19ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv"
Sep 16 19:51:49.004718 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:49.004694    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22be106b-a8a0-447a-b8c5-ee22d46a19ea-tmp-dir\") pod \"prestop-hook-test-kserve-87c8d84b4-677mv\" (UID: \"22be106b-a8a0-447a-b8c5-ee22d46a19ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv"
Sep 16 19:51:49.004795 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:49.004737    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22be106b-a8a0-447a-b8c5-ee22d46a19ea-kserve-provision-location\") pod \"prestop-hook-test-kserve-87c8d84b4-677mv\" (UID: \"22be106b-a8a0-447a-b8c5-ee22d46a19ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv"
Sep 16 19:51:49.004966 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:49.004946    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22be106b-a8a0-447a-b8c5-ee22d46a19ea-cabundle-cert\") pod \"prestop-hook-test-kserve-87c8d84b4-677mv\" (UID: \"22be106b-a8a0-447a-b8c5-ee22d46a19ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv"
Sep 16 19:51:49.006406 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:49.006377    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22be106b-a8a0-447a-b8c5-ee22d46a19ea-dshm\") pod \"prestop-hook-test-kserve-87c8d84b4-677mv\" (UID: \"22be106b-a8a0-447a-b8c5-ee22d46a19ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv"
Sep 16 19:51:49.006681 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:49.006663    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22be106b-a8a0-447a-b8c5-ee22d46a19ea-tls-certs\") pod \"prestop-hook-test-kserve-87c8d84b4-677mv\" (UID: \"22be106b-a8a0-447a-b8c5-ee22d46a19ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv"
Sep 16 19:51:49.013174 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:49.013155    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5xct\" (UniqueName: \"kubernetes.io/projected/22be106b-a8a0-447a-b8c5-ee22d46a19ea-kube-api-access-b5xct\") pod \"prestop-hook-test-kserve-87c8d84b4-677mv\" (UID: \"22be106b-a8a0-447a-b8c5-ee22d46a19ea\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv"
Sep 16 19:51:49.197591 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:49.197557    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv"
Sep 16 19:51:49.480784 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:49.480757    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv"]
Sep 16 19:51:49.482629 ip-10-0-128-247 kubenswrapper[2577]: W0916 19:51:49.482594    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22be106b_a8a0_447a_b8c5_ee22d46a19ea.slice/crio-52d50d0e25d792135b5d3b203e11a0fd52ae6e8dc5defc69d4edaaaa4d36a921 WatchSource:0}: Error finding container 52d50d0e25d792135b5d3b203e11a0fd52ae6e8dc5defc69d4edaaaa4d36a921: Status 404 returned error can't find the container with id 52d50d0e25d792135b5d3b203e11a0fd52ae6e8dc5defc69d4edaaaa4d36a921
Sep 16 19:51:50.152071 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:50.152032    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv" event={"ID":"22be106b-a8a0-447a-b8c5-ee22d46a19ea","Type":"ContainerStarted","Data":"08b3473ee99b3e80939fb79e390ee8e36c72cd9b551377951de4f608554505fa"}
Sep 16 19:51:50.152592 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:50.152087    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv" event={"ID":"22be106b-a8a0-447a-b8c5-ee22d46a19ea","Type":"ContainerStarted","Data":"52d50d0e25d792135b5d3b203e11a0fd52ae6e8dc5defc69d4edaaaa4d36a921"}
Sep 16 19:51:52.164367 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:52.164327    2577 generic.go:356] "Generic (PLEG): container finished" podID="22be106b-a8a0-447a-b8c5-ee22d46a19ea" containerID="08b3473ee99b3e80939fb79e390ee8e36c72cd9b551377951de4f608554505fa" exitCode=0
Sep 16 19:51:52.164786 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:52.164400    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv" event={"ID":"22be106b-a8a0-447a-b8c5-ee22d46a19ea","Type":"ContainerDied","Data":"08b3473ee99b3e80939fb79e390ee8e36c72cd9b551377951de4f608554505fa"}
Sep 16 19:51:53.170234 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:53.170195    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv" event={"ID":"22be106b-a8a0-447a-b8c5-ee22d46a19ea","Type":"ContainerStarted","Data":"199fd977a389dc360eadd9a6ce8586849dacf23681604a52a78d4adc1fe19c9a"}
Sep 16 19:51:53.196276 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:53.196210    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv" podStartSLOduration=2.985540636 podStartE2EDuration="5.196191692s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.210651056s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:51:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:51:53.194869799 +0000 UTC m=+4270.577759204" watchObservedRunningTime="2026-09-16 19:51:53.196191692 +0000 UTC m=+4270.579081061"
Sep 16 19:51:53.197937 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:53.197906    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv"
Sep 16 19:51:54.882247 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:54.882201    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz" podUID="e86b081c-0927-44f2-acea-c66ea38ec59c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Sep 16 19:51:59.197912 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:59.197878    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv"
Sep 16 19:51:59.199734 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:51:59.199698    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv" podUID="22be106b-a8a0-447a-b8c5-ee22d46a19ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 16 19:52:04.883034 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:04.882979    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz" podUID="e86b081c-0927-44f2-acea-c66ea38ec59c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Sep 16 19:52:09.198484 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:09.198433    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv" podUID="22be106b-a8a0-447a-b8c5-ee22d46a19ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 16 19:52:14.892583 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:14.892522    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz"
Sep 16 19:52:14.900750 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:14.900724    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz"
Sep 16 19:52:19.198586 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:19.198541    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv" podUID="22be106b-a8a0-447a-b8c5-ee22d46a19ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 16 19:52:29.198519 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:29.198436    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv" podUID="22be106b-a8a0-447a-b8c5-ee22d46a19ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 16 19:52:33.332461 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:33.332423    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz"]
Sep 16 19:52:39.197973 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:39.197932    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv" podUID="22be106b-a8a0-447a-b8c5-ee22d46a19ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 16 19:52:48.355360 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:48.355318    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz" podUID="e86b081c-0927-44f2-acea-c66ea38ec59c" containerName="main" containerID="cri-o://84f7438391ed7cf0fdab56d427f919928e512ca9651ea9599baff280f3dadc4a" gracePeriod=45
Sep 16 19:52:49.100120 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:49.100087    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz"
Sep 16 19:52:49.114665 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:49.114635    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e86b081c-0927-44f2-acea-c66ea38ec59c-kserve-provision-location\") pod \"e86b081c-0927-44f2-acea-c66ea38ec59c\" (UID: \"e86b081c-0927-44f2-acea-c66ea38ec59c\") "
Sep 16 19:52:49.114799 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:49.114684    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e86b081c-0927-44f2-acea-c66ea38ec59c-home\") pod \"e86b081c-0927-44f2-acea-c66ea38ec59c\" (UID: \"e86b081c-0927-44f2-acea-c66ea38ec59c\") "
Sep 16 19:52:49.114799 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:49.114727    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnwnk\" (UniqueName: \"kubernetes.io/projected/e86b081c-0927-44f2-acea-c66ea38ec59c-kube-api-access-jnwnk\") pod \"e86b081c-0927-44f2-acea-c66ea38ec59c\" (UID: \"e86b081c-0927-44f2-acea-c66ea38ec59c\") "
Sep 16 19:52:49.114799 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:49.114761    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e86b081c-0927-44f2-acea-c66ea38ec59c-tls-certs\") pod \"e86b081c-0927-44f2-acea-c66ea38ec59c\" (UID: \"e86b081c-0927-44f2-acea-c66ea38ec59c\") "
Sep 16 19:52:49.114799 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:49.114788    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e86b081c-0927-44f2-acea-c66ea38ec59c-tmp-dir\") pod \"e86b081c-0927-44f2-acea-c66ea38ec59c\" (UID: \"e86b081c-0927-44f2-acea-c66ea38ec59c\") "
Sep 16 19:52:49.115023 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:49.114842    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e86b081c-0927-44f2-acea-c66ea38ec59c-dshm\") pod \"e86b081c-0927-44f2-acea-c66ea38ec59c\" (UID: \"e86b081c-0927-44f2-acea-c66ea38ec59c\") "
Sep 16 19:52:49.115023 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:49.114872    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e86b081c-0927-44f2-acea-c66ea38ec59c-cabundle-cert\") pod \"e86b081c-0927-44f2-acea-c66ea38ec59c\" (UID: \"e86b081c-0927-44f2-acea-c66ea38ec59c\") "
Sep 16 19:52:49.115023 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:49.114934    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e86b081c-0927-44f2-acea-c66ea38ec59c-model-cache\") pod \"e86b081c-0927-44f2-acea-c66ea38ec59c\" (UID: \"e86b081c-0927-44f2-acea-c66ea38ec59c\") "
Sep 16 19:52:49.115428 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:49.115380    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e86b081c-0927-44f2-acea-c66ea38ec59c-model-cache" (OuterVolumeSpecName: "model-cache") pod "e86b081c-0927-44f2-acea-c66ea38ec59c" (UID: "e86b081c-0927-44f2-acea-c66ea38ec59c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:52:49.115553 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:49.115438    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e86b081c-0927-44f2-acea-c66ea38ec59c-home" (OuterVolumeSpecName: "home") pod "e86b081c-0927-44f2-acea-c66ea38ec59c" (UID: "e86b081c-0927-44f2-acea-c66ea38ec59c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:52:49.115553 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:49.115459    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e86b081c-0927-44f2-acea-c66ea38ec59c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e86b081c-0927-44f2-acea-c66ea38ec59c" (UID: "e86b081c-0927-44f2-acea-c66ea38ec59c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:52:49.118018 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:49.117973    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e86b081c-0927-44f2-acea-c66ea38ec59c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e86b081c-0927-44f2-acea-c66ea38ec59c" (UID: "e86b081c-0927-44f2-acea-c66ea38ec59c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:52:49.118018 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:49.118005    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e86b081c-0927-44f2-acea-c66ea38ec59c-dshm" (OuterVolumeSpecName: "dshm") pod "e86b081c-0927-44f2-acea-c66ea38ec59c" (UID: "e86b081c-0927-44f2-acea-c66ea38ec59c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:52:49.118301 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:49.118273    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e86b081c-0927-44f2-acea-c66ea38ec59c-kube-api-access-jnwnk" (OuterVolumeSpecName: "kube-api-access-jnwnk") pod "e86b081c-0927-44f2-acea-c66ea38ec59c" (UID: "e86b081c-0927-44f2-acea-c66ea38ec59c"). InnerVolumeSpecName "kube-api-access-jnwnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:52:49.135157 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:49.135081    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e86b081c-0927-44f2-acea-c66ea38ec59c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e86b081c-0927-44f2-acea-c66ea38ec59c" (UID: "e86b081c-0927-44f2-acea-c66ea38ec59c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:52:49.146857 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:49.146814    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e86b081c-0927-44f2-acea-c66ea38ec59c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e86b081c-0927-44f2-acea-c66ea38ec59c" (UID: "e86b081c-0927-44f2-acea-c66ea38ec59c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:52:49.198643 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:49.198602    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv" podUID="22be106b-a8a0-447a-b8c5-ee22d46a19ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 16 19:52:49.215818 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:49.215792    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e86b081c-0927-44f2-acea-c66ea38ec59c-model-cache\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:52:49.215886 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:49.215823    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e86b081c-0927-44f2-acea-c66ea38ec59c-kserve-provision-location\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:52:49.215886 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:49.215839    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e86b081c-0927-44f2-acea-c66ea38ec59c-home\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:52:49.215886 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:49.215854    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnwnk\" (UniqueName: \"kubernetes.io/projected/e86b081c-0927-44f2-acea-c66ea38ec59c-kube-api-access-jnwnk\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:52:49.215886 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:49.215867    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e86b081c-0927-44f2-acea-c66ea38ec59c-tls-certs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:52:49.215886 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:49.215880    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e86b081c-0927-44f2-acea-c66ea38ec59c-tmp-dir\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:52:49.216044 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:49.215892    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e86b081c-0927-44f2-acea-c66ea38ec59c-dshm\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:52:49.216044 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:49.215904    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e86b081c-0927-44f2-acea-c66ea38ec59c-cabundle-cert\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:52:49.435512 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:49.435413    2577 generic.go:356] "Generic (PLEG): container finished" podID="e86b081c-0927-44f2-acea-c66ea38ec59c" containerID="84f7438391ed7cf0fdab56d427f919928e512ca9651ea9599baff280f3dadc4a" exitCode=0
Sep 16 19:52:49.435512 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:49.435473    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz"
Sep 16 19:52:49.435512 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:49.435492    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz" event={"ID":"e86b081c-0927-44f2-acea-c66ea38ec59c","Type":"ContainerDied","Data":"84f7438391ed7cf0fdab56d427f919928e512ca9651ea9599baff280f3dadc4a"}
Sep 16 19:52:49.436099 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:49.435559    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz" event={"ID":"e86b081c-0927-44f2-acea-c66ea38ec59c","Type":"ContainerDied","Data":"2ea1b90ccea08ba445528e0218f8fcb650fd0aaeff83f3edbca4a2dfc48093ba"}
Sep 16 19:52:49.436099 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:49.435583    2577 scope.go:117] "RemoveContainer" containerID="84f7438391ed7cf0fdab56d427f919928e512ca9651ea9599baff280f3dadc4a"
Sep 16 19:52:49.444548 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:49.444503    2577 scope.go:117] "RemoveContainer" containerID="0269f53aaf16c0ebd4e05f41324c024377aa82e38fa85e1ddf326037523671e5"
Sep 16 19:52:49.452493 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:49.452469    2577 scope.go:117] "RemoveContainer" containerID="84f7438391ed7cf0fdab56d427f919928e512ca9651ea9599baff280f3dadc4a"
Sep 16 19:52:49.452794 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:52:49.452773    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84f7438391ed7cf0fdab56d427f919928e512ca9651ea9599baff280f3dadc4a\": container with ID starting with 84f7438391ed7cf0fdab56d427f919928e512ca9651ea9599baff280f3dadc4a not found: ID does not exist" containerID="84f7438391ed7cf0fdab56d427f919928e512ca9651ea9599baff280f3dadc4a"
Sep 16 19:52:49.452875 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:49.452801    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84f7438391ed7cf0fdab56d427f919928e512ca9651ea9599baff280f3dadc4a"} err="failed to get container status \"84f7438391ed7cf0fdab56d427f919928e512ca9651ea9599baff280f3dadc4a\": rpc error: code = NotFound desc = could not find container \"84f7438391ed7cf0fdab56d427f919928e512ca9651ea9599baff280f3dadc4a\": container with ID starting with 84f7438391ed7cf0fdab56d427f919928e512ca9651ea9599baff280f3dadc4a not found: ID does not exist"
Sep 16 19:52:49.452875 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:49.452818    2577 scope.go:117] "RemoveContainer" containerID="0269f53aaf16c0ebd4e05f41324c024377aa82e38fa85e1ddf326037523671e5"
Sep 16 19:52:49.453076 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:52:49.453058    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0269f53aaf16c0ebd4e05f41324c024377aa82e38fa85e1ddf326037523671e5\": container with ID starting with 0269f53aaf16c0ebd4e05f41324c024377aa82e38fa85e1ddf326037523671e5 not found: ID does not exist" containerID="0269f53aaf16c0ebd4e05f41324c024377aa82e38fa85e1ddf326037523671e5"
Sep 16 19:52:49.453150 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:49.453077    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0269f53aaf16c0ebd4e05f41324c024377aa82e38fa85e1ddf326037523671e5"} err="failed to get container status \"0269f53aaf16c0ebd4e05f41324c024377aa82e38fa85e1ddf326037523671e5\": rpc error: code = NotFound desc = could not find container \"0269f53aaf16c0ebd4e05f41324c024377aa82e38fa85e1ddf326037523671e5\": container with ID starting with 0269f53aaf16c0ebd4e05f41324c024377aa82e38fa85e1ddf326037523671e5 not found: ID does not exist"
Sep 16 19:52:49.457777 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:49.457752    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz"]
Sep 16 19:52:49.465919 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:49.465895    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f87dc7d7-wlzzz"]
Sep 16 19:52:51.154999 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:51.154966    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e86b081c-0927-44f2-acea-c66ea38ec59c" path="/var/lib/kubelet/pods/e86b081c-0927-44f2-acea-c66ea38ec59c/volumes"
Sep 16 19:52:59.198457 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:52:59.198416    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv" podUID="22be106b-a8a0-447a-b8c5-ee22d46a19ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 16 19:53:09.198943 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:09.198901    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv" podUID="22be106b-a8a0-447a-b8c5-ee22d46a19ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 16 19:53:19.198824 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:19.198778    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv" podUID="22be106b-a8a0-447a-b8c5-ee22d46a19ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 16 19:53:29.198826 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:29.198777    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv" podUID="22be106b-a8a0-447a-b8c5-ee22d46a19ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 16 19:53:32.284552 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:32.284493    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946"]
Sep 16 19:53:32.285065 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:32.285041    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e86b081c-0927-44f2-acea-c66ea38ec59c" containerName="storage-initializer"
Sep 16 19:53:32.285114 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:32.285072    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e86b081c-0927-44f2-acea-c66ea38ec59c" containerName="storage-initializer"
Sep 16 19:53:32.285114 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:32.285106    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e86b081c-0927-44f2-acea-c66ea38ec59c" containerName="main"
Sep 16 19:53:32.285179 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:32.285116    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e86b081c-0927-44f2-acea-c66ea38ec59c" containerName="main"
Sep 16 19:53:32.285228 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:32.285213    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e86b081c-0927-44f2-acea-c66ea38ec59c" containerName="main"
Sep 16 19:53:32.291952 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:32.291932    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946"
Sep 16 19:53:32.294981 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:32.294959    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-vxvmq\""
Sep 16 19:53:32.295128 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:32.295094    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Sep 16 19:53:32.295205 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:32.295008    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:53:32.295205 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:32.295036    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Sep 16 19:53:32.295205 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:32.295048    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Sep 16 19:53:32.295452 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:32.295435    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Sep 16 19:53:32.299507 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:32.299481    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946"]
Sep 16 19:53:32.359258 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:32.359222    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946\" (UID: \"fd346e50-c9ff-4804-a93c-d77ebbcf6d77\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946"
Sep 16 19:53:32.359258 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:32.359258    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946\" (UID: \"fd346e50-c9ff-4804-a93c-d77ebbcf6d77\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946"
Sep 16 19:53:32.359508 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:32.359290    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946\" (UID: \"fd346e50-c9ff-4804-a93c-d77ebbcf6d77\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946"
Sep 16 19:53:32.359508 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:32.359346    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946\" (UID: \"fd346e50-c9ff-4804-a93c-d77ebbcf6d77\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946"
Sep 16 19:53:32.359508 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:32.359404    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946\" (UID: \"fd346e50-c9ff-4804-a93c-d77ebbcf6d77\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946"
Sep 16 19:53:32.359508 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:32.359419    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946\" (UID: \"fd346e50-c9ff-4804-a93c-d77ebbcf6d77\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946"
Sep 16 19:53:32.359508 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:32.359437    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946\" (UID: \"fd346e50-c9ff-4804-a93c-d77ebbcf6d77\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946"
Sep 16 19:53:32.359508 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:32.359465    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhft7\" (UniqueName: \"kubernetes.io/projected/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-kube-api-access-fhft7\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946\" (UID: \"fd346e50-c9ff-4804-a93c-d77ebbcf6d77\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946"
Sep 16 19:53:32.460385 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:32.460349    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946\" (UID: \"fd346e50-c9ff-4804-a93c-d77ebbcf6d77\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946"
Sep 16 19:53:32.460385 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:32.460384    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946\" (UID: \"fd346e50-c9ff-4804-a93c-d77ebbcf6d77\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946"
Sep 16 19:53:32.460667 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:32.460406    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946\" (UID: \"fd346e50-c9ff-4804-a93c-d77ebbcf6d77\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946"
Sep 16 19:53:32.460667 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:32.460435    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946\" (UID: \"fd346e50-c9ff-4804-a93c-d77ebbcf6d77\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946"
Sep 16 19:53:32.460667 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:32.460481    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946\" (UID: \"fd346e50-c9ff-4804-a93c-d77ebbcf6d77\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946"
Sep 16 19:53:32.460667 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:32.460497    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946\" (UID: \"fd346e50-c9ff-4804-a93c-d77ebbcf6d77\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946"
Sep 16 19:53:32.460667 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:32.460583    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946\" (UID: \"fd346e50-c9ff-4804-a93c-d77ebbcf6d77\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946"
Sep 16 19:53:32.460667 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:32.460613    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhft7\" (UniqueName: \"kubernetes.io/projected/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-kube-api-access-fhft7\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946\" (UID: \"fd346e50-c9ff-4804-a93c-d77ebbcf6d77\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946"
Sep 16 19:53:32.461065 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:32.460835    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946\" (UID: \"fd346e50-c9ff-4804-a93c-d77ebbcf6d77\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946"
Sep 16 19:53:32.461065 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:32.460866    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946\" (UID: \"fd346e50-c9ff-4804-a93c-d77ebbcf6d77\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946"
Sep 16 19:53:32.461065 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:32.460883    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946\" (UID: \"fd346e50-c9ff-4804-a93c-d77ebbcf6d77\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946"
Sep 16 19:53:32.461238 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:32.461129    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946\" (UID: \"fd346e50-c9ff-4804-a93c-d77ebbcf6d77\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946"
Sep 16 19:53:32.461426 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:32.461403    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946\" (UID: \"fd346e50-c9ff-4804-a93c-d77ebbcf6d77\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946"
Sep 16 19:53:32.463037 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:32.463012    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946\" (UID: \"fd346e50-c9ff-4804-a93c-d77ebbcf6d77\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946"
Sep 16 19:53:32.463217 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:32.463199    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946\" (UID: \"fd346e50-c9ff-4804-a93c-d77ebbcf6d77\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946"
Sep 16 19:53:32.472260 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:32.472227    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhft7\" (UniqueName: \"kubernetes.io/projected/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-kube-api-access-fhft7\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946\" (UID: \"fd346e50-c9ff-4804-a93c-d77ebbcf6d77\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946"
Sep 16 19:53:32.604012 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:32.603911    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946"
Sep 16 19:53:32.886047 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:32.886022    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946"]
Sep 16 19:53:32.888719 ip-10-0-128-247 kubenswrapper[2577]: W0916 19:53:32.888690    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd346e50_c9ff_4804_a93c_d77ebbcf6d77.slice/crio-43f030de4f5a0291f88a7a76fa21456cab969d4c9cc645659f9c4e9eb376a5fb WatchSource:0}: Error finding container 43f030de4f5a0291f88a7a76fa21456cab969d4c9cc645659f9c4e9eb376a5fb: Status 404 returned error can't find the container with id 43f030de4f5a0291f88a7a76fa21456cab969d4c9cc645659f9c4e9eb376a5fb
Sep 16 19:53:33.632906 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:33.632865    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946" event={"ID":"fd346e50-c9ff-4804-a93c-d77ebbcf6d77","Type":"ContainerStarted","Data":"e092832ff5fc346098dea96e5c62c0a3f00bfa5c4a3b68f875b8282bad11bd02"}
Sep 16 19:53:33.633321 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:33.632935    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946" event={"ID":"fd346e50-c9ff-4804-a93c-d77ebbcf6d77","Type":"ContainerStarted","Data":"43f030de4f5a0291f88a7a76fa21456cab969d4c9cc645659f9c4e9eb376a5fb"}
Sep 16 19:53:35.645293 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:35.645259    2577 generic.go:356] "Generic (PLEG): container finished" podID="fd346e50-c9ff-4804-a93c-d77ebbcf6d77" containerID="e092832ff5fc346098dea96e5c62c0a3f00bfa5c4a3b68f875b8282bad11bd02" exitCode=0
Sep 16 19:53:35.645705 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:35.645336    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946" event={"ID":"fd346e50-c9ff-4804-a93c-d77ebbcf6d77","Type":"ContainerDied","Data":"e092832ff5fc346098dea96e5c62c0a3f00bfa5c4a3b68f875b8282bad11bd02"}
Sep 16 19:53:36.653200 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:36.653151    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946" event={"ID":"fd346e50-c9ff-4804-a93c-d77ebbcf6d77","Type":"ContainerStarted","Data":"6ae04e939203f4e54939c31e552d38d4ed027a9c7d95e8c8674e9d5e70033409"}
Sep 16 19:53:36.679730 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:36.679664    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946" podStartSLOduration=2.485939477 podStartE2EDuration="4.679641795s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.193702318s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:53:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:53:36.677151026 +0000 UTC m=+4374.060040392" watchObservedRunningTime="2026-09-16 19:53:36.679641795 +0000 UTC m=+4374.062531343"
Sep 16 19:53:37.605073 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:37.605026    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946"
Sep 16 19:53:39.198865 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:39.198818    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv" podUID="22be106b-a8a0-447a-b8c5-ee22d46a19ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 16 19:53:42.604827 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:42.604789    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946"
Sep 16 19:53:42.606442 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:42.606413    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946" podUID="fd346e50-c9ff-4804-a93c-d77ebbcf6d77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 16 19:53:49.207973 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:49.207940    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv"
Sep 16 19:53:49.215828 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:49.215801    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv"
Sep 16 19:53:49.919963 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:49.919927    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv"]
Sep 16 19:53:52.604960 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:53:52.604922    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946" podUID="fd346e50-c9ff-4804-a93c-d77ebbcf6d77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 16 19:54:02.604635 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:02.604584    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946" podUID="fd346e50-c9ff-4804-a93c-d77ebbcf6d77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 16 19:54:05.741937 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:05.741810    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv" podUID="22be106b-a8a0-447a-b8c5-ee22d46a19ea" containerName="main" containerID="cri-o://199fd977a389dc360eadd9a6ce8586849dacf23681604a52a78d4adc1fe19c9a" gracePeriod=45
Sep 16 19:54:06.209545 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:06.209487    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv" podUID="22be106b-a8a0-447a-b8c5-ee22d46a19ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 16 19:54:06.486847 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:06.486820    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv"
Sep 16 19:54:06.556174 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:06.556141    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22be106b-a8a0-447a-b8c5-ee22d46a19ea-model-cache\") pod \"22be106b-a8a0-447a-b8c5-ee22d46a19ea\" (UID: \"22be106b-a8a0-447a-b8c5-ee22d46a19ea\") "
Sep 16 19:54:06.556336 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:06.556180    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5xct\" (UniqueName: \"kubernetes.io/projected/22be106b-a8a0-447a-b8c5-ee22d46a19ea-kube-api-access-b5xct\") pod \"22be106b-a8a0-447a-b8c5-ee22d46a19ea\" (UID: \"22be106b-a8a0-447a-b8c5-ee22d46a19ea\") "
Sep 16 19:54:06.556336 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:06.556212    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22be106b-a8a0-447a-b8c5-ee22d46a19ea-dshm\") pod \"22be106b-a8a0-447a-b8c5-ee22d46a19ea\" (UID: \"22be106b-a8a0-447a-b8c5-ee22d46a19ea\") "
Sep 16 19:54:06.556336 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:06.556229    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22be106b-a8a0-447a-b8c5-ee22d46a19ea-tmp-dir\") pod \"22be106b-a8a0-447a-b8c5-ee22d46a19ea\" (UID: \"22be106b-a8a0-447a-b8c5-ee22d46a19ea\") "
Sep 16 19:54:06.556336 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:06.556273    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22be106b-a8a0-447a-b8c5-ee22d46a19ea-cabundle-cert\") pod \"22be106b-a8a0-447a-b8c5-ee22d46a19ea\" (UID: \"22be106b-a8a0-447a-b8c5-ee22d46a19ea\") "
Sep 16 19:54:06.556336 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:06.556322    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22be106b-a8a0-447a-b8c5-ee22d46a19ea-kserve-provision-location\") pod \"22be106b-a8a0-447a-b8c5-ee22d46a19ea\" (UID: \"22be106b-a8a0-447a-b8c5-ee22d46a19ea\") "
Sep 16 19:54:06.556715 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:06.556382    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22be106b-a8a0-447a-b8c5-ee22d46a19ea-tls-certs\") pod \"22be106b-a8a0-447a-b8c5-ee22d46a19ea\" (UID: \"22be106b-a8a0-447a-b8c5-ee22d46a19ea\") "
Sep 16 19:54:06.556715 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:06.556410    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22be106b-a8a0-447a-b8c5-ee22d46a19ea-home\") pod \"22be106b-a8a0-447a-b8c5-ee22d46a19ea\" (UID: \"22be106b-a8a0-447a-b8c5-ee22d46a19ea\") "
Sep 16 19:54:06.556715 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:06.556481    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22be106b-a8a0-447a-b8c5-ee22d46a19ea-model-cache" (OuterVolumeSpecName: "model-cache") pod "22be106b-a8a0-447a-b8c5-ee22d46a19ea" (UID: "22be106b-a8a0-447a-b8c5-ee22d46a19ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:54:06.556715 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:06.556691    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22be106b-a8a0-447a-b8c5-ee22d46a19ea-model-cache\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:54:06.557226 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:06.557201    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22be106b-a8a0-447a-b8c5-ee22d46a19ea-home" (OuterVolumeSpecName: "home") pod "22be106b-a8a0-447a-b8c5-ee22d46a19ea" (UID: "22be106b-a8a0-447a-b8c5-ee22d46a19ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:54:06.558974 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:06.558927    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22be106b-a8a0-447a-b8c5-ee22d46a19ea-dshm" (OuterVolumeSpecName: "dshm") pod "22be106b-a8a0-447a-b8c5-ee22d46a19ea" (UID: "22be106b-a8a0-447a-b8c5-ee22d46a19ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:54:06.558974 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:06.558949    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22be106b-a8a0-447a-b8c5-ee22d46a19ea-kube-api-access-b5xct" (OuterVolumeSpecName: "kube-api-access-b5xct") pod "22be106b-a8a0-447a-b8c5-ee22d46a19ea" (UID: "22be106b-a8a0-447a-b8c5-ee22d46a19ea"). InnerVolumeSpecName "kube-api-access-b5xct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:54:06.559138 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:06.559065    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22be106b-a8a0-447a-b8c5-ee22d46a19ea-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "22be106b-a8a0-447a-b8c5-ee22d46a19ea" (UID: "22be106b-a8a0-447a-b8c5-ee22d46a19ea"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:54:06.559322 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:06.559289    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22be106b-a8a0-447a-b8c5-ee22d46a19ea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "22be106b-a8a0-447a-b8c5-ee22d46a19ea" (UID: "22be106b-a8a0-447a-b8c5-ee22d46a19ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:54:06.576682 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:06.576626    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22be106b-a8a0-447a-b8c5-ee22d46a19ea-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "22be106b-a8a0-447a-b8c5-ee22d46a19ea" (UID: "22be106b-a8a0-447a-b8c5-ee22d46a19ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:54:06.584027 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:06.584003    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22be106b-a8a0-447a-b8c5-ee22d46a19ea-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "22be106b-a8a0-447a-b8c5-ee22d46a19ea" (UID: "22be106b-a8a0-447a-b8c5-ee22d46a19ea"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:54:06.657153 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:06.657128    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22be106b-a8a0-447a-b8c5-ee22d46a19ea-kserve-provision-location\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:54:06.657153 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:06.657150    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22be106b-a8a0-447a-b8c5-ee22d46a19ea-tls-certs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:54:06.657153 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:06.657160    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22be106b-a8a0-447a-b8c5-ee22d46a19ea-home\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:54:06.657340 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:06.657169    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5xct\" (UniqueName: \"kubernetes.io/projected/22be106b-a8a0-447a-b8c5-ee22d46a19ea-kube-api-access-b5xct\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:54:06.657340 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:06.657178    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22be106b-a8a0-447a-b8c5-ee22d46a19ea-dshm\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:54:06.657340 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:06.657185    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22be106b-a8a0-447a-b8c5-ee22d46a19ea-tmp-dir\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:54:06.657340 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:06.657193    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22be106b-a8a0-447a-b8c5-ee22d46a19ea-cabundle-cert\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:54:06.798931 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:06.798851    2577 generic.go:356] "Generic (PLEG): container finished" podID="22be106b-a8a0-447a-b8c5-ee22d46a19ea" containerID="199fd977a389dc360eadd9a6ce8586849dacf23681604a52a78d4adc1fe19c9a" exitCode=0
Sep 16 19:54:06.798931 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:06.798895    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv"
Sep 16 19:54:06.799351 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:06.798934    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv" event={"ID":"22be106b-a8a0-447a-b8c5-ee22d46a19ea","Type":"ContainerDied","Data":"199fd977a389dc360eadd9a6ce8586849dacf23681604a52a78d4adc1fe19c9a"}
Sep 16 19:54:06.799351 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:06.798990    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv" event={"ID":"22be106b-a8a0-447a-b8c5-ee22d46a19ea","Type":"ContainerDied","Data":"52d50d0e25d792135b5d3b203e11a0fd52ae6e8dc5defc69d4edaaaa4d36a921"}
Sep 16 19:54:06.799351 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:06.799014    2577 scope.go:117] "RemoveContainer" containerID="199fd977a389dc360eadd9a6ce8586849dacf23681604a52a78d4adc1fe19c9a"
Sep 16 19:54:06.809903 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:06.809881    2577 scope.go:117] "RemoveContainer" containerID="08b3473ee99b3e80939fb79e390ee8e36c72cd9b551377951de4f608554505fa"
Sep 16 19:54:06.819143 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:06.819122    2577 scope.go:117] "RemoveContainer" containerID="199fd977a389dc360eadd9a6ce8586849dacf23681604a52a78d4adc1fe19c9a"
Sep 16 19:54:06.819433 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:54:06.819412    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"199fd977a389dc360eadd9a6ce8586849dacf23681604a52a78d4adc1fe19c9a\": container with ID starting with 199fd977a389dc360eadd9a6ce8586849dacf23681604a52a78d4adc1fe19c9a not found: ID does not exist" containerID="199fd977a389dc360eadd9a6ce8586849dacf23681604a52a78d4adc1fe19c9a"
Sep 16 19:54:06.819488 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:06.819441    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"199fd977a389dc360eadd9a6ce8586849dacf23681604a52a78d4adc1fe19c9a"} err="failed to get container status \"199fd977a389dc360eadd9a6ce8586849dacf23681604a52a78d4adc1fe19c9a\": rpc error: code = NotFound desc = could not find container \"199fd977a389dc360eadd9a6ce8586849dacf23681604a52a78d4adc1fe19c9a\": container with ID starting with 199fd977a389dc360eadd9a6ce8586849dacf23681604a52a78d4adc1fe19c9a not found: ID does not exist"
Sep 16 19:54:06.819488 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:06.819459    2577 scope.go:117] "RemoveContainer" containerID="08b3473ee99b3e80939fb79e390ee8e36c72cd9b551377951de4f608554505fa"
Sep 16 19:54:06.819743 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:54:06.819725    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08b3473ee99b3e80939fb79e390ee8e36c72cd9b551377951de4f608554505fa\": container with ID starting with 08b3473ee99b3e80939fb79e390ee8e36c72cd9b551377951de4f608554505fa not found: ID does not exist" containerID="08b3473ee99b3e80939fb79e390ee8e36c72cd9b551377951de4f608554505fa"
Sep 16 19:54:06.819810 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:06.819759    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08b3473ee99b3e80939fb79e390ee8e36c72cd9b551377951de4f608554505fa"} err="failed to get container status \"08b3473ee99b3e80939fb79e390ee8e36c72cd9b551377951de4f608554505fa\": rpc error: code = NotFound desc = could not find container \"08b3473ee99b3e80939fb79e390ee8e36c72cd9b551377951de4f608554505fa\": container with ID starting with 08b3473ee99b3e80939fb79e390ee8e36c72cd9b551377951de4f608554505fa not found: ID does not exist"
Sep 16 19:54:06.823981 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:06.823960    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv"]
Sep 16 19:54:06.834851 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:06.831094    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-677mv"]
Sep 16 19:54:07.155737 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:07.155693    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22be106b-a8a0-447a-b8c5-ee22d46a19ea" path="/var/lib/kubelet/pods/22be106b-a8a0-447a-b8c5-ee22d46a19ea/volumes"
Sep 16 19:54:12.605047 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:12.605010    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946" podUID="fd346e50-c9ff-4804-a93c-d77ebbcf6d77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 16 19:54:22.604553 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:22.604483    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946" podUID="fd346e50-c9ff-4804-a93c-d77ebbcf6d77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 16 19:54:32.604617 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:32.604574    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946" podUID="fd346e50-c9ff-4804-a93c-d77ebbcf6d77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 16 19:54:42.604866 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:42.604825    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946" podUID="fd346e50-c9ff-4804-a93c-d77ebbcf6d77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 16 19:54:52.604917 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:54:52.604871    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946" podUID="fd346e50-c9ff-4804-a93c-d77ebbcf6d77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 16 19:55:02.604507 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:55:02.604463    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946" podUID="fd346e50-c9ff-4804-a93c-d77ebbcf6d77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 16 19:55:12.604738 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:55:12.604695    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946" podUID="fd346e50-c9ff-4804-a93c-d77ebbcf6d77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 16 19:55:22.604637 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:55:22.604591    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946" podUID="fd346e50-c9ff-4804-a93c-d77ebbcf6d77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 16 19:55:32.614090 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:55:32.614058    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946"
Sep 16 19:55:32.621749 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:55:32.621720    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946"
Sep 16 19:55:56.070712 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:55:56.070623    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946"]
Sep 16 19:55:57.645576 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:55:57.645543    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slsjr"]
Sep 16 19:55:57.645970 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:55:57.645856    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22be106b-a8a0-447a-b8c5-ee22d46a19ea" containerName="main"
Sep 16 19:55:57.645970 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:55:57.645868    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="22be106b-a8a0-447a-b8c5-ee22d46a19ea" containerName="main"
Sep 16 19:55:57.645970 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:55:57.645893    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22be106b-a8a0-447a-b8c5-ee22d46a19ea" containerName="storage-initializer"
Sep 16 19:55:57.645970 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:55:57.645899    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="22be106b-a8a0-447a-b8c5-ee22d46a19ea" containerName="storage-initializer"
Sep 16 19:55:57.645970 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:55:57.645954    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="22be106b-a8a0-447a-b8c5-ee22d46a19ea" containerName="main"
Sep 16 19:55:57.650433 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:55:57.650416    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slsjr"
Sep 16 19:55:57.653588 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:55:57.653566    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-b6wch\""
Sep 16 19:55:57.653588 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:55:57.653577    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 16 19:55:57.654777 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:55:57.654760    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 16 19:55:57.654868 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:55:57.654795    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 16 19:55:57.660631 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:55:57.660611    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slsjr"]
Sep 16 19:55:57.763938 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:55:57.763913    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0825f3b-1db0-4450-b180-3e6c205991be-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-slsjr\" (UID: \"c0825f3b-1db0-4450-b180-3e6c205991be\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slsjr"
Sep 16 19:55:57.764056 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:55:57.763945    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0825f3b-1db0-4450-b180-3e6c205991be-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-slsjr\" (UID: \"c0825f3b-1db0-4450-b180-3e6c205991be\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slsjr"
Sep 16 19:55:57.764056 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:55:57.763979    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0825f3b-1db0-4450-b180-3e6c205991be-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-slsjr\" (UID: \"c0825f3b-1db0-4450-b180-3e6c205991be\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slsjr"
Sep 16 19:55:57.764056 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:55:57.764046    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0825f3b-1db0-4450-b180-3e6c205991be-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-slsjr\" (UID: \"c0825f3b-1db0-4450-b180-3e6c205991be\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slsjr"
Sep 16 19:55:57.764165 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:55:57.764087    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k86r4\" (UniqueName: \"kubernetes.io/projected/c0825f3b-1db0-4450-b180-3e6c205991be-kube-api-access-k86r4\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-slsjr\" (UID: \"c0825f3b-1db0-4450-b180-3e6c205991be\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slsjr"
Sep 16 19:55:57.764165 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:55:57.764109    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0825f3b-1db0-4450-b180-3e6c205991be-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-slsjr\" (UID: \"c0825f3b-1db0-4450-b180-3e6c205991be\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slsjr"
Sep 16 19:55:57.865042 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:55:57.865016    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0825f3b-1db0-4450-b180-3e6c205991be-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-slsjr\" (UID: \"c0825f3b-1db0-4450-b180-3e6c205991be\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slsjr"
Sep 16 19:55:57.865140 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:55:57.865052    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0825f3b-1db0-4450-b180-3e6c205991be-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-slsjr\" (UID: \"c0825f3b-1db0-4450-b180-3e6c205991be\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slsjr"
Sep 16 19:55:57.865140 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:55:57.865073    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k86r4\" (UniqueName: \"kubernetes.io/projected/c0825f3b-1db0-4450-b180-3e6c205991be-kube-api-access-k86r4\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-slsjr\" (UID: \"c0825f3b-1db0-4450-b180-3e6c205991be\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slsjr"
Sep 16 19:55:57.865140 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:55:57.865098    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0825f3b-1db0-4450-b180-3e6c205991be-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-slsjr\" (UID: \"c0825f3b-1db0-4450-b180-3e6c205991be\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slsjr"
Sep 16 19:55:57.865293 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:55:57.865154    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0825f3b-1db0-4450-b180-3e6c205991be-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-slsjr\" (UID: \"c0825f3b-1db0-4450-b180-3e6c205991be\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slsjr"
Sep 16 19:55:57.865293 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:55:57.865177    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0825f3b-1db0-4450-b180-3e6c205991be-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-slsjr\" (UID: \"c0825f3b-1db0-4450-b180-3e6c205991be\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slsjr"
Sep 16 19:55:57.865577 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:55:57.865522    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0825f3b-1db0-4450-b180-3e6c205991be-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-slsjr\" (UID: \"c0825f3b-1db0-4450-b180-3e6c205991be\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slsjr"
Sep 16 19:55:57.865686 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:55:57.865589    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0825f3b-1db0-4450-b180-3e6c205991be-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-slsjr\" (UID: \"c0825f3b-1db0-4450-b180-3e6c205991be\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slsjr"
Sep 16 19:55:57.865686 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:55:57.865628    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0825f3b-1db0-4450-b180-3e6c205991be-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-slsjr\" (UID: \"c0825f3b-1db0-4450-b180-3e6c205991be\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slsjr"
Sep 16 19:55:57.867539 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:55:57.867505    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0825f3b-1db0-4450-b180-3e6c205991be-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-slsjr\" (UID: \"c0825f3b-1db0-4450-b180-3e6c205991be\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slsjr"
Sep 16 19:55:57.867954 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:55:57.867938    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0825f3b-1db0-4450-b180-3e6c205991be-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-slsjr\" (UID: \"c0825f3b-1db0-4450-b180-3e6c205991be\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slsjr"
Sep 16 19:55:57.874208 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:55:57.874183    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k86r4\" (UniqueName: \"kubernetes.io/projected/c0825f3b-1db0-4450-b180-3e6c205991be-kube-api-access-k86r4\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-slsjr\" (UID: \"c0825f3b-1db0-4450-b180-3e6c205991be\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slsjr"
Sep 16 19:55:57.961219 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:55:57.961164    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slsjr"
Sep 16 19:55:58.233126 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:55:58.232988    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slsjr"]
Sep 16 19:55:58.235853 ip-10-0-128-247 kubenswrapper[2577]: W0916 19:55:58.235821    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0825f3b_1db0_4450_b180_3e6c205991be.slice/crio-a9c069590c969e3f4cc06d72d09b39b4e14d54b681ada2759b6c9e0b30139926 WatchSource:0}: Error finding container a9c069590c969e3f4cc06d72d09b39b4e14d54b681ada2759b6c9e0b30139926: Status 404 returned error can't find the container with id a9c069590c969e3f4cc06d72d09b39b4e14d54b681ada2759b6c9e0b30139926
Sep 16 19:55:58.288065 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:55:58.288040    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slsjr" event={"ID":"c0825f3b-1db0-4450-b180-3e6c205991be","Type":"ContainerStarted","Data":"a9c069590c969e3f4cc06d72d09b39b4e14d54b681ada2759b6c9e0b30139926"}
Sep 16 19:55:59.293677 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:55:59.293630    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slsjr" event={"ID":"c0825f3b-1db0-4450-b180-3e6c205991be","Type":"ContainerStarted","Data":"7b7a7f54f27d5ae8462e66a48ed4d0abcf9ed697e33a5ec31c0add9e0792d965"}
Sep 16 19:55:59.313519 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:55:59.313471    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slsjr" podStartSLOduration=2.313453677 podStartE2EDuration="2.313453677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:55:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:55:59.312663542 +0000 UTC m=+4516.695552924" watchObservedRunningTime="2026-09-16 19:55:59.313453677 +0000 UTC m=+4516.696343046"
Sep 16 19:55:59.962230 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:55:59.962192    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slsjr"
Sep 16 19:56:03.665820 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:03.665781    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slsjr"]
Sep 16 19:56:03.675488 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:56:03.675454    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:56:03.675488 ip-10-0-128-247 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:56:03.675488 ip-10-0-128-247 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 19:56:03.675488 ip-10-0-128-247 kubenswrapper[2577]:  > containerID="7b7a7f54f27d5ae8462e66a48ed4d0abcf9ed697e33a5ec31c0add9e0792d965" cmd=["/bin/sleep","15"]
Sep 16 19:56:03.675858 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:03.675503    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slsjr" message="" err=<
Sep 16 19:56:03.675858 ip-10-0-128-247 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:56:03.675858 ip-10-0-128-247 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 19:56:03.675858 ip-10-0-128-247 kubenswrapper[2577]:  >
Sep 16 19:56:03.675858 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:56:03.675516    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:56:03.675858 ip-10-0-128-247 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:56:03.675858 ip-10-0-128-247 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 19:56:03.675858 ip-10-0-128-247 kubenswrapper[2577]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slsjr" podUID="c0825f3b-1db0-4450-b180-3e6c205991be" containerName="main" containerID="cri-o://7b7a7f54f27d5ae8462e66a48ed4d0abcf9ed697e33a5ec31c0add9e0792d965"
Sep 16 19:56:03.675858 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:03.675591    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slsjr" podUID="c0825f3b-1db0-4450-b180-3e6c205991be" containerName="main" containerID="cri-o://7b7a7f54f27d5ae8462e66a48ed4d0abcf9ed697e33a5ec31c0add9e0792d965" gracePeriod=60
Sep 16 19:56:03.918840 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:03.918781    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slsjr"
Sep 16 19:56:04.012230 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:04.012200    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0825f3b-1db0-4450-b180-3e6c205991be-model-cache\") pod \"c0825f3b-1db0-4450-b180-3e6c205991be\" (UID: \"c0825f3b-1db0-4450-b180-3e6c205991be\") "
Sep 16 19:56:04.012363 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:04.012266    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0825f3b-1db0-4450-b180-3e6c205991be-tls-certs\") pod \"c0825f3b-1db0-4450-b180-3e6c205991be\" (UID: \"c0825f3b-1db0-4450-b180-3e6c205991be\") "
Sep 16 19:56:04.012363 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:04.012316    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0825f3b-1db0-4450-b180-3e6c205991be-dshm\") pod \"c0825f3b-1db0-4450-b180-3e6c205991be\" (UID: \"c0825f3b-1db0-4450-b180-3e6c205991be\") "
Sep 16 19:56:04.012363 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:04.012343    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k86r4\" (UniqueName: \"kubernetes.io/projected/c0825f3b-1db0-4450-b180-3e6c205991be-kube-api-access-k86r4\") pod \"c0825f3b-1db0-4450-b180-3e6c205991be\" (UID: \"c0825f3b-1db0-4450-b180-3e6c205991be\") "
Sep 16 19:56:04.012564 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:04.012404    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0825f3b-1db0-4450-b180-3e6c205991be-home\") pod \"c0825f3b-1db0-4450-b180-3e6c205991be\" (UID: \"c0825f3b-1db0-4450-b180-3e6c205991be\") "
Sep 16 19:56:04.012564 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:04.012440    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0825f3b-1db0-4450-b180-3e6c205991be-tmp-dir\") pod \"c0825f3b-1db0-4450-b180-3e6c205991be\" (UID: \"c0825f3b-1db0-4450-b180-3e6c205991be\") "
Sep 16 19:56:04.012564 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:04.012455    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0825f3b-1db0-4450-b180-3e6c205991be-model-cache" (OuterVolumeSpecName: "model-cache") pod "c0825f3b-1db0-4450-b180-3e6c205991be" (UID: "c0825f3b-1db0-4450-b180-3e6c205991be"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:56:04.012717 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:04.012693    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0825f3b-1db0-4450-b180-3e6c205991be-home" (OuterVolumeSpecName: "home") pod "c0825f3b-1db0-4450-b180-3e6c205991be" (UID: "c0825f3b-1db0-4450-b180-3e6c205991be"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:56:04.012763 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:04.012713    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0825f3b-1db0-4450-b180-3e6c205991be-model-cache\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:56:04.012763 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:04.012739    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0825f3b-1db0-4450-b180-3e6c205991be-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c0825f3b-1db0-4450-b180-3e6c205991be" (UID: "c0825f3b-1db0-4450-b180-3e6c205991be"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:56:04.014704 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:04.014678    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0825f3b-1db0-4450-b180-3e6c205991be-dshm" (OuterVolumeSpecName: "dshm") pod "c0825f3b-1db0-4450-b180-3e6c205991be" (UID: "c0825f3b-1db0-4450-b180-3e6c205991be"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:56:04.014789 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:04.014718    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0825f3b-1db0-4450-b180-3e6c205991be-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c0825f3b-1db0-4450-b180-3e6c205991be" (UID: "c0825f3b-1db0-4450-b180-3e6c205991be"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:56:04.014789 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:04.014731    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0825f3b-1db0-4450-b180-3e6c205991be-kube-api-access-k86r4" (OuterVolumeSpecName: "kube-api-access-k86r4") pod "c0825f3b-1db0-4450-b180-3e6c205991be" (UID: "c0825f3b-1db0-4450-b180-3e6c205991be"). InnerVolumeSpecName "kube-api-access-k86r4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:56:04.113669 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:04.113644    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0825f3b-1db0-4450-b180-3e6c205991be-home\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:56:04.113669 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:04.113666    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0825f3b-1db0-4450-b180-3e6c205991be-tmp-dir\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:56:04.113806 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:04.113675    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0825f3b-1db0-4450-b180-3e6c205991be-tls-certs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:56:04.113806 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:04.113682    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0825f3b-1db0-4450-b180-3e6c205991be-dshm\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:56:04.113806 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:04.113690    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k86r4\" (UniqueName: \"kubernetes.io/projected/c0825f3b-1db0-4450-b180-3e6c205991be-kube-api-access-k86r4\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:56:04.314810 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:04.314726    2577 generic.go:356] "Generic (PLEG): container finished" podID="c0825f3b-1db0-4450-b180-3e6c205991be" containerID="7b7a7f54f27d5ae8462e66a48ed4d0abcf9ed697e33a5ec31c0add9e0792d965" exitCode=0
Sep 16 19:56:04.314810 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:04.314777    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slsjr"
Sep 16 19:56:04.314958 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:04.314828    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slsjr" event={"ID":"c0825f3b-1db0-4450-b180-3e6c205991be","Type":"ContainerDied","Data":"7b7a7f54f27d5ae8462e66a48ed4d0abcf9ed697e33a5ec31c0add9e0792d965"}
Sep 16 19:56:04.314958 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:04.314882    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slsjr" event={"ID":"c0825f3b-1db0-4450-b180-3e6c205991be","Type":"ContainerDied","Data":"a9c069590c969e3f4cc06d72d09b39b4e14d54b681ada2759b6c9e0b30139926"}
Sep 16 19:56:04.314958 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:04.314910    2577 scope.go:117] "RemoveContainer" containerID="7b7a7f54f27d5ae8462e66a48ed4d0abcf9ed697e33a5ec31c0add9e0792d965"
Sep 16 19:56:04.324846 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:04.324829    2577 scope.go:117] "RemoveContainer" containerID="7b7a7f54f27d5ae8462e66a48ed4d0abcf9ed697e33a5ec31c0add9e0792d965"
Sep 16 19:56:04.325094 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:56:04.325076    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b7a7f54f27d5ae8462e66a48ed4d0abcf9ed697e33a5ec31c0add9e0792d965\": container with ID starting with 7b7a7f54f27d5ae8462e66a48ed4d0abcf9ed697e33a5ec31c0add9e0792d965 not found: ID does not exist" containerID="7b7a7f54f27d5ae8462e66a48ed4d0abcf9ed697e33a5ec31c0add9e0792d965"
Sep 16 19:56:04.325158 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:04.325099    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b7a7f54f27d5ae8462e66a48ed4d0abcf9ed697e33a5ec31c0add9e0792d965"} err="failed to get container status \"7b7a7f54f27d5ae8462e66a48ed4d0abcf9ed697e33a5ec31c0add9e0792d965\": rpc error: code = NotFound desc = could not find container \"7b7a7f54f27d5ae8462e66a48ed4d0abcf9ed697e33a5ec31c0add9e0792d965\": container with ID starting with 7b7a7f54f27d5ae8462e66a48ed4d0abcf9ed697e33a5ec31c0add9e0792d965 not found: ID does not exist"
Sep 16 19:56:04.339342 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:04.339318    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slsjr"]
Sep 16 19:56:04.346055 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:04.346035    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slsjr"]
Sep 16 19:56:05.155080 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:05.155045    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0825f3b-1db0-4450-b180-3e6c205991be" path="/var/lib/kubelet/pods/c0825f3b-1db0-4450-b180-3e6c205991be/volumes"
Sep 16 19:56:11.094835 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:11.094782    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946" podUID="fd346e50-c9ff-4804-a93c-d77ebbcf6d77" containerName="main" containerID="cri-o://6ae04e939203f4e54939c31e552d38d4ed027a9c7d95e8c8674e9d5e70033409" gracePeriod=45
Sep 16 19:56:11.614970 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:11.614937    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946" podUID="fd346e50-c9ff-4804-a93c-d77ebbcf6d77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 16 19:56:11.736829 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:11.736796    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946"
Sep 16 19:56:11.870156 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:11.870093    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-model-cache\") pod \"fd346e50-c9ff-4804-a93c-d77ebbcf6d77\" (UID: \"fd346e50-c9ff-4804-a93c-d77ebbcf6d77\") "
Sep 16 19:56:11.870156 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:11.870133    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-cabundle-cert\") pod \"fd346e50-c9ff-4804-a93c-d77ebbcf6d77\" (UID: \"fd346e50-c9ff-4804-a93c-d77ebbcf6d77\") "
Sep 16 19:56:11.870321 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:11.870169    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-tmp-dir\") pod \"fd346e50-c9ff-4804-a93c-d77ebbcf6d77\" (UID: \"fd346e50-c9ff-4804-a93c-d77ebbcf6d77\") "
Sep 16 19:56:11.870321 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:11.870200    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-dshm\") pod \"fd346e50-c9ff-4804-a93c-d77ebbcf6d77\" (UID: \"fd346e50-c9ff-4804-a93c-d77ebbcf6d77\") "
Sep 16 19:56:11.870321 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:11.870225    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-home\") pod \"fd346e50-c9ff-4804-a93c-d77ebbcf6d77\" (UID: \"fd346e50-c9ff-4804-a93c-d77ebbcf6d77\") "
Sep 16 19:56:11.870321 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:11.870244    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-tls-certs\") pod \"fd346e50-c9ff-4804-a93c-d77ebbcf6d77\" (UID: \"fd346e50-c9ff-4804-a93c-d77ebbcf6d77\") "
Sep 16 19:56:11.870321 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:11.870277    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-kserve-provision-location\") pod \"fd346e50-c9ff-4804-a93c-d77ebbcf6d77\" (UID: \"fd346e50-c9ff-4804-a93c-d77ebbcf6d77\") "
Sep 16 19:56:11.870321 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:11.870302    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhft7\" (UniqueName: \"kubernetes.io/projected/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-kube-api-access-fhft7\") pod \"fd346e50-c9ff-4804-a93c-d77ebbcf6d77\" (UID: \"fd346e50-c9ff-4804-a93c-d77ebbcf6d77\") "
Sep 16 19:56:11.870666 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:11.870319    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-model-cache" (OuterVolumeSpecName: "model-cache") pod "fd346e50-c9ff-4804-a93c-d77ebbcf6d77" (UID: "fd346e50-c9ff-4804-a93c-d77ebbcf6d77"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:56:11.870666 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:11.870511    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-model-cache\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:56:11.870666 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:11.870582    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fd346e50-c9ff-4804-a93c-d77ebbcf6d77" (UID: "fd346e50-c9ff-4804-a93c-d77ebbcf6d77"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:56:11.870952 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:11.870920    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-home" (OuterVolumeSpecName: "home") pod "fd346e50-c9ff-4804-a93c-d77ebbcf6d77" (UID: "fd346e50-c9ff-4804-a93c-d77ebbcf6d77"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:56:11.872697 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:11.872666    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-dshm" (OuterVolumeSpecName: "dshm") pod "fd346e50-c9ff-4804-a93c-d77ebbcf6d77" (UID: "fd346e50-c9ff-4804-a93c-d77ebbcf6d77"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:56:11.872827 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:11.872803    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-kube-api-access-fhft7" (OuterVolumeSpecName: "kube-api-access-fhft7") pod "fd346e50-c9ff-4804-a93c-d77ebbcf6d77" (UID: "fd346e50-c9ff-4804-a93c-d77ebbcf6d77"). InnerVolumeSpecName "kube-api-access-fhft7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:56:11.873233 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:11.873210    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fd346e50-c9ff-4804-a93c-d77ebbcf6d77" (UID: "fd346e50-c9ff-4804-a93c-d77ebbcf6d77"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:56:11.886439 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:11.886398    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fd346e50-c9ff-4804-a93c-d77ebbcf6d77" (UID: "fd346e50-c9ff-4804-a93c-d77ebbcf6d77"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:56:11.894011 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:11.893989    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fd346e50-c9ff-4804-a93c-d77ebbcf6d77" (UID: "fd346e50-c9ff-4804-a93c-d77ebbcf6d77"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:56:11.971843 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:11.971819    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-dshm\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:56:11.971843 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:11.971842    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-home\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:56:11.971973 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:11.971851    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-tls-certs\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:56:11.971973 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:11.971864    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-kserve-provision-location\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:56:11.971973 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:11.971874    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhft7\" (UniqueName: \"kubernetes.io/projected/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-kube-api-access-fhft7\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:56:11.971973 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:11.971882    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-cabundle-cert\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:56:11.971973 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:11.971910    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd346e50-c9ff-4804-a93c-d77ebbcf6d77-tmp-dir\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:56:12.348650 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:12.348621    2577 generic.go:356] "Generic (PLEG): container finished" podID="fd346e50-c9ff-4804-a93c-d77ebbcf6d77" containerID="6ae04e939203f4e54939c31e552d38d4ed027a9c7d95e8c8674e9d5e70033409" exitCode=0
Sep 16 19:56:12.348978 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:12.348673    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946"
Sep 16 19:56:12.348978 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:12.348708    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946" event={"ID":"fd346e50-c9ff-4804-a93c-d77ebbcf6d77","Type":"ContainerDied","Data":"6ae04e939203f4e54939c31e552d38d4ed027a9c7d95e8c8674e9d5e70033409"}
Sep 16 19:56:12.348978 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:12.348755    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946" event={"ID":"fd346e50-c9ff-4804-a93c-d77ebbcf6d77","Type":"ContainerDied","Data":"43f030de4f5a0291f88a7a76fa21456cab969d4c9cc645659f9c4e9eb376a5fb"}
Sep 16 19:56:12.348978 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:12.348778    2577 scope.go:117] "RemoveContainer" containerID="6ae04e939203f4e54939c31e552d38d4ed027a9c7d95e8c8674e9d5e70033409"
Sep 16 19:56:12.357701 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:12.357684    2577 scope.go:117] "RemoveContainer" containerID="e092832ff5fc346098dea96e5c62c0a3f00bfa5c4a3b68f875b8282bad11bd02"
Sep 16 19:56:12.365370 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:12.365352    2577 scope.go:117] "RemoveContainer" containerID="6ae04e939203f4e54939c31e552d38d4ed027a9c7d95e8c8674e9d5e70033409"
Sep 16 19:56:12.365705 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:56:12.365687    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ae04e939203f4e54939c31e552d38d4ed027a9c7d95e8c8674e9d5e70033409\": container with ID starting with 6ae04e939203f4e54939c31e552d38d4ed027a9c7d95e8c8674e9d5e70033409 not found: ID does not exist" containerID="6ae04e939203f4e54939c31e552d38d4ed027a9c7d95e8c8674e9d5e70033409"
Sep 16 19:56:12.365758 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:12.365714    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ae04e939203f4e54939c31e552d38d4ed027a9c7d95e8c8674e9d5e70033409"} err="failed to get container status \"6ae04e939203f4e54939c31e552d38d4ed027a9c7d95e8c8674e9d5e70033409\": rpc error: code = NotFound desc = could not find container \"6ae04e939203f4e54939c31e552d38d4ed027a9c7d95e8c8674e9d5e70033409\": container with ID starting with 6ae04e939203f4e54939c31e552d38d4ed027a9c7d95e8c8674e9d5e70033409 not found: ID does not exist"
Sep 16 19:56:12.365758 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:12.365729    2577 scope.go:117] "RemoveContainer" containerID="e092832ff5fc346098dea96e5c62c0a3f00bfa5c4a3b68f875b8282bad11bd02"
Sep 16 19:56:12.365998 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:56:12.365979    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e092832ff5fc346098dea96e5c62c0a3f00bfa5c4a3b68f875b8282bad11bd02\": container with ID starting with e092832ff5fc346098dea96e5c62c0a3f00bfa5c4a3b68f875b8282bad11bd02 not found: ID does not exist" containerID="e092832ff5fc346098dea96e5c62c0a3f00bfa5c4a3b68f875b8282bad11bd02"
Sep 16 19:56:12.366072 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:12.366007    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e092832ff5fc346098dea96e5c62c0a3f00bfa5c4a3b68f875b8282bad11bd02"} err="failed to get container status \"e092832ff5fc346098dea96e5c62c0a3f00bfa5c4a3b68f875b8282bad11bd02\": rpc error: code = NotFound desc = could not find container \"e092832ff5fc346098dea96e5c62c0a3f00bfa5c4a3b68f875b8282bad11bd02\": container with ID starting with e092832ff5fc346098dea96e5c62c0a3f00bfa5c4a3b68f875b8282bad11bd02 not found: ID does not exist"
Sep 16 19:56:12.373759 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:12.373737    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946"]
Sep 16 19:56:12.378420 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:12.378400    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58db669fb6lh946"]
Sep 16 19:56:13.155201 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:13.155158    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd346e50-c9ff-4804-a93c-d77ebbcf6d77" path="/var/lib/kubelet/pods/fd346e50-c9ff-4804-a93c-d77ebbcf6d77/volumes"
Sep 16 19:56:58.522953 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:58.522910    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-l6pv9/must-gather-7944z"]
Sep 16 19:56:58.523379 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:58.523256    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd346e50-c9ff-4804-a93c-d77ebbcf6d77" containerName="main"
Sep 16 19:56:58.523379 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:58.523267    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd346e50-c9ff-4804-a93c-d77ebbcf6d77" containerName="main"
Sep 16 19:56:58.523379 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:58.523276    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0825f3b-1db0-4450-b180-3e6c205991be" containerName="main"
Sep 16 19:56:58.523379 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:58.523282    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0825f3b-1db0-4450-b180-3e6c205991be" containerName="main"
Sep 16 19:56:58.523379 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:58.523297    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd346e50-c9ff-4804-a93c-d77ebbcf6d77" containerName="storage-initializer"
Sep 16 19:56:58.523379 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:58.523303    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd346e50-c9ff-4804-a93c-d77ebbcf6d77" containerName="storage-initializer"
Sep 16 19:56:58.523379 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:58.523352    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd346e50-c9ff-4804-a93c-d77ebbcf6d77" containerName="main"
Sep 16 19:56:58.523379 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:58.523361    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0825f3b-1db0-4450-b180-3e6c205991be" containerName="main"
Sep 16 19:56:58.527724 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:58.527705    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l6pv9/must-gather-7944z"
Sep 16 19:56:58.530691 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:58.530672    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-l6pv9\"/\"kube-root-ca.crt\""
Sep 16 19:56:58.531984 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:58.531964    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-l6pv9\"/\"openshift-service-ca.crt\""
Sep 16 19:56:58.532056 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:58.531992    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-l6pv9\"/\"default-dockercfg-8p8t7\""
Sep 16 19:56:58.559067 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:58.559043    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l6pv9/must-gather-7944z"]
Sep 16 19:56:58.716722 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:58.716699    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a282d8f2-48f0-409b-9fb8-9ae0bb5ae125-must-gather-output\") pod \"must-gather-7944z\" (UID: \"a282d8f2-48f0-409b-9fb8-9ae0bb5ae125\") " pod="openshift-must-gather-l6pv9/must-gather-7944z"
Sep 16 19:56:58.716835 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:58.716752    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j55vj\" (UniqueName: \"kubernetes.io/projected/a282d8f2-48f0-409b-9fb8-9ae0bb5ae125-kube-api-access-j55vj\") pod \"must-gather-7944z\" (UID: \"a282d8f2-48f0-409b-9fb8-9ae0bb5ae125\") " pod="openshift-must-gather-l6pv9/must-gather-7944z"
Sep 16 19:56:58.817108 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:58.817044    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a282d8f2-48f0-409b-9fb8-9ae0bb5ae125-must-gather-output\") pod \"must-gather-7944z\" (UID: \"a282d8f2-48f0-409b-9fb8-9ae0bb5ae125\") " pod="openshift-must-gather-l6pv9/must-gather-7944z"
Sep 16 19:56:58.817230 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:58.817127    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j55vj\" (UniqueName: \"kubernetes.io/projected/a282d8f2-48f0-409b-9fb8-9ae0bb5ae125-kube-api-access-j55vj\") pod \"must-gather-7944z\" (UID: \"a282d8f2-48f0-409b-9fb8-9ae0bb5ae125\") " pod="openshift-must-gather-l6pv9/must-gather-7944z"
Sep 16 19:56:58.817389 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:58.817366    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a282d8f2-48f0-409b-9fb8-9ae0bb5ae125-must-gather-output\") pod \"must-gather-7944z\" (UID: \"a282d8f2-48f0-409b-9fb8-9ae0bb5ae125\") " pod="openshift-must-gather-l6pv9/must-gather-7944z"
Sep 16 19:56:58.828381 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:58.828357    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j55vj\" (UniqueName: \"kubernetes.io/projected/a282d8f2-48f0-409b-9fb8-9ae0bb5ae125-kube-api-access-j55vj\") pod \"must-gather-7944z\" (UID: \"a282d8f2-48f0-409b-9fb8-9ae0bb5ae125\") " pod="openshift-must-gather-l6pv9/must-gather-7944z"
Sep 16 19:56:58.837240 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:58.837222    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l6pv9/must-gather-7944z"
Sep 16 19:56:59.106162 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:59.106014    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l6pv9/must-gather-7944z"]
Sep 16 19:56:59.108856 ip-10-0-128-247 kubenswrapper[2577]: W0916 19:56:59.108827    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda282d8f2_48f0_409b_9fb8_9ae0bb5ae125.slice/crio-44623b5be29f0d133b81508c2faa364680986fbbbb222a027c8453ba67c12b22 WatchSource:0}: Error finding container 44623b5be29f0d133b81508c2faa364680986fbbbb222a027c8453ba67c12b22: Status 404 returned error can't find the container with id 44623b5be29f0d133b81508c2faa364680986fbbbb222a027c8453ba67c12b22
Sep 16 19:56:59.110599 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:59.110582    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:56:59.541439 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:56:59.541405    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l6pv9/must-gather-7944z" event={"ID":"a282d8f2-48f0-409b-9fb8-9ae0bb5ae125","Type":"ContainerStarted","Data":"44623b5be29f0d133b81508c2faa364680986fbbbb222a027c8453ba67c12b22"}
Sep 16 19:57:05.574799 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:05.574703    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l6pv9/must-gather-7944z" event={"ID":"a282d8f2-48f0-409b-9fb8-9ae0bb5ae125","Type":"ContainerStarted","Data":"d3fdd28e79b71fc4d9bb719dfdfab72743a8efd38c65ffdc975473601f1e8cfd"}
Sep 16 19:57:05.574799 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:05.574748    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l6pv9/must-gather-7944z" event={"ID":"a282d8f2-48f0-409b-9fb8-9ae0bb5ae125","Type":"ContainerStarted","Data":"d6cd2338eb68719adc42163557ec93f798e04dda06b294e079648ddd5b0c80ec"}
Sep 16 19:57:05.600436 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:05.596277    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-l6pv9/must-gather-7944z" podStartSLOduration=1.477194594 podStartE2EDuration="7.596257448s" totalImagesPullingTime="6.119062854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:56:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:57:05.594649125 +0000 UTC m=+4582.977538493" watchObservedRunningTime="2026-09-16 19:57:05.596257448 +0000 UTC m=+4582.979146818"
Sep 16 19:57:21.611401 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:21.611366    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5bb5799fbb-klzv5_cf4b4dde-c989-408c-8a0d-a443fa66e2bf/router/0.log"
Sep 16 19:57:22.510579 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:22.510544    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5bb5799fbb-klzv5_cf4b4dde-c989-408c-8a0d-a443fa66e2bf/router/0.log"
Sep 16 19:57:24.140785 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:24.140758    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-4g87j_6d452793-9c5c-4869-b57e-650ab85fe261/limitador/0.log"
Sep 16 19:57:28.688372 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:28.688337    2577 generic.go:356] "Generic (PLEG): container finished" podID="a282d8f2-48f0-409b-9fb8-9ae0bb5ae125" containerID="d6cd2338eb68719adc42163557ec93f798e04dda06b294e079648ddd5b0c80ec" exitCode=0
Sep 16 19:57:28.688795 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:28.688386    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l6pv9/must-gather-7944z" event={"ID":"a282d8f2-48f0-409b-9fb8-9ae0bb5ae125","Type":"ContainerDied","Data":"d6cd2338eb68719adc42163557ec93f798e04dda06b294e079648ddd5b0c80ec"}
Sep 16 19:57:28.688795 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:28.688722    2577 scope.go:117] "RemoveContainer" containerID="d6cd2338eb68719adc42163557ec93f798e04dda06b294e079648ddd5b0c80ec"
Sep 16 19:57:29.541611 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:29.541577    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-l6pv9_must-gather-7944z_a282d8f2-48f0-409b-9fb8-9ae0bb5ae125/gather/0.log"
Sep 16 19:57:33.089769 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:33.089735    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-88mzt_2b544a31-67f6-4a93-beeb-a21fd49b00b1/global-pull-secret-syncer/0.log"
Sep 16 19:57:33.326595 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:33.326566    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jpq9w_025a78d4-1397-4057-89d3-56bbf1d96e2c/konnectivity-agent/0.log"
Sep 16 19:57:33.350886 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:33.350816    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-247.ec2.internal_146a57ab6e613bbac55bc1f7a3764a51/haproxy/0.log"
Sep 16 19:57:34.990660 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:34.990622    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-l6pv9/must-gather-7944z"]
Sep 16 19:57:34.991172 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:34.990807    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-l6pv9/must-gather-7944z" podUID="a282d8f2-48f0-409b-9fb8-9ae0bb5ae125" containerName="copy" containerID="cri-o://d3fdd28e79b71fc4d9bb719dfdfab72743a8efd38c65ffdc975473601f1e8cfd" gracePeriod=2
Sep 16 19:57:35.002575 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:35.002547    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-l6pv9/must-gather-7944z"]
Sep 16 19:57:35.005698 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:57:35.005652    2577 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"a282d8f2-48f0-409b-9fb8-9ae0bb5ae125\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"openshift-must-gather-l6pv9\"/\"must-gather-7944z\": pods \"must-gather-7944z\" not found" pod="openshift-must-gather-l6pv9/must-gather-7944z"
Sep 16 19:57:35.219805 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:35.219779    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-l6pv9_must-gather-7944z_a282d8f2-48f0-409b-9fb8-9ae0bb5ae125/copy/0.log"
Sep 16 19:57:35.221223 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:35.221206    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l6pv9/must-gather-7944z"
Sep 16 19:57:35.313413 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:35.313362    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a282d8f2-48f0-409b-9fb8-9ae0bb5ae125-must-gather-output\") pod \"a282d8f2-48f0-409b-9fb8-9ae0bb5ae125\" (UID: \"a282d8f2-48f0-409b-9fb8-9ae0bb5ae125\") "
Sep 16 19:57:35.313413 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:35.313403    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j55vj\" (UniqueName: \"kubernetes.io/projected/a282d8f2-48f0-409b-9fb8-9ae0bb5ae125-kube-api-access-j55vj\") pod \"a282d8f2-48f0-409b-9fb8-9ae0bb5ae125\" (UID: \"a282d8f2-48f0-409b-9fb8-9ae0bb5ae125\") "
Sep 16 19:57:35.315736 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:35.315714    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a282d8f2-48f0-409b-9fb8-9ae0bb5ae125-kube-api-access-j55vj" (OuterVolumeSpecName: "kube-api-access-j55vj") pod "a282d8f2-48f0-409b-9fb8-9ae0bb5ae125" (UID: "a282d8f2-48f0-409b-9fb8-9ae0bb5ae125"). InnerVolumeSpecName "kube-api-access-j55vj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:57:35.320414 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:35.320385    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a282d8f2-48f0-409b-9fb8-9ae0bb5ae125-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "a282d8f2-48f0-409b-9fb8-9ae0bb5ae125" (UID: "a282d8f2-48f0-409b-9fb8-9ae0bb5ae125"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:57:35.414592 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:35.414570    2577 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a282d8f2-48f0-409b-9fb8-9ae0bb5ae125-must-gather-output\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:57:35.414592 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:35.414591    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j55vj\" (UniqueName: \"kubernetes.io/projected/a282d8f2-48f0-409b-9fb8-9ae0bb5ae125-kube-api-access-j55vj\") on node \"ip-10-0-128-247.ec2.internal\" DevicePath \"\""
Sep 16 19:57:35.718691 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:35.718665    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-l6pv9_must-gather-7944z_a282d8f2-48f0-409b-9fb8-9ae0bb5ae125/copy/0.log"
Sep 16 19:57:35.719046 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:35.719022    2577 generic.go:356] "Generic (PLEG): container finished" podID="a282d8f2-48f0-409b-9fb8-9ae0bb5ae125" containerID="d3fdd28e79b71fc4d9bb719dfdfab72743a8efd38c65ffdc975473601f1e8cfd" exitCode=143
Sep 16 19:57:35.719127 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:35.719074    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l6pv9/must-gather-7944z"
Sep 16 19:57:35.719127 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:35.719118    2577 scope.go:117] "RemoveContainer" containerID="d3fdd28e79b71fc4d9bb719dfdfab72743a8efd38c65ffdc975473601f1e8cfd"
Sep 16 19:57:35.727494 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:35.727380    2577 scope.go:117] "RemoveContainer" containerID="d6cd2338eb68719adc42163557ec93f798e04dda06b294e079648ddd5b0c80ec"
Sep 16 19:57:35.741193 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:35.741176    2577 scope.go:117] "RemoveContainer" containerID="d3fdd28e79b71fc4d9bb719dfdfab72743a8efd38c65ffdc975473601f1e8cfd"
Sep 16 19:57:35.741439 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:57:35.741419    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3fdd28e79b71fc4d9bb719dfdfab72743a8efd38c65ffdc975473601f1e8cfd\": container with ID starting with d3fdd28e79b71fc4d9bb719dfdfab72743a8efd38c65ffdc975473601f1e8cfd not found: ID does not exist" containerID="d3fdd28e79b71fc4d9bb719dfdfab72743a8efd38c65ffdc975473601f1e8cfd"
Sep 16 19:57:35.741499 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:35.741445    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3fdd28e79b71fc4d9bb719dfdfab72743a8efd38c65ffdc975473601f1e8cfd"} err="failed to get container status \"d3fdd28e79b71fc4d9bb719dfdfab72743a8efd38c65ffdc975473601f1e8cfd\": rpc error: code = NotFound desc = could not find container \"d3fdd28e79b71fc4d9bb719dfdfab72743a8efd38c65ffdc975473601f1e8cfd\": container with ID starting with d3fdd28e79b71fc4d9bb719dfdfab72743a8efd38c65ffdc975473601f1e8cfd not found: ID does not exist"
Sep 16 19:57:35.741499 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:35.741463    2577 scope.go:117] "RemoveContainer" containerID="d6cd2338eb68719adc42163557ec93f798e04dda06b294e079648ddd5b0c80ec"
Sep 16 19:57:35.741744 ip-10-0-128-247 kubenswrapper[2577]: E0916 19:57:35.741723    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6cd2338eb68719adc42163557ec93f798e04dda06b294e079648ddd5b0c80ec\": container with ID starting with d6cd2338eb68719adc42163557ec93f798e04dda06b294e079648ddd5b0c80ec not found: ID does not exist" containerID="d6cd2338eb68719adc42163557ec93f798e04dda06b294e079648ddd5b0c80ec"
Sep 16 19:57:35.741790 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:35.741751    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6cd2338eb68719adc42163557ec93f798e04dda06b294e079648ddd5b0c80ec"} err="failed to get container status \"d6cd2338eb68719adc42163557ec93f798e04dda06b294e079648ddd5b0c80ec\": rpc error: code = NotFound desc = could not find container \"d6cd2338eb68719adc42163557ec93f798e04dda06b294e079648ddd5b0c80ec\": container with ID starting with d6cd2338eb68719adc42163557ec93f798e04dda06b294e079648ddd5b0c80ec not found: ID does not exist"
Sep 16 19:57:37.154675 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:37.154641    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a282d8f2-48f0-409b-9fb8-9ae0bb5ae125" path="/var/lib/kubelet/pods/a282d8f2-48f0-409b-9fb8-9ae0bb5ae125/volumes"
Sep 16 19:57:38.666870 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:38.666840    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-4g87j_6d452793-9c5c-4869-b57e-650ab85fe261/limitador/0.log"
Sep 16 19:57:39.936398 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:39.936368    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef/alertmanager/0.log"
Sep 16 19:57:39.963605 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:39.963575    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef/config-reloader/0.log"
Sep 16 19:57:39.990272 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:39.990250    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef/kube-rbac-proxy-web/0.log"
Sep 16 19:57:40.016635 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:40.016613    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef/kube-rbac-proxy/0.log"
Sep 16 19:57:40.047098 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:40.047078    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef/kube-rbac-proxy-metric/0.log"
Sep 16 19:57:40.075038 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:40.075015    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef/prom-label-proxy/0.log"
Sep 16 19:57:40.101327 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:40.101306    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_17b4a9e5-49dd-4f0e-b209-c7d7bf1c3bef/init-config-reloader/0.log"
Sep 16 19:57:40.200475 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:40.200420    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-tf57h_1ec56458-30a8-48bb-bb63-4ac4d6262080/kube-state-metrics/0.log"
Sep 16 19:57:40.226143 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:40.226122    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-tf57h_1ec56458-30a8-48bb-bb63-4ac4d6262080/kube-rbac-proxy-main/0.log"
Sep 16 19:57:40.253313 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:40.253293    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-tf57h_1ec56458-30a8-48bb-bb63-4ac4d6262080/kube-rbac-proxy-self/0.log"
Sep 16 19:57:40.316350 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:40.316333    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-l6f5z_c67af528-b4c8-4dc4-97e2-2e91547e6020/monitoring-plugin/0.log"
Sep 16 19:57:40.463481 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:40.463420    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tqgtg_3e478983-4465-4a94-b2ed-fd0032d4dc38/node-exporter/0.log"
Sep 16 19:57:40.482950 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:40.482933    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tqgtg_3e478983-4465-4a94-b2ed-fd0032d4dc38/kube-rbac-proxy/0.log"
Sep 16 19:57:40.507222 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:40.507198    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tqgtg_3e478983-4465-4a94-b2ed-fd0032d4dc38/init-textfile/0.log"
Sep 16 19:57:41.192781 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:41.192753    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-779cc55674-bq9jx_1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1/thanos-query/0.log"
Sep 16 19:57:41.218343 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:41.218319    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-779cc55674-bq9jx_1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1/kube-rbac-proxy-web/0.log"
Sep 16 19:57:41.244790 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:41.244774    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-779cc55674-bq9jx_1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1/kube-rbac-proxy/0.log"
Sep 16 19:57:41.269716 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:41.269692    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-779cc55674-bq9jx_1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1/prom-label-proxy/0.log"
Sep 16 19:57:41.296902 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:41.296879    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-779cc55674-bq9jx_1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1/kube-rbac-proxy-rules/0.log"
Sep 16 19:57:41.324787 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:41.324762    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-779cc55674-bq9jx_1e0b0b6b-b2c8-4e3a-92a0-0c67d14794f1/kube-rbac-proxy-metrics/0.log"
Sep 16 19:57:43.181711 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:43.181670    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mkvvc/perf-node-gather-daemonset-5t6dj"]
Sep 16 19:57:43.182169 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:43.182122    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a282d8f2-48f0-409b-9fb8-9ae0bb5ae125" containerName="gather"
Sep 16 19:57:43.182169 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:43.182141    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a282d8f2-48f0-409b-9fb8-9ae0bb5ae125" containerName="gather"
Sep 16 19:57:43.182169 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:43.182156    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a282d8f2-48f0-409b-9fb8-9ae0bb5ae125" containerName="copy"
Sep 16 19:57:43.182169 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:43.182163    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a282d8f2-48f0-409b-9fb8-9ae0bb5ae125" containerName="copy"
Sep 16 19:57:43.182381 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:43.182244    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a282d8f2-48f0-409b-9fb8-9ae0bb5ae125" containerName="copy"
Sep 16 19:57:43.182381 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:43.182259    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a282d8f2-48f0-409b-9fb8-9ae0bb5ae125" containerName="gather"
Sep 16 19:57:43.184170 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:43.184150    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-5t6dj"
Sep 16 19:57:43.187376 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:43.187360    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mkvvc\"/\"kube-root-ca.crt\""
Sep 16 19:57:43.188718 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:43.188702    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mkvvc\"/\"default-dockercfg-h7jhd\""
Sep 16 19:57:43.188777 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:43.188722    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mkvvc\"/\"openshift-service-ca.crt\""
Sep 16 19:57:43.196237 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:43.196216    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mkvvc/perf-node-gather-daemonset-5t6dj"]
Sep 16 19:57:43.277337 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:43.277309    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5b1b9ac0-8a37-4ce1-bea0-3ffecd872b05-sys\") pod \"perf-node-gather-daemonset-5t6dj\" (UID: \"5b1b9ac0-8a37-4ce1-bea0-3ffecd872b05\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-5t6dj"
Sep 16 19:57:43.277440 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:43.277344    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5b1b9ac0-8a37-4ce1-bea0-3ffecd872b05-lib-modules\") pod \"perf-node-gather-daemonset-5t6dj\" (UID: \"5b1b9ac0-8a37-4ce1-bea0-3ffecd872b05\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-5t6dj"
Sep 16 19:57:43.277482 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:43.277450    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5b1b9ac0-8a37-4ce1-bea0-3ffecd872b05-podres\") pod \"perf-node-gather-daemonset-5t6dj\" (UID: \"5b1b9ac0-8a37-4ce1-bea0-3ffecd872b05\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-5t6dj"
Sep 16 19:57:43.277620 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:43.277588    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbwbb\" (UniqueName: \"kubernetes.io/projected/5b1b9ac0-8a37-4ce1-bea0-3ffecd872b05-kube-api-access-nbwbb\") pod \"perf-node-gather-daemonset-5t6dj\" (UID: \"5b1b9ac0-8a37-4ce1-bea0-3ffecd872b05\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-5t6dj"
Sep 16 19:57:43.277709 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:43.277687    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5b1b9ac0-8a37-4ce1-bea0-3ffecd872b05-proc\") pod \"perf-node-gather-daemonset-5t6dj\" (UID: \"5b1b9ac0-8a37-4ce1-bea0-3ffecd872b05\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-5t6dj"
Sep 16 19:57:43.378621 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:43.378594    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbwbb\" (UniqueName: \"kubernetes.io/projected/5b1b9ac0-8a37-4ce1-bea0-3ffecd872b05-kube-api-access-nbwbb\") pod \"perf-node-gather-daemonset-5t6dj\" (UID: \"5b1b9ac0-8a37-4ce1-bea0-3ffecd872b05\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-5t6dj"
Sep 16 19:57:43.378714 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:43.378632    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5b1b9ac0-8a37-4ce1-bea0-3ffecd872b05-proc\") pod \"perf-node-gather-daemonset-5t6dj\" (UID: \"5b1b9ac0-8a37-4ce1-bea0-3ffecd872b05\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-5t6dj"
Sep 16 19:57:43.378714 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:43.378658    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5b1b9ac0-8a37-4ce1-bea0-3ffecd872b05-sys\") pod \"perf-node-gather-daemonset-5t6dj\" (UID: \"5b1b9ac0-8a37-4ce1-bea0-3ffecd872b05\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-5t6dj"
Sep 16 19:57:43.378714 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:43.378679    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5b1b9ac0-8a37-4ce1-bea0-3ffecd872b05-lib-modules\") pod \"perf-node-gather-daemonset-5t6dj\" (UID: \"5b1b9ac0-8a37-4ce1-bea0-3ffecd872b05\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-5t6dj"
Sep 16 19:57:43.378830 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:43.378717    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5b1b9ac0-8a37-4ce1-bea0-3ffecd872b05-podres\") pod \"perf-node-gather-daemonset-5t6dj\" (UID: \"5b1b9ac0-8a37-4ce1-bea0-3ffecd872b05\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-5t6dj"
Sep 16 19:57:43.378830 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:43.378740    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5b1b9ac0-8a37-4ce1-bea0-3ffecd872b05-sys\") pod \"perf-node-gather-daemonset-5t6dj\" (UID: \"5b1b9ac0-8a37-4ce1-bea0-3ffecd872b05\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-5t6dj"
Sep 16 19:57:43.378830 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:43.378717    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5b1b9ac0-8a37-4ce1-bea0-3ffecd872b05-proc\") pod \"perf-node-gather-daemonset-5t6dj\" (UID: \"5b1b9ac0-8a37-4ce1-bea0-3ffecd872b05\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-5t6dj"
Sep 16 19:57:43.378830 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:43.378801    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5b1b9ac0-8a37-4ce1-bea0-3ffecd872b05-lib-modules\") pod \"perf-node-gather-daemonset-5t6dj\" (UID: \"5b1b9ac0-8a37-4ce1-bea0-3ffecd872b05\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-5t6dj"
Sep 16 19:57:43.378964 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:43.378840    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5b1b9ac0-8a37-4ce1-bea0-3ffecd872b05-podres\") pod \"perf-node-gather-daemonset-5t6dj\" (UID: \"5b1b9ac0-8a37-4ce1-bea0-3ffecd872b05\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-5t6dj"
Sep 16 19:57:43.388812 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:43.388788    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbwbb\" (UniqueName: \"kubernetes.io/projected/5b1b9ac0-8a37-4ce1-bea0-3ffecd872b05-kube-api-access-nbwbb\") pod \"perf-node-gather-daemonset-5t6dj\" (UID: \"5b1b9ac0-8a37-4ce1-bea0-3ffecd872b05\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-5t6dj"
Sep 16 19:57:43.493639 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:43.493581    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-5t6dj"
Sep 16 19:57:43.771422 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:43.771392    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mkvvc/perf-node-gather-daemonset-5t6dj"]
Sep 16 19:57:43.774692 ip-10-0-128-247 kubenswrapper[2577]: W0916 19:57:43.774666    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5b1b9ac0_8a37_4ce1_bea0_3ffecd872b05.slice/crio-796e56d63853fc82c157f64b6f774b1a1d6aa57b3237d42ad01083ccabdba376 WatchSource:0}: Error finding container 796e56d63853fc82c157f64b6f774b1a1d6aa57b3237d42ad01083ccabdba376: Status 404 returned error can't find the container with id 796e56d63853fc82c157f64b6f774b1a1d6aa57b3237d42ad01083ccabdba376
Sep 16 19:57:44.216399 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:44.216371    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-2z2cr_ecc4cc43-8c85-4a54-b0ee-6107f507e1bd/volume-data-source-validator/0.log"
Sep 16 19:57:44.760813 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:44.760774    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-5t6dj" event={"ID":"5b1b9ac0-8a37-4ce1-bea0-3ffecd872b05","Type":"ContainerStarted","Data":"1dcc310a3979da11b00ea4d770579ae76c7f5d5d0b31c82856741fdedf1e5d3d"}
Sep 16 19:57:44.761003 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:44.760854    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-5t6dj" event={"ID":"5b1b9ac0-8a37-4ce1-bea0-3ffecd872b05","Type":"ContainerStarted","Data":"796e56d63853fc82c157f64b6f774b1a1d6aa57b3237d42ad01083ccabdba376"}
Sep 16 19:57:44.761003 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:44.760918    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-5t6dj"
Sep 16 19:57:44.780079 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:44.780035    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-5t6dj" podStartSLOduration=1.780020903 podStartE2EDuration="1.780020903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:57:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:57:44.778242174 +0000 UTC m=+4622.161131540" watchObservedRunningTime="2026-09-16 19:57:44.780020903 +0000 UTC m=+4622.162910311"
Sep 16 19:57:45.315213 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:45.315182    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sxvsf_6a2f00df-e26f-4f86-89ec-eefef7fa3ed0/dns/0.log"
Sep 16 19:57:45.338956 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:45.338930    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sxvsf_6a2f00df-e26f-4f86-89ec-eefef7fa3ed0/kube-rbac-proxy/0.log"
Sep 16 19:57:45.447349 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:45.447327    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wmfjw_590e4879-7945-4583-b644-0caa6b1c6ebd/dns-node-resolver/0.log"
Sep 16 19:57:46.045948 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:46.045915    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-76b8cdc779-5f86t_ea1706f1-a58e-418b-957b-c7965c54efc9/registry/0.log"
Sep 16 19:57:46.073065 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:46.073040    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8cxsn_f935c176-f27b-4179-bc5e-802ee61499d3/node-ca/0.log"
Sep 16 19:57:47.461651 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:47.461623    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5bb5799fbb-klzv5_cf4b4dde-c989-408c-8a0d-a443fa66e2bf/router/0.log"
Sep 16 19:57:48.086355 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:48.086326    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ghpxc_f5bcc8e6-e1b2-4b7d-840f-f4ca1b31b520/serve-healthcheck-canary/0.log"
Sep 16 19:57:48.818681 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:48.818649    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vf4vz_ce0ea9d9-201e-4df7-aaa4-2549409434a8/kube-rbac-proxy/0.log"
Sep 16 19:57:48.845010 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:48.844984    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vf4vz_ce0ea9d9-201e-4df7-aaa4-2549409434a8/exporter/0.log"
Sep 16 19:57:48.872639 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:48.872615    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vf4vz_ce0ea9d9-201e-4df7-aaa4-2549409434a8/extractor/0.log"
Sep 16 19:57:50.775421 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:50.775389    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-5t6dj"
Sep 16 19:57:51.973623 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:51.973592    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-846d5d4569-f72mc_2103e78d-58d1-4458-8fe1-903caef6cf25/manager/0.log"
Sep 16 19:57:53.082516 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:53.082468    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5bb5799fbb-klzv5_cf4b4dde-c989-408c-8a0d-a443fa66e2bf/router/0.log"
Sep 16 19:57:54.587676 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:54.587632    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5bb5799fbb-klzv5_cf4b4dde-c989-408c-8a0d-a443fa66e2bf/router/0.log"
Sep 16 19:57:54.598137 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:54.598113    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-sb42p_96f8bb5f-041a-48db-89a6-7ba9cf3192e1/s3-init/0.log"
Sep 16 19:57:54.621204 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:54.621185    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-sb42p_96f8bb5f-041a-48db-89a6-7ba9cf3192e1/download-hf-model/0.log"
Sep 16 19:57:59.552772 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:59.552741    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-rhbfr_0edc11ae-5a70-42d6-81fc-2c7591912cdb/migrator/0.log"
Sep 16 19:57:59.576231 ip-10-0-128-247 kubenswrapper[2577]: I0916 19:57:59.576207    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-rhbfr_0edc11ae-5a70-42d6-81fc-2c7591912cdb/graceful-termination/0.log"